Artificial intelligence device, operation method of artificial intelligence device, and computer-readable recording medium in which program for performing operation method of artificial intelligence device is recorded

Abstract

An artificial intelligence device according to an embodiment of the present disclosure may comprise: a communication interface for communicating with a server; a memory; and one or more processors which: transmit driving-related state data to the server on the basis that a safety rate according to execution of rule-based autonomous emergency braking (AEB) control is less than a reference safety rate; receive, from the server, a large language model (LLM) trained on the basis of the driving-related state data, and store the LLM in the memory; acquire new driving-related state data; infer a cause of failure of the AEB control from the new driving-related state data through the LLM; acquire control logic of an electronic control unit (ECU) corresponding to the inferred cause of the failure of the AEB control; and transmit, to the ECU, the acquired control logic of the ECU.

Description

An artificial intelligence device, an operating method of an artificial intelligence device, and a computer-readable recording medium recording a program for performing the operating method of an artificial intelligence device

[0001] The present disclosure relates to an artificial intelligence device for enhancing vehicle control performance.

[0002] A vehicle is a device that moves its user in the desired direction. A representative example is an automobile. Recent vehicles are equipped with Advanced Driver Assistance Systems (ADAS) to assist drivers in driving.

[0003] ADAS is an active safety device that uses sensors or cameras to detect dangerous situations, warn the driver of the risk of an accident, and help the driver make decisions and respond accordingly.

[0004] Among ADAS functions, Autonomous Emergency Braking (AEB) control is a function that automatically applies the brakes to prevent vehicle accidents.

[0005] Conventional roll-based AEB control operates according to established rules and conditions, which assess and determine the risk of a collision based on data collected from the vehicle's sensors.

[0006] However, conventional rule-based AEB control relies on predefined rules, so its response to complex or unexpected driving situations is limited.

[0007] In addition, conventional rule-based AEB control has a problem in accurately recognizing and evaluating all objects in complex environments where multiple objects exist simultaneously.

[0008] Additionally, conventional rule-based AEB control may have limited performance depending on the accuracy and range of the sensor.

[0009] An object of the present disclosure may be to improve the safety factor of AEB control through LLM embedded in an edge device.

[0010] The purpose of the present disclosure may be to retrain or update the LLM to respond to various driving situations.

[0011] The purpose of the present disclosure may be to infer the cause of failure of AEB control through LLM and efficiently control the ECU according to the cause of failure.

[0012] An artificial intelligence device according to one embodiment of the present disclosure may include a communication interface for communicating with a server; a memory; and one or more processors configured to transmit driving-related state data to the server based on a safety factor according to execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety factor, receive a Large Language Model (LLM) learned based on the driving-related state data from the server and store it in the memory, acquire new driving-related state data, infer a cause of failure of the AEB control from the new driving-related state data through the LLM, acquire control logic of an electronic control unit (ECU) corresponding to the inferred cause of failure of the AEB control, and transmit the acquired ECU control logic to the ECU.

[0013] A computer-readable recording medium storing a program for performing an operating method of an artificial intelligence device according to one embodiment of the present disclosure, the operating method may include: transmitting driving-related state data to a server based on a safety factor according to execution of rule-based autonomous emergency braking (AEB) control being less than a reference safety factor; receiving and storing a Large Language Model (LLM) learned based on the driving-related state data from the server; obtaining new driving-related state data; inferring a cause of failure of the AEB control from the new driving-related state data through the LLM; obtaining a control logic of an electronic control unit (ECU) corresponding to the inferred cause of failure of the AEB control; and transmitting the obtained ECU control logic to the ECU.

[0014] An operating method of an artificial intelligence device according to an embodiment of the present disclosure may include: transmitting driving-related state data to a server based on a safety rate according to execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety rate; receiving and storing a Large Language Model (LLM) learned based on the driving-related state data from the server; obtaining new driving-related state data; inferring a cause of failure of the AEB control from the new driving-related state data through the LLM; obtaining a control logic of an electronic control unit (ECU) corresponding to the inferred cause of failure of the AEB control; and transmitting the obtained ECU control logic to the ECU.

[0015] According to an embodiment of the present disclosure, it is possible to efficiently respond to fail-safe AEB control.

[0016] According to an embodiment of the present disclosure, various driving situations can be analyzed through LLM, and AEB control can be flexibly performed.

[0017] According to an embodiment of the present disclosure, the possibility of a vehicle collision can be prevented in advance by analyzing the cause of failure of AEB control in real time and controlling the ECU (Electronic Control Unit) in response to the cause of failure.

[0018] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0019] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0020] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0021] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0022] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0023] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0024] FIG. 6 is a drawing for explaining the operation of a vehicle system according to an embodiment of the present disclosure.

[0025] FIG. 7 is a sequence diagram for explaining a fail-safe response method based on LLM of a system according to one embodiment of the present disclosure.

[0026] FIG. 8 is a diagram illustrating a learning process of LLM according to an embodiment of the present disclosure.

[0027] FIG. 9 is a diagram illustrating a table matching the cause of failure of AEB control and ECU control logic according to one embodiment of the present disclosure.

[0028] FIG. 10A and FIG. 10B are diagrams illustrating a warning message including a cause of failure of AEB control according to an embodiment of the present disclosure.

[0029] FIG. 11 is a diagram illustrating the frame structure of a Controller Area Network (CAN) message transmitted to a server by an artificial intelligence device according to an embodiment of the present disclosure.

[0030] FIG. 12 is a diagram illustrating modes related to data processed by an ECU according to one embodiment of the present disclosure.

[0031] FIG. 13 is a flowchart illustrating an operation method of an artificial intelligence device according to an embodiment of the present disclosure.

[0032] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0033] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0034] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0035] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0036] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0037] Meanwhile, the vehicle (200) may be equipped with multiple displays (180a, 180b) for displaying images, information, etc. inside.

[0038] In Fig. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) is also possible.

[0039] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be called a center information display.

[0040] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0041] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0042] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0043] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the plurality of zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the plurality of zones (Z1 to Z4).

[0044] Meanwhile, the signal processing device (170a) may further include, in addition to the vehicle communication gateway (GWDa), an autonomous driving control module (ACC), a cockpit control module (CPG), etc.

[0045] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be an HPC (High Performance Computing) gateway.

[0046] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) within a plurality of zones (Z1 to Z4).

[0047] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0048] Referring to the drawing, the interior of the vehicle may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a room mirror display (not shown), etc.

[0049] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0050] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0051] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0052] The signal processing device (170) has a processor (175) therein and can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0053] A second virtual machine (not shown) can operate for the first display (180a), and a third virtual machine (not shown) can operate for the second display (180b).

[0054] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0055] Meanwhile, the first virtual machine (not shown) within the processor (175) shares at least a portion of data with the second virtual machine (not shown) and the third virtual machine (not shown) for data sharing processing. Accordingly, data can be shared and processed among multiple virtual machines for multiple displays within the vehicle.

[0056] Meanwhile, a first virtual machine (not shown) within a processor (175) may receive and process vehicle wheel speed sensor data, and transmit the processed wheel speed sensor data to at least one of a second virtual machine (not shown) or a third virtual machine (not shown). Accordingly, the vehicle wheel speed sensor data may be shared with at least one virtual machine.

[0057] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information, or images.

[0058] The signal processing device (170) can control the RSE display (180c) by executing a fourth virtual machine (not shown) in addition to the first virtual machine to the third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0059] Accordingly, it is possible to control various displays (180a to 180c) using one signal processing device (170).

[0060] Meanwhile, some of the multiple displays (180a~180c) may operate under Linux OS, while others may operate under Web OS.

[0061] The signal processing device (170) according to the embodiment of the present disclosure can control the same information or the same image to be displayed in synchronization on displays (180a to 180c) operating under various operating systems (OS).

[0062] Meanwhile, in FIG. 3b, a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications and a vehicle speed indicator (212b) and a vehicle interior temperature indicator (213b) are displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) are displayed on a third display (180c).

[0063] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0064] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may include a lamp driving unit (751), a steering driving unit (752), a brake driving unit (753), a power source driving unit (754), a suspension driving unit (756), an air conditioning driving unit (757), a window driving unit (758), a seat driving unit (761), and a signal processing device (170).

[0065] Meanwhile, the vehicle (200) may further include an ECU (770), multiple sensor devices (SN), and multiple communication modules (EMa to EMd).

[0066] Meanwhile, a vehicle (200) according to an embodiment of the present disclosure may further include a vehicle display device (100).

[0067] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120) for communication with an external device, a plurality of communication modules (EMa to EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a to 180c), an audio output unit (185), and a power supply unit (190).

[0068] A plurality of communication modules (EMa to EMd) can be arranged, for example, in a plurality of zones (Z1 to Z4) of FIG. 2, respectively.

[0069] Meanwhile, the signal processing device (170) may have a communication switch (736b) for data communication with each communication module (EM1 to EM4) inside.

[0070] Each communication module (EM1 to EM4) can perform data communication with multiple sensor devices (SN) or ECUs (770) or area signal processing devices (170Z).

[0071] Meanwhile, the plurality of sensor devices (SN) may include a camera (195), a lidar (196), a radar (197), or a position sensor (198).

[0072] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0073] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0074] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0075] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0076] The communication device (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (900). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0077] Meanwhile, the communication device (120) can exchange data with an adjacent vehicle wirelessly.

[0078] For example, the communication device (120) can exchange vehicle messages with adjacent vehicles wirelessly through V2X (Vehicle-to-everything) communication.

[0079] A plurality of communication modules (EM1 to EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or an area signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0080] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0081] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0082] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0083] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can transmit location information data sensed by a GPS module or location sensor (198) to a signal processing device (170).

[0084] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195), a lidar (196), or a radar (197), and transmit the received information to a signal processing device (170).

[0085] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0086] For example, the memory (140) may store data regarding a hypervisor, a first virtual machine, a third virtual machine, or the like, for execution within the processor (175).

[0087] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0088] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0089] The signal processing device (170) controls the overall operation of each unit in the vehicle display device (100) or the vehicle (200).

[0090] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0091] The processor (175) can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0092] Among the first virtual machine to the third virtual machine (not shown), the first virtual machine (not shown) may be named a server virtual machine (Server Virtual Maschine), and the second virtual machine to the third virtual machine (not shown) may be named a guest virtual machine (Guest Virtual Maschine).

[0093] For example, a first virtual machine (not shown) within a processor (175) may receive, process, or output sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data.

[0094] In this way, by performing most of the data processing in the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0095] As another example, a first virtual machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for a second virtual machine or a third virtual machine (not shown).

[0096] And, the first virtual machine (not shown) can transmit processed data to the second virtual machine or the third virtual machine (not shown).

[0097] Accordingly, among the first virtual machine to the third virtual machine (not shown), only the first virtual machine (not shown) receives sensor data, communication data, or external input data from multiple sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtual machines, enabling 1:N data communication, and enabling synchronization when sharing data.

[0098] Meanwhile, the first virtual machine (not shown) can control the second virtual machine (not shown) and the third virtual machine (not shown) to share the same data by writing data to the shared memory (508).

[0099] For example, a first virtual machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in shared memory (508) and control the same data to be shared with a second virtual machine (not shown) and a third virtual machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0100] Ultimately, by performing most of the data processing on the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0101] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown).

[0102] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0103] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0104] Referring to FIG. 5, the server (900) may refer to a device that trains an artificial neural network using a machine learning algorithm or utilizes a trained artificial neural network. Here, the server (900) may be composed of multiple servers to perform distributed processing, and may be defined as a 5G network.

[0105] The server (900) may be included as part of the configuration of the vehicle (200) and may perform at least part of the AI ​​processing.

[0106] The server (900) may include a communication interface (510), memory (530), a running processor (540), and a processor (560).

[0107] The communication interface (510) can transmit and receive data with the vehicle (200) or an external device.

[0108] The memory (530) may include a model storage unit (931). The model storage unit (531) may store a model (or artificial neural network, 531a) being learned or learned through the learning processor (540).

[0109] The learning processor (540) can train an artificial neural network (531a) using learning data. The learning model can be used while mounted on the artificial neural network server (900), or can be mounted on an external device such as a vehicle (200).

[0110] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (530).

[0111] The processor (560) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0112] FIG. 6 is a drawing for explaining the operation of a system according to an embodiment of the present disclosure.

[0113] Referring to FIG. 6, the system (60) may include a server (900), an artificial intelligence device (600), and an ECU (770).

[0114] The server (900) may include a preprocessing unit (910), a learning unit (920), a model unit (930), and a model distribution unit (940).

[0115] The preprocessing unit (910) can preprocess driving-related status data received from the artificial intelligence device (600).

[0116] The learning unit (920) can learn a Large Language Model (LLM) using a training data set including preprocessed driving-related status data and labeled data labeled with the driving-related status data. The LLM may be a model that infers one or more causes of AEB control failure or AEB control logic from the driving-related status data.

[0117] As another example, the LLM may be a model that outputs the braking timing of the brake, the braking force of the brake, the status of the engine intake valve, and a warning message from the sensing data collected from the artificial intelligence device (600).

[0118] The model unit (930) can store the LLM for which learning or updating has been completed. The model unit (930) can be included in the model storage unit (531) of FIG. 5.

[0119] The model distribution unit (940) can transmit a plurality of learned embedding vectors matching the LLM and the causes of failure of multiple AEB controls, for which learning or updating has been completed, to the edge device. The edge device may be an artificial intelligence device (600). The model distribution unit (940) may be included in the communication interface (510) of FIG. 5.

[0120] The preprocessing unit (910) and the learning unit (920) may be included in the learning processor (540) or processor (560) of FIG. 5.

[0121] The artificial intelligence device (600) may include a data expansion unit (610) and a data inference unit (620).

[0122] The data expansion unit (610) may include an LLM (Large Language Model) agent (611), a vector conversion unit (612), and a vector storage unit (613).

[0123] The LLM (Large Language Model) agent (611) can determine the suitability of the data inference unit (620). The data inference unit (620) may be referred to as LLM.

[0124] The LLM agent (611) can generate ECU control logic corresponding to the cause of AEB failure. The LLM agent (611) can generate a control command including the ECU control logic and transmit the generated control command to the ECU (770).

[0125] The vector conversion unit (612) can convert driving-related status data into an embedding vector and place the converted embedding vector on the embedding space.

[0126] The vector storage unit (613) may store a plurality of learning embedding vectors corresponding to a plurality of causes of failure of AEB control.

[0127] The data inference unit (620) may be an LLM that infers the cause of failure of AEB control from driving-related status data.

[0128] The data inference unit (620) can compare the embedding vector converted by the vector conversion unit (612) with a plurality of learning embedding vectors stored in the vector storage unit (613). As a result of the comparison, the data inference unit (620) can obtain a learning embedding vector that is most similar to the converted embedding vector among the plurality of learning embedding vectors. The data inference unit (620) can output the cause of failure of the AEB control that matches the obtained learning embedding vector.

[0129] The LLM agent (611) can obtain ECU control logic corresponding to the cause of failure of AEB control inferred by the data inference unit (620) and transmit a control command corresponding to the obtained ECU control logic to the ECU (770). The LLM agent (611) can transmit the control command to the ECU (770) via Ethernet communication.

[0130] The artificial intelligence device (600) may be the signal processing device (170) of FIG. 1. In this case, the data expansion unit (610) and the data inference unit (620) may be included in the signal processing device (170). In particular, the data expansion unit (610) and the data inference unit (620) may be included in the processor (175) of the signal processing device (170). The artificial intelligence device (600) may further include a communication device (120) and a plurality of communication modules (EMa to EMd).

[0131] The artificial intelligence device (600) may be the vehicle display device (100) of FIG. 1.

[0132] The artificial intelligence device (600) may be the vehicle (200) of FIG. 1.

[0133] The artificial intelligence device (600) may be a cockpit domain controller (CDC). The CDC may be an electronic control unit for a vehicle that integrates and manages multiple electronic systems within the vehicle. The CDC may integrate and manage the instrument panel, infotainment system, driver assistance system, and the like. The CDC may be included in the vehicle (200) or a signal processing device (170a).

[0134] FIG. 7 is a sequence diagram for explaining a fail-safe response method based on LLM of a system according to one embodiment of the present disclosure.

[0135] Hereinafter, the artificial intelligence device (600) may include all components of the vehicle (200). In particular, the artificial intelligence device (600) may include a memory (140, 952), a communication device (120), and a processor (175) of the vehicle (200). The communication device (120) may be referred to as a communication interface (120). The processor (175) may be provided in at least one number.

[0136] The processor (175) of the artificial intelligence device (600) can obtain a safety factor according to the execution of Autonomous Emergency Braking (AEB) control (S701).

[0137] The safety margin may be the percentage of times an AEB system can successfully prevent a collision.

[0138] In one embodiment, the processor (175) can obtain a safety factor according to the execution of rule-based AEB control. The processor (175) can obtain the success rate of AEB control performed according to predefined rules or conditions stored in the memory (140) as the safety factor.

[0139] In another embodiment, the processor (175) can obtain a safety factor according to the execution of AEB control based on an artificial intelligence model. The processor (175) can obtain a success rate of AEB control according to the AEB control logic output by the artificial intelligence model stored in the memory (140) as a safety factor.

[0140] The processor (175) of the artificial intelligence device (600) can determine whether the acquired safety rate is less than the reference safety rate (S703).

[0141] In one embodiment, the reference safety factor may be 95%, but this is only an example.

[0142] The processor (175) of the artificial intelligence device (600) can collect driving-related status data based on the safety rate being less than the reference safety rate (S705).

[0143] In one embodiment, the driving-related status data may include at least one of ECU information, aging information of the vehicle (200), or surrounding information.

[0144] ECU information may include at least one of sensing information of a camera (195) of a vehicle (200), sensing information of a lidar (196), sensing information of a radar (197), control information of a steering wheel (150), control information of an accelerator, control information of a brake pad, or control information of a gear. ECU information may further include at least one of a vehicle component failure, an engine status, a coolant status, RPM, and a battery status.

[0145] Control information of the steering wheel (150) may include at least one of a steering angle or a steering torque.

[0146] The control information of the brake pad may include at least one of the time the brake pad is pressed or the pressure (pressure) applied to the brake pad.

[0147] The control information of the Excel may include at least one of the engine rotation speed, the speed of the vehicle (200), the engine ignition time, or the engine fuel inlet opening time.

[0148] The gear control information may include information about gear shifting.

[0149] The aging information of the vehicle (200) may include the degree of aging of at least one of the engine, brake pads, accelerator, gear, or steering wheel (150).

[0150] The ambient information may include at least one of weather information, temperature, humidity, or precipitation.

[0151] The processor (175) of the artificial intelligence device (600) can transmit driving-related status data collected through the communication device (120) to the server (900) (S707).

[0152] The processor (175) can transmit driving-related status data to the server (900) via CAN communication. The server (900) can collect driving-related status data from the artificial intelligence device (600) and other vehicles.

[0153] The processor (560) of the server (900) can learn LLM based on driving-related status data received from the artificial intelligence device (600) (S709).

[0154] In one embodiment, the LLM may be a model that infers one or more causes of failure of AEB control or AEB control logic from driving-related state data.

[0155] The learning processor (540) or processor (560) of the server (900) can learn the LLM using driving-related status data. The LLM can be learned through supervised learning. The learning processor (540) or processor (560) of the server (900) can also receive driving-related status data through devices other than the artificial intelligence device (600).

[0156] FIG. 8 is a diagram illustrating a learning process of LLM according to an embodiment of the present disclosure.

[0157] Referring to FIG. 8, the training data set used for supervised learning of the LLM may include training driving-related state data and labeling data indicating causes of AEB control failure corresponding to the driving-related state data. The training driving-related state data may be data received from the artificial intelligence device (600) and other vehicles.

[0158] The running processor (540) or processor (560) of the server (900) can preprocess driving-related status data and convert the preprocessed information into an embedding vector.

[0159] The running processor (540) or processor (560) of the server (900) can train the LLM model by assigning label data indicating the cause of AEB failure to the converted embedding vector.

[0160] The reasons for the failure of the labeled AEB can be any of the following: failure to recognize a forward obstacle, error in calculating the time to collision (TTC) due to weather or road conditions, sensor error, or aging of the brakes.

[0161] The cause of AEB failure being labeled may be either a TTC over-calculation error or a TTC under-calculation error.

[0162] The learning processor (540) or processor (560) of the server (900) can train the LLM to minimize a loss function representing the difference between the inference result inferred from the training driving-related state data and the correct cause of failure of AEB control.

[0163] The running processor (540) or processor (560) of the server (900) can convert driving-related state data for training into an embedding vector, and can place the converted embedding vector in the embedding space by matching it with the cause of failure of AEB control.

[0164] The running processor (540) or processor (560) of the server (900) can update an existing LLM using driving-related status data received from the artificial intelligence device (600). An LLM for which learning has been completed may refer to an LLM that has been updated with new data.

[0165] Again, Figure 7 is explained.

[0166] The server (900) can transmit the LLM for which learning has been completed to the artificial intelligence device (600) through the communication interface (510) (S711).

[0167] The server (900) can transmit parameters, weights, code or libraries for executing the LLM, etc. of the LLM for which learning has been completed, to the artificial intelligence device (600) through the communication interface (510). Transmitting the LLM may mean transmitting parameters, weights, code or libraries for executing the LLM, etc. of the LLM.

[0168] The server (900) can additionally transmit learning embedding vectors matching the cause of failure of AEB control placed in the embedding space to the artificial intelligence device (600) through the communication interface (510).

[0169] The processor (175) of the artificial intelligence device (600) can store the LLM and learning embedding vectors received from the server (900) in the memory (140, 952).

[0170] The processor (175) of the artificial intelligence device (600) can obtain new driving-related status data (S713) and infer the cause of failure of AEB control from the new driving-related status data obtained through LLM (S715).

[0171] The driving-related status data may include at least one of the ECU information described in step S705, aging information of the vehicle (200), or surrounding information.

[0172] The processor (175) can obtain the cause of failure of AEB control from new driving-related status data through the LLM stored in the memory (140, 952).

[0173] The processor (175) can convert new driving-related data into an embedding vector. The processor (175) can measure the similarity between the converted embedding vector and the learning embedding vectors stored in the memory (140, 952). The processor (175) can obtain the cause of AEB control failure that matches the learning embedding vector with the highest similarity to the converted embedding vector.

[0174] The processor (175) of the artificial intelligence device (600) can obtain ECU control logic corresponding to the cause of failure of the inferred AEB control (S717) and transmit the obtained ECU control logic to the ECU (S719).

[0175] The processor (175) can transmit ECU control logic to the ECU (770) via Ethernet communication.

[0176] A single ECU control logic can be matched to a cause of failure in AEB control. The memory (140, 952) can store multiple ECU control logics corresponding to each of multiple failure causes.

[0177] The memory (140, 952) may store a table matching multiple failure causes and multiple ECU control logics corresponding to each of the multiple failure causes.

[0178] The processor (175) can extract ECU control logic corresponding to the cause of failure of AEB control inferred from the table stored in the memory (140, 952).

[0179] The processor (175) can transmit the extracted ECU control logic to the ECU (770), and the ECU (770) can control the brake drive unit (753) according to the received ECU control logic.

[0180] According to an embodiment of the present disclosure, the possibility of a vehicle collision can be prevented in advance by analyzing the cause of failure of AEB control in real time and controlling the ECU in response to the cause of failure.

[0181] Meanwhile, even after embedding the LLM received from the server (900), there may be cases where the inference of the cause of failure according to the AEB control is incorrect. The artificial intelligence device (600) can transmit the cause of the correct AEB control failure and the corresponding driving-related status data to the server (900), and the server (900) can relearn the LLM using the cause of the correct AEB control failure and the corresponding driving-related status data. The server (900) can transmit the relearned LLM to the artificial intelligence device (600), and the artificial intelligence device (600) can infer the cause of the AEB control failure using the relearned LLM.

[0182] FIG. 9 is a diagram illustrating a table matching the cause of failure of AEB control and ECU control logic according to one embodiment of the present disclosure.

[0183] Referring to FIG. 9, a table (901) is illustrated that includes multiple failure causes and multiple control logics corresponding to each of the multiple failure causes.

[0184] For example, the first cause of failure may be an overestimation of the TTC. The first ECU control logic corresponding to the first cause of failure may be to advance the braking timing of the brakes and increase the braking force (or braking force value) of the brakes to the maximum.

[0185] Table (901) may further include detailed AEB control failure causes of the first failure cause. The detailed AEB control failure cause may be a sensing error of the rider (196).

[0186] For example, the second cause of failure may be an undercalculation of the TTC. The second ECU control logic corresponding to the second cause of failure may delay the braking timing of the brakes and gradually increase the braking force of the brakes. Table (901) may further include detailed causes of failure of the AEB control for the second cause of failure. The detailed cause of failure of the AEB control may be a sensing error of the camera (195).

[0187] Another example of AEB control failure could be a weakened Parker connector on the intake valve. A weakened Parker connector can cause the signal interruption interval to increase over time. The ECU control logic to respond to this could be logic that lowers the RPM (vehicle speed) relative to the accelerator pedal pressure.

[0188] Another example is that AEB control failure could be caused by brake malfunction due to pad wear under the brakes. The ECU control logic to address this could be to gradually reduce RPM.

[0189] Meanwhile, the processor (175) of the artificial intelligence device (600) can display the inferred cause of failure of AEB control on at least one of the first to third displays (180a, 180b, 180c) or transmit it to the mobile terminal (800).

[0190] The processor (175) can display a warning message including the cause of the inferred AEB control failure and a warning phrase on at least one of the first to third displays (180a, 180b, 180c) or transmit it to the mobile terminal (800).

[0191] FIG. 10A and FIG. 10B are diagrams illustrating a warning message including a cause of failure of AEB control according to an embodiment of the present disclosure.

[0192] Referring to FIG. 10a, the first warning message (1010) may be a message corresponding to the first failure cause of FIG. 9. The first warning message (1010) may include the first failure cause and the first ECU control logic corresponding to the first failure cause.

[0193] Referring to FIG. 10b, the second warning message (1020) may be a message corresponding to the second failure cause of FIG. 9. The second warning message (1020) may include the second failure cause and the second ECU control logic corresponding to the second failure cause.

[0194] The first warning message (1010) or the second warning message (1020) may be displayed on the first to third displays (180a, 180b, 180c) of the vehicle (200). Through this, the user may be provided with information on the cause of the AEB control failure and may attempt to take appropriate action.

[0195] FIG. 11 is a diagram illustrating a frame structure of a CAN (Controller Area Network) message transmitted to a server by an artificial intelligence device according to an embodiment of the present disclosure, and FIG. 12 is a diagram illustrating modes related to data processed by an ECU according to an embodiment of the present disclosure.

[0196] The CAN message (1100) of FIG. 11 and the data bytes of FIG. 12 may follow the OBD-II (On-Board Diagnostics-II) standard, which is a standard protocol for obtaining information from the ECU (770) of the vehicle (200).

[0197] Referring to FIG. 11, a CAN message (1100) may include a Header 1 field, a Header 2 field, a Header 3 field, a Data 1 field to a Data 7 field, and a CRC (Cyclic Redundancy Check) field.

[0198] Each of the Header 1 field, the Header 2 field, and the Header 3 field may include metadata of the CAN message (1100). The metadata may include at least one of the type of the CAN message (1100), the identifier of the artificial intelligence device (600), the identifier of the CAN message (1100), or the status of the CAN message (1100).

[0199] Each of the Data 1 field to the Data 7 field may include data collected from sensors equipped in the vehicle (200) or from the ECU (770). The Data 2 field may be a field indicating a PID (Parameter ID) and may be a field identifying a parameter (e.g., speed, RPM, etc.) so that a user can check data received from the ECU (770).

[0200] Each of the Data 3 field to the Data 7 field may be a field containing ECU information. The ECU information may include at least one of vehicle component failure status, engine status, coolant status, RPM, and battery status.

[0201] The CRC field may be a field for verifying the integrity of data. The server (900) can determine whether data is damaged during the transmission of the CAN message (1100) through the CRC field.

[0202] Each mode illustrated in FIG. 12 may represent a function of requesting specific data collected by the ECU (770) when the artificial intelligence device (600) or server (900) communicates with the ECU (770).

[0203] FIG. 13 is a flowchart illustrating an operation method of an artificial intelligence device according to an embodiment of the present disclosure.

[0204] Hereinafter, the artificial intelligence device (600) may include all components of the vehicle (200). In particular, the artificial intelligence device (600) may include a memory (140, 952), a communication device (120), and a processor (175) of the vehicle (200).

[0205] The processor (175) may be provided in at least one number.

[0206] Below, detailed descriptions of the overlapping contents with the steps of Fig. 5 are borrowed from the description of Fig. 5.

[0207] The processor (175) of the artificial intelligence device (600) can obtain a safety factor according to the execution of Autonomous Emergency Braking (AEB) control (S1301).

[0208] The processor (175) of the artificial intelligence device (600) can determine whether the acquired safety rate is less than the reference safety rate (S1303).

[0209] The processor (175) of the artificial intelligence device (600) can collect driving-related status data based on the safety rate being less than the reference safety rate (S1305).

[0210] The processor (175) of the artificial intelligence device (600) can learn LLM based on driving-related status data received from the artificial intelligence device (600) (S1307).

[0211] The processor (175) can learn the LLM using driving-related status data. The LLM can be learned through supervised learning. The processor (175) can receive driving-related status data from devices other than the artificial intelligence device (600).

[0212] The processor (175) can learn an LLM in the manner illustrated in FIG. 8 and embed the learned LLM in the artificial intelligence device (600). The learned LLM or the updated LLM can be stored in the memory (140, 952).

[0213] The processor (175) of the artificial intelligence device (600) can obtain new driving-related status data (S1309) and infer the cause of failure of AEB control from the new driving-related status data obtained through LLM (S1311).

[0214] The processor (175) of the artificial intelligence device (600) can obtain ECU control logic corresponding to the cause of failure of the inferred AEB control (S1313) and transmit the obtained ECU control logic to the ECU (S1315).

[0215] In this way, according to the embodiment of FIG. 13, the artificial intelligence device (600) can also directly perform collection of a training data set for learning of LLM and learning of LLM.

[0216] An artificial intelligence device (600) according to an embodiment of the present disclosure may include a communication interface (120) for communicating with a server (900); a memory (140, 952); and one or more processors (175) for transmitting driving-related state data to the server based on a safety factor according to execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety factor, receiving a Large Language Model (LLM) learned based on the driving-related state data from the server and storing it in the memory, obtaining new driving-related state data, inferring a cause of failure of the AEB control from the new driving-related state data through the LLM, obtaining a control logic of an electronic control unit (ECU) corresponding to the inferred cause of failure of the AEB control, and transmitting the obtained ECU control logic to the ECU.

[0217] If the cause of failure inferred by the LLM is incorrect, the one or more processors (175) can transmit the cause of failure of correct AEB control and corresponding driving-related status data to the server through the communication interface, receive a re-learned LLM from the server, and store the re-learned LLM in the memory.

[0218] The cause of the failure of the above AEB control is a calculation error of the time to collision (TTC), and the control logic of the ECU may be the braking timing and braking force value of the brake.

[0219] The artificial intelligence device (600) may further include one or more displays (180a to 180c), and the one or more processors (175) may display a warning message including the cause of the failure of the AEB control on the one or more displays.

[0220] The above memory (140, 952) can store a plurality of learning embedding vectors corresponding to each of the causes of failure of a plurality of AEB controls received from the server.

[0221] The above LLM may be a model that converts the new driving-related state data into an embedding vector, obtains a learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and infers a cause of failure of AEB control corresponding to the obtained learning embedding vector.

[0222] The above memory (140, 952) can store a table including the causes of failure of the plurality of AEB controls and the plurality of ECU control logics matched to the causes of failure of the plurality of AEB controls.

[0223] The one or more processors (175) can obtain the ECU control logic matching the cause of the failure of the AEB control inferred from the table.

[0224] The above driving-related status data includes at least one of ECU information, vehicle aging information, or surrounding information, and the ECU information may include at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information.

[0225] The above artificial intelligence device (600) may be a CDC (Cockpit domain controller).

[0226] The above-described present disclosure can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of the computer-readable medium include a hard disk drive (HDD), a solid-state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. In addition, the computer may include a processor (175).

[0227] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In artificial intelligence devices, Communication interface for communicating with the server; memory; and A system comprising: a system for transmitting driving-related state data to a server based on a safety factor of execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety factor; receiving a Large Language Model (LLM) learned based on the driving-related state data from the server and storing it in the memory; obtaining new driving-related state data; inferring a cause of failure of the AEB control from the new driving-related state data through the LLM; obtaining a control logic of an electronic control unit (ECU) corresponding to the inferred cause of failure of the AEB control; and transmitting the obtained ECU control logic to the ECU. Artificial intelligence device.

2. In paragraph 1, One or more of the above processors If the cause of failure inferred by the LLM is incorrect, the cause of failure of correct AEB control and the corresponding driving-related status data are transmitted to the server through the communication interface, the re-learned LLM is received from the server, and the re-learned LLM is stored in the memory. Artificial intelligence device.

3. In paragraph 1, The cause of the above AEB control failure is It is a calculation error in the Time To Collision (TTC), The control logic of the above ECU is The braking point and braking force value of the brake Artificial intelligence device.

4. In paragraph 1, Including one or more displays, One or more of the above processors Displaying a warning message including the cause of failure of the AEB control on the one or more displays; Artificial intelligence device.

5. In paragraph 1, The above memory is Store a plurality of learning embedding vectors corresponding to each of the causes of failure of multiple AEB controls received from the above server. Artificial intelligence device.

6. In paragraph 5, The above LLM is A model that converts the above new driving-related status data into an embedding vector, obtains a learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and infers the cause of failure of AEB control corresponding to the obtained learning embedding vector. Artificial intelligence device.

7. In paragraph 6, The above memory is A table including the causes of failure of the plurality of AEB controls and the plurality of ECU control logics matching the causes of failure of the plurality of AEB controls is stored. Artificial intelligence device.

8. In paragraph 7, One or more of the above processors Obtaining the ECU control logic matching the cause of failure of the AEB control inferred from the above table. Artificial intelligence device.

9. In paragraph 1, The above driving related status data is Contains at least one of ECU information, vehicle aging information, or surrounding information; The above ECU information is Contains at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information. Artificial intelligence device.

10. In paragraph 1, The above artificial intelligence device CDC (Cockpit domain controller) Artificial intelligence device.

11. A computer-readable recording medium recording a program for performing an operation method of an artificial intelligence device, The above operation method A step of transmitting driving-related status data to a server based on the safety rate resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety rate; A step of receiving and storing a large language model (LLM) learned based on the driving-related status data from the server; Step of acquiring new driving-related status data; A step of inferring the cause of failure of the AEB control from the new driving-related status data through the LLM; A step of obtaining the control logic of an electronic control unit (ECU) corresponding to the cause of the failure of the inferred AEB control; and A step of transmitting the acquired ECU control logic to the ECU is included. Recording medium.

12. In paragraph 11, The above operation method If the cause of failure inferred by the LLM is incorrect, a step of transmitting the cause of failure of the correct AEB control and the corresponding driving-related status data to the server; and Further comprising a step of receiving the re-learned LLM from the above server and storing the re-learned LLM. Recording medium.

13. In paragraph 11, The cause of the above AEB control failure is It is a calculation error in the Time To Collision (TTC), The control logic of the above ECU is The braking point and braking force value of the brake Recording medium.

14. In paragraph 11, The above operation method Further comprising a step of displaying a warning message including the cause of the failure of the AEB control. Recording medium.

15. In paragraph 11, The above operation method Further comprising a step of storing a plurality of learning embedding vectors corresponding to each of the causes of failure of a plurality of AEB controls received from the server. Recording medium.

16. In paragraph 15, The above LLM is A model that converts the above new driving-related status data into an embedding vector, obtains a learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and infers the cause of failure of AEB control corresponding to the obtained learning embedding vector. Recording medium.

17. In paragraph 16, The above operation method Further comprising a step of storing a table including the causes of failure of the plurality of AEB controls and the plurality of ECU control logics matching the causes of failure of the plurality of AEB controls. Recording medium.

18. In paragraph 17, The step of obtaining the above ECU control logic is A step of obtaining the ECU control logic matching the cause of failure of the AEB control inferred from the table. Recording medium.

19. In paragraph 11, The above driving related status data is Contains at least one of ECU information, vehicle aging information, or surrounding information; The above ECU information is Contains at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information. Recording medium.

20. In the method of operating an artificial intelligence device, A step of transmitting driving-related status data to a server based on the safety rate resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control being less than a reference safety rate; A step of receiving and storing a large language model (LLM) learned based on the driving-related status data from the server; Step of acquiring new driving-related status data; A step of inferring the cause of failure of the AEB control from the new driving-related status data through the LLM; A step of obtaining the control logic of an electronic control unit (ECU) corresponding to the cause of the failure of the inferred AEB control; and A step of transmitting the acquired ECU control logic to the ECU is included. How an artificial intelligence device operates.

Images