Signal processing apparatus and vehicle control apparatus including same

Abstract

A signal processing apparatus and a vehicle control apparatus including same, according to an embodiment of the present disclosure, comprise a processor that receives and processes internal images from an internal camera, wherein the processor calculates a driver's forward gaze level on the basis of the internal images, calculates forward collision time or forward collision avoidance time with respect to an object in front a vehicle, and varies the timing of controlling forward collision avoidance on the basis of the forward collision time or the forward collision avoidance time, and the forward gaze level. Accordingly, adaptive vehicle control can be performed according to the driver's forward gaze level.

Description

Signal processing device and vehicle control device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle control device having the same, and more particularly, to a signal processing device capable of performing adaptive vehicle control according to a driver's forward gaze level and a vehicle control device having the same.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, for the convenience of vehicle users, a vehicle signal processing device is installed inside the vehicle.

[0004] The signal processing device inside the vehicle receives and processes sensor data from various sensor devices inside the vehicle due to advanced driver assistance systems (ADAS) or autonomous driving.

[0005] Prior art document 1, European Patent No. EP3643586, relates to a driver assistance system, which discloses outputting a steering signal to the steering device of a vehicle when a collision with a front object of the vehicle is expected and a side object is not detected.

[0006] However, according to prior art document 1, since a steering signal is output to the steering device of the vehicle regardless of the state of the vehicle driver, there is a disadvantage in that adaptive control corresponding to the state of the vehicle driver cannot be performed.

[0007] Prior art document 2, U.S. Patent No. US11840220, relates to a driver assistance system and a control method thereof, and discloses controlling at least one of a steering system or a speed control system of a vehicle to avoid a rear-end vehicle when there is a risk of collision with a rear-end vehicle.

[0008] However, according to prior art 2, since control is performed to avoid the rear vehicle regardless of the state of the vehicle driver, there is a disadvantage in that adaptive control corresponding to the state of the vehicle driver is not performed.

[0009] The problem to be solved by the present disclosure is to provide a signal processing device capable of performing adaptive vehicle control according to a driver's forward gaze level and a vehicle control device having the same.

[0010] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of adaptively outputting a warning message according to a driver's forward gaze level and a vehicle control device having the same.

[0011] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of efficiently performing vehicle control using microservices and a vehicle control device having the same.

[0012] A signal processing device and a vehicle control device including the same according to one embodiment of the present disclosure for solving the above technical problem include a processor for receiving and processing an internal image from an internal camera, and the processor calculates a driver's forward gaze level based on the internal image, calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle, and varies a forward collision avoidance control time based on the forward collision time or forward collision avoidance time and the forward gaze level.

[0013] Meanwhile, the processor can receive a forward image from the front camera and, based on the forward image, calculate a forward collision time or forward collision avoidance time with an object in front of the vehicle.

[0014] Meanwhile, the processor may set the forward collision avoidance control time point to the first time point when the forward gaze level is the first level, and may set the forward collision avoidance control time point to the second time point before the first time point when the forward gaze level is the second level lower than the first level.

[0015] Meanwhile, the processor may set the forward collision avoidance control time point to the third time point between the first time point and the second time point when the forward gaze level is the third level between the first level and the second level.

[0016] Meanwhile, the processor can control the forward collision avoidance control timing to be delayed as the forward gaze level increases.

[0017] Meanwhile, the processor can vary the output timing of a warning message for forward collision avoidance in response to the forward gaze level.

[0018] Meanwhile, the processor may set the output time of a warning message for forward collision avoidance to the first output time when the forward gaze level is the first level, and may set the output time of the warning message for forward collision avoidance to the second output time before the first output time when the forward gaze level is the second level lower than the first level.

[0019] Meanwhile, the processor can control the output timing of a warning message for forward collision avoidance to be delayed as the forward gaze level increases.

[0020] Meanwhile, the processor can vary the steering angle change level for forward collision avoidance in response to the forward gaze level.

[0021] Meanwhile, the processor may set the steering angle change level for forward collision avoidance to a first set level when the forward gaze level is a first level, and may set the steering angle change level for forward collision avoidance to a second set level lower than the first set level when the forward gaze level is a second level lower than the first level.

[0022] Meanwhile, the processor can control the steering angle change level for forward collision avoidance to increase as the forward gaze level increases.

[0023] Meanwhile, the processor can execute a driver monitoring application based on the internal image and an automatic emergency steering control application based on the forward image.

[0024] Meanwhile, the processor executes a warning application for forward collision avoidance based on the internal image and the front image, and the safety level of the warning application may be lower than the safety level of the driver monitoring application or the automatic emergency steering control application.

[0025] Meanwhile, the processor may run multiple virtual machines on the hypervisor, and some of the multiple virtual machines may run a driver monitoring application based on the internal image and an automatic emergency steering control application based on the forward image.

[0026] Meanwhile, some other virtual machines among the multiple virtual machines run a warning application for forward collision avoidance, and the safety level of the warning application may be lower than the safety level of the driver monitoring application or the automatic emergency steering control application.

[0027] Meanwhile, some of the virtual machines among the plurality of virtual machines may run multiple microservices for an automatic emergency steering control application based on the forward image, and run multiple microservices for a driver monitoring application based on the internal image.

[0028] A signal processing device and a vehicle control device having the same according to another embodiment of the present disclosure include a processor that receives and processes an interior image from an interior camera and a front image from a front camera, and the processor calculates a driver's forward gaze level based on the interior image, calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle based on the forward collision time or forward collision avoidance time and the forward gaze level, and varies a forward collision avoidance control time point or a steering angle change level for forward collision avoidance.

[0029] A signal processing device and a vehicle control device including the same according to one embodiment of the present disclosure include a processor that receives and processes an interior image from an interior camera, and the processor calculates a driver's forward gaze level based on the interior image, calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle, and varies a forward collision avoidance control timing based on the forward collision time or forward collision avoidance time and the forward gaze level. Accordingly, adaptive vehicle control can be performed based on the driver's forward gaze level.

[0030] Meanwhile, the processor can receive a forward image from the front camera and, based on the forward image, calculate the forward collision time or forward collision avoidance time with an object ahead of the vehicle. This enables adaptive vehicle control based on the driver's forward gaze level.

[0031] Meanwhile, the processor may set the forward collision avoidance control timing to the first timing when the forward gaze level is the first level, and may set the forward collision avoidance control timing to the second timing, which is prior to the first timing, when the forward gaze level is the second level, which is lower than the first level. Accordingly, adaptive vehicle control can be performed depending on the driver's forward gaze level.

[0032] Meanwhile, the processor can set the forward collision avoidance control timing to a third time point, which is between the first and second time points, when the forward gaze level is a third time point, which is between the first and second time points. Accordingly, adaptive vehicle control can be performed according to the driver's forward gaze level.

[0033] Meanwhile, the processor can control the timing of forward collision avoidance control to be delayed as the forward gaze level increases. This enables adaptive vehicle control based on the driver's forward gaze level.

[0034] Meanwhile, the processor can vary the timing of the warning message output for forward collision avoidance based on the driver's forward attention level. This allows adaptive vehicle control to be performed based on the driver's forward attention level.

[0035] Meanwhile, the processor may set the output time of the warning message for forward collision avoidance to the first output time when the forward gaze level is the first level, and may set the output time of the warning message for forward collision avoidance to the second output time before the first output time when the forward gaze level is the second level lower than the first level. Accordingly, it is possible to output the warning message adaptively depending on the driver's forward gaze level.

[0036] Meanwhile, the processor can control the timing of the warning message output for forward collision avoidance so that the higher the forward attention level, the later it is output. Accordingly, the warning message can be output adaptively based on the driver's forward attention level.

[0037] Meanwhile, the processor can vary the steering angle change level for forward collision avoidance in response to the driver's forward gaze level. This enables adaptive vehicle control based on the driver's forward gaze level.

[0038] Meanwhile, the processor may set the steering angle change level for forward collision avoidance to the first preset level when the forward gaze level is the first level, and may set the steering angle change level for forward collision avoidance to the second preset level lower than the first level when the forward gaze level is the second level lower than the first level. Accordingly, adaptive vehicle control can be performed according to the driver's forward gaze level.

[0039] Meanwhile, the processor can control the steering angle change level for forward collision avoidance to increase as the forward gaze level increases. Accordingly, adaptive vehicle control can be performed based on the driver's forward gaze level.

[0040] Meanwhile, the processor can execute a driver monitoring application based on the internal images and an automatic emergency steering control application based on the forward images. This enables adaptive vehicle control based on the driver's forward attention level.

[0041] Meanwhile, the processor executes a warning application for forward collision avoidance based on the internal and forward images. The safety level of the warning application may be lower than that of the driver monitoring application or the automatic emergency steering control application. Accordingly, a warning message can be output adaptively based on the driver's forward attention level.

[0042] Meanwhile, the processor can run multiple virtual machines on the hypervisor, some of which can run a driver monitoring application based on internal images and an automatic emergency steering control application based on forward images. This enables adaptive vehicle control based on the driver's forward attention level.

[0043] Meanwhile, some of the other virtual machines run a warning application for forward collision avoidance. The safety level of the warning application may be lower than that of the driver monitoring application or the automatic emergency steering control application. This allows for adaptive warning messages to be output based on the driver's level of forward attention.

[0044] Meanwhile, some of the multiple virtual machines can run multiple microservices for the automatic emergency steering control application based on the forward image, and multiple microservices for the driver monitoring application based on the internal image. This allows for stable and efficient execution of the automatic emergency steering control application and the driver monitoring application. Furthermore, vehicle control can be efficiently performed using microservices.

[0045] A signal processing device and a vehicle control device including the same according to another embodiment of the present disclosure include a processor that receives and processes an interior image from an interior camera and a forward image from a front camera, and the processor calculates a driver's forward gaze level based on the interior image, calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle based on the forward collision time or forward collision avoidance time and the forward gaze level, and varies a forward collision avoidance control time point or a steering angle change level for forward collision avoidance. Accordingly, adaptive vehicle control can be performed according to the driver's forward gaze level.

[0046] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0047] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0048] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0049] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0050] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0051] Figures 5a to 5d are drawings showing various examples of vehicle control devices.

[0052] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0053] FIG. 7a is a drawing for reference in the description of a signal processing device related to the present disclosure.

[0054] FIG. 7b is a diagram illustrating an example of execution of a microservice according to an embodiment of the present disclosure.

[0055] FIG. 8 is an example of a signal processing system according to one embodiment of the present disclosure.

[0056] FIG. 9 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0057] FIG. 10A is a flowchart illustrating an operation method of a signal processing device according to one embodiment of the present disclosure.

[0058] FIG. 10b is a flowchart showing an operation method of a signal processing device according to another embodiment of the present disclosure.

[0059] Figures 11a to 15 are drawings referenced in the operation description of Figures 9 to 10b.

[0060] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0061] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0062] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0063] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0064] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0065] Meanwhile, the vehicle (200) may be equipped with multiple displays (180a, 180b) for displaying images, information, etc. inside.

[0066] In Fig. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) is also possible.

[0067] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be called a center information display.

[0068] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0069] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0070] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0071] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the plurality of zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the plurality of zones (Z1 to Z4).

[0072] Meanwhile, the signal processing device (170a) may further include, in addition to the vehicle communication gateway (GWDa), an autonomous driving control module (ACC), a cockpit control module (CPG), etc.

[0073] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be an HPC (High Performance Computing) gateway.

[0074] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) within a plurality of zones (Z1 to Z4).

[0075] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0076] Referring to the drawing, the interior of the vehicle may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a room mirror display (not shown), etc.

[0077] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0078] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0079] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0080] The signal processing device (170) has a processor (175) therein and can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0081] A second virtual machine (not shown) can operate for the first display (180a), and a third virtual machine (not shown) can operate for the second display (180b).

[0082] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0083] Meanwhile, the first virtual machine (not shown) within the processor (175) shares at least a portion of data with the second virtual machine (not shown) and the third virtual machine (not shown) for data sharing processing. Accordingly, data can be shared and processed among multiple virtual machines for multiple displays within the vehicle.

[0084] Meanwhile, a first virtual machine (not shown) within a processor (175) may receive and process vehicle wheel speed sensor data, and transmit the processed wheel speed sensor data to at least one of a second virtual machine (not shown) or a third virtual machine (not shown). Accordingly, the vehicle wheel speed sensor data may be shared with at least one virtual machine.

[0085] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information, or images.

[0086] The signal processing device (170) can control the RSE display (180c) by executing a fourth virtual machine (not shown) in addition to the first virtual machine to the third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0087] Accordingly, it is possible to control various displays (180a to 180c) using one signal processing device (170).

[0088] Meanwhile, some of the multiple displays (180a~180c) may operate under Linux OS, while others may operate under Web OS.

[0089] The signal processing device (170) according to the embodiment of the present disclosure can control the same information or the same image to be displayed in synchronization on displays (180a to 180c) operating under various operating systems (OS).

[0090] Meanwhile, in FIG. 3b, a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications and a vehicle speed indicator (212b) and a vehicle interior temperature indicator (213b) are displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) are displayed on a third display (180c).

[0091] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0092] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may include a lamp driving unit (751), a steering driving unit (752), a brake driving unit (753), a power source driving unit (754), a suspension driving unit (756), an air conditioning driving unit (757), a window driving unit (758), a seat driving unit (761), and a signal processing device (170).

[0093] Meanwhile, the vehicle (200) may further include an ECU (770), multiple sensor devices (SN), and multiple communication modules (EMa to EMd).

[0094] Meanwhile, a vehicle (200) according to an embodiment of the present disclosure may further include a vehicle display device (100).

[0095] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa to EMd) for internal communication, a memory (140), a signal processing unit (170), a plurality of displays (180a to 180c), an audio output unit (185), and a power supply unit (190).

[0096] A plurality of communication modules (EMa to EMd) can be arranged, for example, in a plurality of zones (Z1 to Z4) of FIG. 2, respectively.

[0097] Meanwhile, the signal processing device (170) may have a communication switch (736b) for data communication with each communication module (EM1 to EM4) inside.

[0098] Each communication module (EM1 to EM4) can perform data communication with multiple sensor devices (SN) or ECUs (770) or area signal processing devices (170Z).

[0099] Meanwhile, the plurality of sensor devices (SN) may include a camera (195), a lidar (196), a radar (197), or a position sensor (198).

[0100] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0101] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0102] The communication unit (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0103] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0104] The communication unit (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (900). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0105] A plurality of communication modules (EM1 to EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or an area signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0106] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0107] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0108] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0109] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can transmit location information data sensed by a GPS module or location sensor (198) to a signal processing device (170).

[0110] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0111] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0112] For example, the memory (140) may store data regarding a hypervisor, a first virtual machine, a third virtual machine, or the like, for execution within the processor (175).

[0113] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0114] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0115] The signal processing device (170) controls the overall operation of each unit in the vehicle display device (100) or the vehicle (200).

[0116] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0117] The processor (175) can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0118] Among the first virtual machine to the third virtual machine (not shown), the first virtual machine (not shown) may be named a server virtual machine (Server Virtual Maschine), and the second virtual machine to the third virtual machine (not shown) may be named a guest virtual machine (Guest Virtual Maschine).

[0119] For example, a first virtual machine (not shown) within a processor (175) may receive, process, or output sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data.

[0120] In this way, by performing most of the data processing in the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0121] As another example, a first virtual machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for a second virtual machine or a third virtual machine (not shown).

[0122] And, the first virtual machine (not shown) can transmit processed data to the second virtual machine or the third virtual machine (not shown).

[0123] Accordingly, among the first virtual machine to the third virtual machine (not shown), only the first virtual machine (not shown) receives sensor data, communication data, or external input data from multiple sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtual machines, enabling 1:N data communication, and enabling synchronization when sharing data.

[0124] Meanwhile, the first virtual machine (not shown) can control the second virtual machine (not shown) and the third virtual machine (not shown) to share the same data by writing data to the shared memory (508).

[0125] For example, a first virtual machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in shared memory (508) and control the same data to be shared with a second virtual machine (not shown) and a third virtual machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0126] Ultimately, by performing most of the data processing on the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0127] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown).

[0128] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0129] Meanwhile, the signal processing device (170) of FIG. 4 may be the same as the signal processing device (170, 170a1, 170a2) of the vehicle control device of FIG. 5a or lower.

[0130] Figures 5a to 5d are drawings showing various examples of vehicle control devices.

[0131] FIG. 5a illustrates an example of a vehicle control device according to an embodiment of the present disclosure.

[0132] Referring to the drawings, a vehicle control device (800a) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0133] Meanwhile, the vehicle control device (800a) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0134] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0135] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0136] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0137] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0138] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0139] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0140] For example, sensor data within a vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle exterior radar data, and vehicle exterior lidar data.

[0141] Meanwhile, camera data may include vehicle exterior camera data and vehicle interior camera data.

[0142] Meanwhile, the signal processing device (170a1, 170a2) can execute multiple virtual machines (820, 830, 840) based on safety standards.

[0143] In the drawing, it is illustrated that a processor (175) within a signal processing device (170a) executes a hypervisor (505) and, on the hypervisor (505), executes first to third virtual machines (820 to 840) according to an automotive safety integrity level (Automotive SIL; ASIL).

[0144] The first virtual machine (820) may be a virtual machine corresponding to Quality Management (QM), which is the lowest safety level in the Automotive Safety Integrity Level (ASIL) and is a non-enforceable grade.

[0145] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0146] The second virtual machine (820) may be a virtual machine corresponding to ASIL A or ASIL B, where the sum of severity, exposure, and controllability is 7 or 8 in the automotive safety integrity level (ASIL).

[0147] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0148] The third virtual machine (840) may be a virtual machine corresponding to ASIL C or ASIL D, in which the sum of severity, exposure, and controllability is 9 or 10 in the automotive safety integrity level (ASIL).

[0149] Meanwhile, ASIL D can correspond to the grade that requires the highest safety level.

[0150] The third virtual machine (840) can run a safety operating system (842) and an application (845) on the operating system (842).

[0151] Meanwhile, the third virtual machine (840) may also execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0152] Meanwhile, unlike the drawing, the third virtual machine (840) can also be executed through a separate core rather than the processor (175). This will be described later with reference to FIG. 5b.

[0153] FIG. 5b illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0154] Referring to the drawing, the vehicle control device (800b) according to the embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0155] Meanwhile, the vehicle control device (800b) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0156] The vehicle control device (800b) of FIG. 5b is similar to the vehicle control device (800a) of FIG. 5a, but the signal processing device (170a1) has some differences from the signal processing device (170a1) of FIG. 5a.

[0157] To describe the difference, the signal processing device (170a1) may include a processor (175) and a second processor (177).

[0158] The processor (175) within the signal processing unit (170a1) executes a hypervisor (505), and executes first and second virtual machines (820 to 830) on the hypervisor (505) according to the automotive safety integrity level (Automotive SIL; ASIL).

[0159] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0160] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0161] Meanwhile, the second processor (177) within the signal processing device (170a1) can execute a third virtual machine (840).

[0162] The third virtual machine (840) can execute a safety operating system (842), an auto-execution (845) on the operating system (842), and an application (845) on the auto-execution (845). That is, unlike FIG. 5A, an auto-execution (846) on the operating system (842) can be executed.

[0163] Meanwhile, the third virtual machine (840) may, similarly to FIG. 5a, execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0164] Meanwhile, the third virtual machine (840) requiring a high level of security is preferably executed on a second processor (177), which is a different core or different processor, unlike the first and second virtual machines (820 to 830).

[0165] Meanwhile, in the signal processing devices (170a1, 170a2) of FIGS. 5a and 5b, when the first signal processing device (170a) malfunctions, the second signal processing device (170a2), which is a backup device, can operate.

[0166] Alternatively, it is also possible for the signal processing devices (170a1, 170a2) to operate simultaneously, with the first signal processing device (170a) operating as the main device and the second signal processing device (170a2) operating as the sub device. This will be described with reference to FIGS. 5c and 5d.

[0167] FIG. 5c illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0168] Referring to the drawing, a vehicle control device (800c) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0169] Meanwhile, the vehicle control device (800c) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0170] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0171] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0172] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0173] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0174] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0175] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0176] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175) in the first signal processing device (170a1) executes a hypervisor (505) and can execute a safety virtualization machine (860) and a non-safety virtualization machine (870) on the hypervisor (505), respectively.

[0177] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175b) in the second signal processing device (170a2) executes the hypervisor (505b) and can execute only the safety virtualization machine (880) on the hypervisor (505).

[0178] In this way, since the processing for safety is separated between the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0179] Meanwhile, high-speed network communication can be performed between the first signal processing device (170a1) and the second signal processing device (170a2).

[0180] FIG. 5d illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0181] Referring to the drawing, a vehicle control device (800d) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0182] Meanwhile, the vehicle control device (800d) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0183] The vehicle control device (800d) of FIG. 5d is similar to the vehicle control device (800c) of FIG. 5c, but the second signal processing device (170a2) has some differences from the second signal processing device (170a2) of FIG. 5c.

[0184] The processor (175b) in the second signal processing device (170a2) of FIG. 5d executes a hypervisor (505b) and can execute a safety virtualization machine (880) and a non-safety virtualization machine (890) on the hypervisor (505).

[0185] That is, unlike FIG. 5c, the difference is that the processor (175b) within the second signal processing device (170a2) further executes a non-safety virtualization machine (890).

[0186] In this way, since the processing for safety and non-safety is separated into the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0187] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0188] Referring to the drawing, a vehicle control device (900) according to an embodiment of the present disclosure includes a signal processing device (170).

[0189] The vehicle control device (900) according to the embodiment of the present disclosure may further include at least one display.

[0190] Meanwhile, the vehicle control device (900) according to the embodiment of the present disclosure may further include a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), an ECU (770), or a plurality of sensor devices (SN) of FIG. 4.

[0191] Meanwhile, the vehicle control device (900) according to the embodiment of the present disclosure may further include a lamp driving unit (751), a suspension driving unit (756), an air conditioning driving unit (757), a window driving unit (758), a seat driving unit (761), or a plurality of communication modules (EMa to EMd) of FIG. 4.

[0192] In the drawing, at least one display is illustrated, a cluster display (180a) and an AVN display (180b).

[0193] Meanwhile, the vehicle control device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0194] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having multiple CPUs (175), GPUs (178), NPUs (179), etc., and may be called an HPC (High Performance Computing) signal processing device or a central signal processing device.

[0195] A plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) are connected by wired cables (CB1 to CB4).

[0196] Meanwhile, multiple area signal processing devices (170Z1 to 170Z4) can be connected to each other with wired cables (CBa to CBd).

[0197] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0198] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may be equipped with at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0199] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0200] Meanwhile, sensor data may be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data may be stored in a storage device (925) within the signal processing device (170).

[0201] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0202] In the drawing, it is exemplified that camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2), a third area signal processing device (170Z3), etc.

[0203] Meanwhile, since the data read speed or write speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the plurality of area signal processing devices (170Z1 to 170Z4) to the signal processing device (170), it is preferable that multi-path routing be performed so that a network bottleneck does not occur.

[0204] To this end, the signal processing device (170) according to the embodiment of the present disclosure can perform multi-path routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data from the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, the network configuration can be dynamically changed to transmit data.

[0205] Data communication between a plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) in a vehicle control device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication for high-bandwidth, low-latency communication.

[0206] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can receive an internal image from an internal camera (195i) and perform signal processing on the internal image.

[0207] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can receive a front image from a front camera (195a) and perform signal processing on the front image.

[0208] FIG. 7a is a drawing for reference in the description of a signal processing device related to the present disclosure.

[0209] Referring to the drawing, a signal processing device (170x) related to the present disclosure can execute an application (785) based on sensor data or camera data of a vehicle, and output result data of the application (785) through multiple passes.

[0210] According to this method, the result data of the application (785) is output only after the execution of the application (785) is completed, so inefficiency occurs and a significant amount of time is likely to be required until the execution of the application (785) is completed.

[0211] Accordingly, in this disclosure, a method for sharing intermediate result data of an application, etc., when the application is executed is proposed.

[0212] To this end, the signal processing device (170) according to the embodiment of the present disclosure divides the application into a plurality of micro services, and executes other micro services based on the results of the micro services, etc., thereby efficiently distributing the workload.

[0213] FIG. 7b is a diagram illustrating an example of execution of a microservice according to an embodiment of the present disclosure.

[0214] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure can execute an application (795) based on sensor data or camera data of a vehicle, etc.

[0215] At this time, the signal processing device (170) according to the embodiment of the present disclosure can execute a plurality of micro-services separately for the application (795).

[0216] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute applications or microservices by distinguishing them according to the safety level.

[0217] At this time, the signal processing device (170) according to the embodiment of the present disclosure transmits the result data of the transmitting application or microservice when the security level of the transmitting application or microservice is higher or equal to that of the receiving application or microservice.

[0218] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure prevents the transmission of result data of the transmission application or microservice when the security level of the transmission application or microservice is lower than that of the reception application or microservice.

[0219] In the drawing, based on input data, a first microservice (910) corresponding to the second safety level, ASIL D, is executed, and the result data of the first microservice (910) corresponding to ASIL D can be transmitted to a second microservice (920a) corresponding to the third safety level, QM, a third microservice (920b) corresponding to the first safety level, ASIL B, a fourth microservice (920c) corresponding to the first safety level, ASIL B, and a fifth microservice (920d) corresponding to the third safety level, ASIL D, respectively.

[0220] Since the security level of the first microservice (910) is higher than the security levels of the second microservice (920a), the third microservice (920b), and the fourth microservice (920c), transmission of the result data of the first microservice (910) becomes possible.

[0221] Meanwhile, since the safety level of the first microservice (910) is the same as the safety level of the fifth microservice (920d), transmission of the result data of the first microservice (910) becomes possible.

[0222] Next, the sixth microservice (930a) corresponding to the third safety level, QM, is executed based on the result data of the second microservice (920a), and the result data can be output through the first pass.

[0223] Meanwhile, the seventh microservice (930b) corresponding to the first safety level, ASIL B, is executed based on the result data of the third microservice (920b) and the result data of the fourth microservice (920c), and the result data can be output through the second pass.

[0224] Meanwhile, the 8th microservice (930c) corresponding to the second safety level, ASIL D, is executed based on the result data of the 5th microservice (920d), and the result data can be output through the third pass.

[0225] As shown in the drawing, in addition to outputting the result data of the application (795) through multiple passes, unlike FIG. 7a, by executing and processing the corresponding microservice through each pass within the signal processing device (170), the workload can be efficiently distributed, thereby enabling data processing to be performed efficiently.

[0226] FIG. 8 is an example of a signal processing system according to one embodiment of the present disclosure.

[0227] Referring to the drawing, a signal processing system (1000) according to one embodiment of the present disclosure may include a central signal processing device (170) and an area signal processing device (170z).

[0228] Meanwhile, a signal processing device (170) in a system (1000) according to one embodiment of the present disclosure has a plurality of processor cores (CR1 to CRn, MR).

[0229] Meanwhile, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) may correspond to processor cores in the central processor (CPU) of FIG. 6.

[0230] For example, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) may correspond to application processor cores within the central processor (CPU) of FIG. 6.

[0231] Meanwhile, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) operate based on a hypervisor (505), and the hypervisor (505) can execute multiple virtual machines (820 to 850).

[0232] Meanwhile, some of the other processor cores (CR1 to CRn, MR) may correspond to M cores or MCUs (micom units).

[0233] Meanwhile, some other processor cores (MR) among the plurality of processor cores (CR1 to CRn, MR) can execute an operating system (805a) corresponding to a second safety level such as ASIL D without executing a hypervisor (505), and execute a fourth virtual machine (840) on the operating system (805a).

[0234] Meanwhile, the fourth virtual machine (840) can execute an application corresponding to a second safety level, such as ASIL D, or a microservice (843) corresponding to an application corresponding to the second safety level. Accordingly, the application or microservice (843) corresponding to the second safety level can be stably performed.

[0235] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the first processor core (CR1) can execute a hypervisor (505), execute an operating system (805b) corresponding to a second safety level such as ASIL D on the hypervisor (505), and execute a first virtual machine (850) on the operating system (805b).

[0236] Meanwhile, the first virtual machine (850) can execute an application corresponding to a first safety level, such as ASIL B, or a microservice (853a, 853b) corresponding to an application corresponding to the first safety level. Accordingly, the application or microservice (853a, 853b) corresponding to the first safety level can be stably performed.

[0237] Meanwhile, unlike the drawing, the first processor core (CR1) among the multiple processor cores (CR1 to CRn, MR) may execute an operating system corresponding to the first safety level, such as ASIL B, on the hypervisor (505).

[0238] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the second processor core (CR2) and the third processor core (CR3) can execute a hypervisor (505), execute an operating system (805c) corresponding to a first safety level such as ASIL B on the hypervisor (505), and execute a second virtualization machine (850) on the operating system (805c).

[0239] Meanwhile, the second virtual machine (850) can execute a third application corresponding to a first safety level, such as ASIL B, or a microservice (833a to 833d) corresponding to the third application corresponding to the first safety level, on an operating system (805c) corresponding to the first safety level. Accordingly, the application or microservice (833a to 833d) corresponding to the first safety level can be stably performed.

[0240] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the remaining processor cores (CR4 to CRn) can execute a hypervisor (505), execute an operating system (805d) corresponding to a third safety level such as QM on the hypervisor (505), and execute a third virtualization machine (820) on the operating system (805d).

[0241] Meanwhile, the third virtual machine (820) can execute a fourth application corresponding to the third safety level, such as QM, or a microservice (823a to 823d) corresponding to the fourth application corresponding to the third safety level, on an operating system (805d) corresponding to a third safety level lower than the first safety level. Accordingly, the application or microservice (823a to 823d) corresponding to the third safety level can be stably performed.

[0242] Meanwhile, the area signal processing device (170z) may be equipped with a plurality of application processor cores (CRR1 to CRRm) and an M core (MRb) for executing applications of ASIL D corresponding to the second safety level, which is the highest safety level.

[0243] Meanwhile, among the plurality of processor cores (CRR1 to CRRm, MRb) within the area signal processing device (170z), some (RR1 to CRRm) may execute an operating system (806b) corresponding to a first safety level such as ASIL B, and may execute a virtual machine (830b) corresponding to the first safety level on the operating system (806a).

[0244] Meanwhile, a virtual machine (830b) corresponding to the first safety level can execute an application corresponding to the first safety level, such as ASIL B, or a microservice (830ba to 830bd) corresponding to the application corresponding to the first safety level. Accordingly, the application or microservice (830ba to 830bd) corresponding to the first safety level can be stably performed.

[0245] Meanwhile, among the plurality of processor cores (CRR1 to CRRm, MRb) within the area signal processing device (170z), another part (MRb) may execute an operating system (806a) corresponding to a second safety level such as ASIL D, and may execute a virtual machine (840b) corresponding to a second safety level such as ASIL D on the operating system (806a).

[0246] Meanwhile, a virtual machine (840b) corresponding to the second safety level can execute an application corresponding to the second safety level, such as ASIL D, or a microservice (843b) corresponding to the application corresponding to the second safety level. Accordingly, the application or microservice (843b) corresponding to the second safety level can be stably performed.

[0247] FIG. 9 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0248] Referring to the drawings, a signal processing device (170) in a signal processing system (1000) according to one embodiment of the present disclosure includes a central processor (175) and at least one neural processor (179a to 179c).

[0249] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may further include a graphics processor (178).

[0250] Meanwhile, the central processor (175) according to the embodiment of the present disclosure executes a hypervisor (505).

[0251] Meanwhile, a system (1100) driven by a signal processing device (170) according to an embodiment of the present disclosure executes multiple virtual machines (810, 830, 850) on a hypervisor (505).

[0252] Specifically, a central processor (175) in a signal processing device (170) according to an embodiment of the present disclosure executes a hypervisor (505) and executes a plurality of virtual machines (810, 830, 850) on the hypervisor (505).

[0253] Meanwhile, the central processor (175) in the signal processing device (170) according to the embodiment of the present disclosure executes an application for driving the vehicle.

[0254] Meanwhile, if the central processor (175) determines that the application has failed to operate, it controls a second application corresponding to the application to be executed in another central processor or another signal processing device, and varies the standard fallback guarantee time for the application's operation failure based on the safety level of the application.

[0255] Accordingly, applications for vehicle operation can be performed reliably. In particular, applications for vehicle operation can be performed reliably based on safety levels.

[0256] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a shared memory (508).

[0257] In the drawing, it is illustrated that a hypervisor (505) is executed on a central processor (175) and a shared memory (508) is executed within the hypervisor (505).

[0258] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may receive data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (196), and perform signal processing using at least one of a central processor (175), a graphic processor (178), and a plurality of neural processors (179a to 179c).

[0259] Meanwhile, the sensor device (700) can continuously output sensor data to the signal processing device (170) during vehicle operation.

[0260] The sensor data at this time is data from sensor devices (700) of various vehicles, and may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle internal temperature data, and vehicle internal humidity data.

[0261] Meanwhile, the camera device (195) can continuously output camera data to the signal processing device (170) during vehicle operation.

[0262] Meanwhile, Lidar (196) can continuously output Lidar data to a signal processing device (170) during vehicle operation.

[0263] Meanwhile, the neural processor (179) can detect an object based on camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0264] Meanwhile, the neural processor (179) can receive camera data at a fixed frame rate, detect an object based on the camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0265] Meanwhile, among the multiple virtualization machines (810, 830, 850), the first virtualization machine (810), which is a server virtualization machine, controls the operation of the neural processor (179).

[0266] Meanwhile, among the multiple virtualization machines (810, 830, 850), the second virtualization machine (850) and the third virtualization machine (830), which are guest virtualization machines, can each execute an application.

[0267] In the drawing, the second virtual machine (850) is illustrated as executing an ADAS (Advanced Driver Assistance Systems) application (Nad) or an autonomous driving application or a Driver monitoring system (DMS) application (Ndm), and the third virtual machine (830) is illustrated as executing an augmented reality (AR) application (Nar).

[0268] The first virtual machine (810) sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications running on at least one of the plurality of virtual machines (810, 830, 850), and if the first operation and the third operation can be processed in parallel, the first neural processor (179a) controls the first operation and the third operation to be processed in parallel, and controls the second operation to be processed after the first operation and the third operation are completed. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0269] Meanwhile, if the first virtual machine (810) receives a request for a fourth operation after the third operation has been requested and sharing of the operation layers between the second and fourth operations is possible, the first neural processor (179a) can be controlled to sequentially process the second and fourth operations after the first and third operations are completed. Accordingly, the neural processor can be operated efficiently.

[0270] Meanwhile, the first virtual machine (810) can control the arrangement of data for multiple operations within the internal memory (1805) of the first neural processor (179a) to vary when multiple operations are requested from multiple applications. Accordingly, the neural processor can be operated efficiently.

[0271] Meanwhile, the first virtual machine (810) can execute a neural system service (1110) to control at least one neural processor (179a to 179c).

[0272] Meanwhile, the neural system service (1110) can control the arrangement of data for multiple operations within the internal memory (1805) of the first neural processor (179a) to vary when multiple operations are requested from multiple applications. This enables efficient operation of the neural processor.

[0273] Meanwhile, the neural system service (1110) may execute or be equipped with a neural manager (1113) for managing at least one neural processor (179a to 179c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (179a to 179c), and a neural interface (1118) for interfacing with at least one neural processor (179a to 179c).

[0274] Meanwhile, the neural system service (1110) may further execute or include a model container (509) that performs interface of model parameters related to the operation of the neural processor (179) and version management of learning files.

[0275] The neural manager (1113) can perform artificial intelligence model management, learning model management, camera data management, sensor data management, or command queue management.

[0276] The neural controller (1115) can determine an optimal inference method of at least one neural processor (179a to 179c), perform queue, partition, caching, or scalable coding, or control at least one neural processor (179a to 179c).

[0277] The neural interface (1118) can execute an application program interface (API) related to the accelerator of at least one neural processor (179a-179c).

[0278] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing between the neural system service (1110) and the model container (509) or the neural system service (1110) and the shared memory (508).

[0279] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing to the first virtual machine (810).

[0280] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing to a vehicle driving assistance application (Nad) or a driver monitoring system application (Ndm) running within the second virtual machine (850) or an augmented reality application (Nar) running within the third virtual machine (830).

[0281] For example, the interface (522) within the first virtual machine (810) can be controlled to transmit camera data, sensor data, or voice data to the neural processor (179) using the shared memory (508).

[0282] Meanwhile, the interface (522) within the first virtual machine (810) can control the transmission of result data output from the neural processor (179) and recorded in the shared memory (508) to the neural system service (1110).

[0283] Meanwhile, the interface (522) within the first virtual machine (810) can control the transmission of result data output from the neural processor (179) and recorded in the shared memory (508) to a vehicle driving assistance application (Nad) or a driver monitoring system application (Ndm) running within the second virtual machine (850) or an augmented reality application (Nar) running within the third virtual machine (830).

[0284] Meanwhile, the first virtual machine (810) may be executed based on the first operating system (805), the second virtual machine (850) may be executed based on the second operating system (805b) with a high safety level, and the third virtual machine (830) may be executed based on the third operating system (805c).

[0285] That is, multiple virtual machines (810, 830, 850) may be run based on different operating systems, or may be run based on at least two operating systems.

[0286] Meanwhile, the second operating system (805b) may be an operating system corresponding to a second safety level, such as ASIL D, and the third operating system (805c) may be an operating system corresponding to a first safety level, such as ASIL B.

[0287] Meanwhile, the first operating system (805) may be an operating system corresponding to a second safety level, such as ASIL D. However, the present invention is not limited thereto, and the first operating system (805) may also be an operating system corresponding to a first safety level, such as ASIL B.

[0288] Meanwhile, the neural manager (1113) can manage the operating requirements of an artificial neural network-based application, control neural network weight data, and process necessary input data.

[0289] Meanwhile, the neural manager (1113) can sequentially process the optimized command queue through a hardware accelerator and transmit the operation result to the application.

[0290] Driving requirements can include the computational priorities, dependencies, and accuracy of a neural network. Operational priorities are preset values, such as whether the first operation should always be processed before the second, or, in the case of a safety-critical neural network, whether it should be processed first in the command queue before other candidate neural networks.

[0291] Meanwhile, neural network weight data refers to a file in which the element values ​​of each matrix are structured and stored in the process of inferring the results of a neural network calculated through a series of matrix operations.

[0292] Neural network weight data can be stored in advance as a model container (509) within the neural system service (1110) through an API call of the neural system service (1110) during the application installation process.

[0293] Meanwhile, the basic weight data loaded into the model container (509) can be automatically converted and stored in various discretization levels during the system initialization process. For example, if the basic weight is defined as FP32, it can be sub-discretized into INT8, INT16, and FP16, and a total of four weight files can be stored.

[0294] Required input data refers to input signals required for the current neural network to operate, such as vehicle speed, current location, radar, lidar, camera images, and intermediate or final calculation results of the preceding neural network.

[0295] The input data can be transmitted in real time to the shared memory (508) within the hypervisor (505) through an interface operating through the central processor (175) in the server virtualization machine.

[0296] A command queue is a memory buffer of a sequential FIFO data structure that can define a series of orders for processing artificial neural networks through hardware accelerators.

[0297] A single neural network operation request entering the command queue can be transmitted along with metadata such as the application name, the location of the application virtualization machine, the storage destination of the operation result, hardware accelerator control settings, the memory location of the input data, and the memory location information for each discretization level of the weight data.

[0298] The hardware accelerator control settings may include a unique number of the hardware accelerator in charge of the operation, the current target discretization level of the weight data (INT8, INT16, FP16, FP32, etc.), and a target neural network weight location mapping table for each hardware accelerator internal memory address.

[0299] Meanwhile, the neural controller (1115) can schedule an optimized command queue based on the requested artificial neural network operation commands and the availability of current hardware resources, and control the actual hardware accelerator to match the expected operation of the command queue.

[0300] The neural controller (1115) can receive neural network operation requirements from the neural manager (1113) and optimize the command queue.

[0301] The optimization process, in other words, can be calculated through a simulated scheduling that checks the priority, dependency, and accuracy metadata for each slot in the current command queue, and applies various queue optimization techniques (such as Partition, Caching, and Accuracy Coding) to all candidate commands in the current command queue, and finds a combination that maximizes hardware utilization and minimizes the latency of individual operation requests within a unit of time.

[0302] Based on the optimal slot location obtained in this way, a weight file (learning model) can be requested from the neural manager (1113) and loaded into the hardware internal memory.

[0303] If two different neural networks are managed as a single virtual neural network using Partition among optimization techniques and input as hardware operation requests, the start and end positions of the weights of the first operation corresponding to the address of the hardware internal memory can be recorded in a mapping table, and then the start and end positions of the weights of the second operation can be recorded in the mapping table.

[0304] Through this, the hardware accelerator performs the process of parallel processing of a virtual neural network, but the neural controller (1115) can separate the results of the operation into the results of the first operation and the second operation through a mapping table and transmit them separately to individual applications.

[0305] After completing the above initialization process, the neural controller (1115) can receive a sequential processing request of the command queue from the neural manager (1113).

[0306] At this time, the neural controller (1115) can be controlled to extract input data prepared in advance by the neural manager (1113) from the input data queue, pair the neural network weights with the corresponding input data, and perform computational processing through the hardware accelerator API.

[0307] If, unlike the initial driving requirements, the discretization level of the current neural network is changed according to a specific situation, the neural controller (1115) can perform a bitwise concanate operation that concatenates the weight conversion difference value (Delta) of the hardware internal memory to the basic weight of the current internal memory, thereby converting the discretization level of the basic weight of the internal memory in real time.

[0308] Meanwhile, the central processor (175) executes an application for driving the vehicle, and when it is determined that the application has failed to operate, it controls a second application corresponding to the application to be executed in another central processor (175) or another signal processing device (170), and varies the standard fallback guarantee time for the application's operation failure based on the safety level of the application.

[0309] Meanwhile, the above-described fallback guarantee time may mean the time from the fallback start time to the fallback end time.

[0310] Alternatively, the fallback guarantee time may mean the time from when the application determines that the operation has failed or when a failure has been determined to the time when the fallback starts to the time when the fallback ends.

[0311] Meanwhile, the safety level may mean the Automotive Safety Integrity Level (ASIL), the autonomous driving level, or a combination of the Automotive Safety Integrity Level and the autonomous driving level.

[0312] Accordingly, applications for vehicle operation can be performed reliably. In particular, applications for vehicle operation can be performed reliably based on safety levels.

[0313] Meanwhile, the central processor (175) may set the reference fallback guarantee time to a first time period when the safety level of the application is the corresponding first safety level, and may set the reference fallback guarantee time to a second time period longer than the first time period when the safety level of the application is the second safety level higher than the first safety level. Accordingly, the application for vehicle driving can be stably performed.

[0314] For example, the central processor (175) may set the standard fallback guarantee time to a first time, approximately 10 seconds, for a first application corresponding to ASIL D when the autonomous driving level is Level 3, and may set the standard fallback guarantee time to a second time, approximately 30 seconds, for a second application corresponding to ASIL D when the autonomous driving level is Level 4. Accordingly, it is possible to stably perform applications for driving a vehicle based on the safety level.

[0315] As another example, the central processor (175) may set the standard fallback guarantee time to approximately 7 seconds for a third application corresponding to ASIL B when the autonomous driving level is Level 3, and may set the standard fallback guarantee time to approximately 10 seconds for a fourth application corresponding to ASIL D when the autonomous driving level is Level 3. Accordingly, it is possible to stably perform applications for driving the vehicle based on the safety level.

[0316] As another example, the central processor (175) may set the reference fallback guarantee time to a first time period of approximately 10 seconds when the safety level of the augmented reality application (Nar) is the first safety level corresponding to ASIL B, and may set the reference fallback guarantee time to a second time period of approximately 30 seconds when the safety level of the vehicle driving assistance application (Nad) is the second safety level corresponding to ASIL D, which is higher than ASIL B. Accordingly, it is possible to stably perform an application for vehicle driving based on the safety level.

[0317] Meanwhile, the central processor (175) can set the standard fallback guarantee time to a third time shorter than the first time when the safety level of the application is a third safety level lower than the first safety level. Accordingly, the application for vehicle driving can be stably performed.

[0318] For example, the central processor (175) can set the standard fallback guarantee time to approximately 1 second, which is the third time, for the fifth application corresponding to ASIL D or ASIL B when the autonomous driving level is Level 2.

[0319] As another example, the central processor (175) may set the standard fallback guarantee time to approximately 0.7 seconds for the sixth application corresponding to QM when the autonomous driving level is Level 2.

[0320] As another example, the central processor (175) may set the standard fallback guarantee time to the third time, approximately 0.5 seconds, when the safety level of the augmented reality application (Nar) corresponds to a QM lower than ASIL B. Accordingly, the application for driving the vehicle can be stably performed based on the safety level.

[0321] Meanwhile, the central processor (175) can control the standard fallback guarantee time of the application executed in the second virtual machine (850) among the plurality of virtual machines (810, 830, 850) to be greater than the standard fallback guarantee time of the application executed in the third virtual machine, when the second virtual machine (850) executes an application with a higher security level than the third virtual machine (830).

[0322] For example, the central processor (175) can be set so that when the second virtual machine (850) executes a first application with an autonomous driving level of Level 4, the standard fallback guarantee time is approximately 30 seconds, and when the third virtual machine (830) executes a second application with an autonomous driving level of Level 3, the standard fallback guarantee time is approximately 10 seconds. Accordingly, it is possible to stably perform applications for vehicle driving based on a safety level.

[0323] As another example, the central processor (175) may set the standard fallback guarantee time of the vehicle driving assistance application (Nad) to be approximately 30 seconds when the second virtual machine (850) executes a vehicle driving assistance application (Nad) corresponding to ASIL D, and may set the standard fallback guarantee time of the augmented reality application (Nar) to be approximately 10 seconds when the third virtual machine (830) executes an augmented reality application (Nar) corresponding to ASIL B. Accordingly, it is possible to stably perform applications for vehicle driving based on a safety level.

[0324] FIG. 10A is a flowchart illustrating an operation method of a signal processing device according to one embodiment of the present disclosure.

[0325] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure calculates a forward collision time or a forward collision avoidance time with an object (OBm) in front of the vehicle (S1010).

[0326] For example, a processor (175) within a signal processing device (170) can receive a forward image from a front camera (195a) and, based on the forward image, calculate a forward collision time or forward collision avoidance time with an object (OBm) in front of the vehicle.

[0327] As another example, a processor (175) within a signal processing device (170) may receive a sensing signal from a lidar (196) or a radar (197) and, based on the sensing signal, calculate a forward collision time or forward collision avoidance time with an object (OBm) in front of the vehicle.

[0328] Next, the processor (175) in the signal processing device (170) calculates the forward gaze level of the driver (OWa) based on the internal image from the internal camera (195i) (S1020).

[0329] For example, the processor (175) within the signal processing device (170) can detect the driver's face within the internal image from the internal camera (195i) and detect the direction of the driver's face or the direction of the driver's gaze.

[0330] And, the processor (175) can calculate the forward gaze level of the driver (OWa) based on the direction of the driver's face or the direction of the driver's gaze.

[0331] Next, the processor (175) in the signal processing device (170) varies the forward collision avoidance control time based on the forward gaze level (S1030).

[0332] Specifically, the processor (175) within the signal processing device (170) varies the forward collision avoidance control timing based on the forward collision time or forward collision avoidance time and the forward gaze level. Accordingly, adaptive vehicle control can be performed based on the forward gaze level of the driver (OWa).

[0333] FIG. 10b is a flowchart showing an operation method of a signal processing device according to another embodiment of the present disclosure.

[0334] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure calculates a forward collision time or a forward collision avoidance time with an object (OBm) in front of the vehicle (S1010).

[0335] For example, a processor (175) within a signal processing device (170) can receive a forward image from a front camera (195a) and, based on the forward image, calculate a forward collision time or forward collision avoidance time with an object (OBm) in front of the vehicle.

[0336] Next, the processor (175) in the signal processing device (170) calculates the forward gaze level of the driver (OWa) based on the internal image from the internal camera (195i) (S1020).

[0337] For example, the processor (175) within the signal processing device (170) can calculate the forward gaze level of the driver (OWa) based on the direction of the driver's face or the direction of the driver's gaze within the internal image from the internal camera (195i).

[0338] Next, the processor (175) within the signal processing device (170) can determine whether the forward gaze level is the first level (S1022), and if so, set the forward collision avoidance control time point to the first time point (Tmb) (S1024).

[0339] Meanwhile, in step 1022 (S1022), if the forward gaze level is not the first level, the processor (175) within the signal processing device (170) determines whether the forward gaze level is a second level lower than the first level (S1026), and if so, the forward collision avoidance control time point can be set to the second time point (Tma) prior to the first time point (Tmb). Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0340] Meanwhile, the processor (175) can set the forward collision avoidance control timing to the third timing, which is between the first timing (Tmb) and the second timing (Tma), when the forward gaze level is the third level between the first and second levels. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0341] Meanwhile, the processor (175) can control the forward collision avoidance control timing to be delayed as the forward gaze level increases. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0342] Figures 11a to 15 are drawings referenced in the operation description of Figures 9 to 10b.

[0343] Figure 11a is a drawing showing an example of the interior of a vehicle.

[0344] Referring to the drawing, inside the vehicle, a driver (OWa), a windshield (Arma) in front of the driver (OWa), a roof (Armf) on the upper side of the windshield (Arma), a driver's seat glass (Armb) on one side of the A pillar, an passenger's seat glass (Armd) on one side of the A pillar, a dashboard (Armc), etc. may be arranged.

[0345] The processor (175) can set the forward gaze level to be the highest when the driver's (OWa) face direction or gaze direction is toward the windshield (Arma).

[0346] That is, the processor (175) can set the forward gaze level to be higher when the driver's (OWa) face direction or gaze direction is toward the windshield (Arma) rather than toward the roof (Armf), driver's seat glass (Armb), passenger seat glass (Armd), or dashboard (Armc).

[0347] Figure 11b is a drawing showing an example of the front of a vehicle.

[0348] Referring to the drawing, the driver's (OWa) gaze can be fixed on any one of a plurality of areas (GRa, GRb, GRc) in front of the windshield (Arma) of the vehicle.

[0349] The processor (175) in the signal processing device (170) can set the forward gaze level of the first area (GRa) where the front vehicle (OBm) is located among the plurality of areas (GRa, GRb, GRc) to be the highest.

[0350] For example, the processor (175) within the signal processing device (170) can set the forward gaze level of the first area (GRa) where the front vehicle (OBm) is located to be the highest, the forward gaze level of the third area (GRc) to be the lowest, and the forward gaze level of the second area (GRb) to be an intermediate level.

[0351] That is, the processor (175) within the signal processing device (170) can control the forward gaze level of the first area (GRa) where the front vehicle (OBm) is located to be the highest, and the forward gaze level to be lower as the distance from the front vehicle (OBm) increases.

[0352] Figure 11c is a drawing showing another example of the front of the vehicle.

[0353] Referring to the drawing, the driver's (OWa) gaze can be fixed on any one of a plurality of areas (GRma, GRmb, GRmc) in front of the vehicle's windshield (Arma).

[0354] Meanwhile, the processor (175) in the signal processing device (170) can vary the forward gaze level according to the area size among the plurality of areas (GRma, GRmb, GRmc) including the front vehicle (OBm).

[0355] For example, the processor (175) in the signal processing device (170) can set the forward gaze level of the area (GRma) having the smallest size among the plurality of areas (GRma, GRmb, GRmc) including the front vehicle (OBm) to be the highest.

[0356] As another example, the processor (175) in the signal processing device (170) may set the forward gaze level of the area (GRmc) having the largest size among the plurality of areas (GRma, GRmb, GRmc) including the front vehicle (OBm) to be the lowest.

[0357] Meanwhile, the processor (175) in the signal processing device (170) can set the forward gaze level of the intermediate-sized area (GRmn) among the plurality of areas (GRma, GRmb, GRmc) including the front vehicle (OBm) to be lower than the forward gaze level of the area (GRma) and higher than the forward gaze level of the area (GRmc).

[0358] That is, the processor (175) within the signal processing device (170) can control the forward gaze level of the area (GRma) having the smallest size among the plurality of areas (GRma, GRmb, GRmc) including the front vehicle (OBm) to be the highest, and the forward gaze level to be lowered as the size of the area increases.

[0359] Figures 12a to 12c are drawings illustrating detection of a driver's line of sight through an internal camera.

[0360] Figure 12a illustrates that the driver's gaze is positioned on the vehicle ahead (OBm).

[0361] Referring to the drawing, the processor (175) within the signal processing device (170) can detect the direction of the driver's (OWa) face or the direction of the driver's (OWa) gaze based on the internal image from the internal camera (195i).

[0362] Meanwhile, the processor (175) within the signal processing device (170) can set the forward gaze level to the first level when the driver's (OWa) gaze direction (DRa) is located at the front vehicle (OBm), as shown in FIG. 12a.

[0363] Figure 12b illustrates that the driver's gaze is positioned on a second display (180b) near the dashboard inside the vehicle.

[0364] Referring to the drawing, the processor (175) within the signal processing device (170) can set the forward gaze level to a second level lower than the first level when the driver's (OWa) gaze direction (DRb) is located at the second display (180b) or mobile terminal (not shown) near the dashboard.

[0365] Figure 12c illustrates that the driver's gaze is positioned between the vehicle in front (OBm) and the interior of the vehicle.

[0366] Referring to the drawing, the processor (175) within the signal processing device (170) can set the forward gaze level to a third level between the first level and the second level when the driver's (OWa) gaze direction (DRc) is located between the front vehicle (OBm) and the interior of the vehicle.

[0367] Referring to FIGS. 12a to 12c, a processor (175) within a signal processing device (170) can calculate a distraction factor based on the gaze direction of the driver (OWa).

[0368] At this time, the distraction factor may be inversely proportional to the forward gaze level.

[0369] For example, the processor (175) within the signal processing device (170) can be set so that, as in FIG. 12a, when the driver's (OWa) gaze direction (DRa) is located at the front vehicle (OBm), the level of the distraction factor is the lowest, and as in FIG. 12b, when the driver's (OWa) gaze direction (DRb) is located at the second display (180b) or mobile terminal (not shown) near the dashboard, the level of the distraction factor is the highest.

[0370] Meanwhile, a processor (175) within a signal processing device (170) calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle based on the level of the distraction factor, and can vary a forward collision avoidance control time based on the forward collision time or forward collision avoidance time and the forward gaze level.

[0371] For example, the processor (175) within the signal processing device (170) can control the forward collision avoidance control timing to be accelerated as the level of the distraction factor increases. Accordingly, adaptive vehicle control can be performed according to the level of the driver's distraction factor.

[0372] Figures 13a to 13c are diagrams illustrating vehicle control in the case of poor forward visibility of the driver.

[0373] Figure 13a illustrates a vehicle (200) and a front vehicle (200b).

[0374] Referring to the drawing, a vehicle (200) is driving on a lane between two lanes (LNa, LNb), and a front vehicle (200b) may be driving on some lanes (LNa).

[0375] Meanwhile, the processor (175) within the signal processing device (170) can set the forward gaze level to a second level or a third level lower than the first level when the driver's (OWa) gaze direction is not located at the front vehicle (200b), as shown in FIG. 12b or FIG. 12c.

[0376] Meanwhile, the processor (175) in the signal processing device (170) can perform control for forward collision avoidance based on the distance from the front vehicle (200b), etc.

[0377] To this end, the processor (175) within the signal processing device (170) can calculate the time-to-collision (TTC) or time-to-avoid (TTA) for forward collision avoidance.

[0378] The drawing illustrates a graph (GRa) showing the expected collision avoidance time with the preceding vehicle.

[0379] Meanwhile, the processor (175) within the signal processing device (170) can be controlled so that the allowable time in the expected collision avoidance time increases as the forward gaze level decreases.

[0380] That is, the processor (175) within the signal processing device (170) can set the allowable time of the expected collision avoidance time to the first time (THa) when the forward gaze level is the second level or the third level.

[0381] Meanwhile, the processor (175) within the signal processing device (170) can output a warning message for forward collision avoidance when the expected collision avoidance time is less than or equal to the first time (THa), which is the allowable time.

[0382] Figure 13b illustrates outputting a warning message (1311) to the first display (180a) at the Tma time point.

[0383] Referring to the drawing, the processor (175) within the signal processing device (170) can be controlled to output a warning message (1311) to the first display (180a) at a time point Tma when the expected collision avoidance time is less than or equal to the first time (THa), which is an allowable time.

[0384] Accordingly, it becomes possible to output a warning message (1311) at the Tma time point, which is earlier than the Tmb time point.

[0385] Ultimately, it becomes possible to induce forward collision avoidance action by recognizing the driver's (OWa) warning message (1311).

[0386] Figure 13c illustrates setting the steering angle change level to the second setting level at the Tma time point.

[0387] Referring to the drawing, the processor (175) in the signal processing device (170) can set the steering angle change level for forward collision avoidance to a second set level (STa) lower than the first set level (STb in FIG. 14c) at a time point Tma when the expected collision avoidance time is less than or equal to the first time (THa), which is an allowable time.

[0388] That is, at the Tma time point, which is earlier than the Tmb time point, the steering angle of the vehicle (200) can be changed based on the second setting level, which is the steering angle change level.

[0389] To this end, the processor (175) within the signal processing device (170) can output a steering angle change signal corresponding to the second set level (STa) at the time point Tma.

[0390] Meanwhile, the steering drive unit (652) receives a steering angle change signal through a signal processing device (170), a communication module (EM), an ECU (770), etc., and can change the steering angle according to the second setting level (STa) of the steering angle change signal.

[0391] Finally, after the Tma point, forward collision avoidance control can be performed. That is, the vehicle (200) can be moved to the right of the vehicle in front (200b).

[0392] Figures 14a to 14c are diagrams illustrating vehicle control when the driver has good forward visibility.

[0393] First, Fig. 14a illustrates a vehicle (200) and a front vehicle (200b).

[0394] Referring to the drawing, a vehicle (200) is driving on a lane between two lanes (LNa, LNb), and a front vehicle (200b) may be driving on some lanes (LNa).

[0395] Meanwhile, the processor (175) within the signal processing device (170) can set the forward gaze level to the first level when the driver's (OWa) gaze direction is located at the front vehicle (200b), as shown in FIG. 12a.

[0396] Meanwhile, the processor (175) in the signal processing device (170) can perform control for forward collision avoidance based on the distance from the front vehicle (200b), etc.

[0397] The drawing illustrates a graph (GRa) showing the expected collision avoidance time with the preceding vehicle.

[0398] Meanwhile, the processor (175) within the signal processing device (170) can be controlled so that the allowable time in the expected collision avoidance time decreases as the forward gaze level increases.

[0399] That is, the processor (175) within the signal processing device (170) can set the allowable time of the expected collision avoidance time to a second time (THb) that is shorter than the first time (THa) when the forward gaze level is the first level.

[0400] Meanwhile, the processor (175) within the signal processing device (170) can output a warning message for forward collision avoidance when the expected collision avoidance time is less than or equal to the second time (THb), which is the allowable time.

[0401] Figure 14b illustrates outputting a warning message (1311) to the first display (180a) at the Tmb time.

[0402] Referring to the drawing, the processor (175) within the signal processing device (170) can control the first display (180a) to output a warning message (1311) at a time point Tmb when the expected collision avoidance time is less than or equal to the second time (THb), which is an allowable time.

[0403] Accordingly, it becomes possible to output a warning message (1311) at the Tmb time point, which is later than the Tma time point.

[0404] Ultimately, it becomes possible to induce forward collision avoidance action by recognizing the driver's (OWa) warning message (1311).

[0405] Figure 14c illustrates setting the steering angle change level to the first set level at the Tma time point.

[0406] Referring to the drawing, the processor (175) in the signal processing device (170) can set the steering angle change level for forward collision avoidance to the first set level at a time point Tmb when the expected collision avoidance time is less than or equal to the second time (THb), which is an allowable time.

[0407] That is, at the Tmb time point, which is later than the Tma time point, the steering angle of the vehicle (200) can be changed based on the first setting level, which is the steering angle change level.

[0408] Meanwhile, the processor (175) within the signal processing device (170) can output a steering angle change signal corresponding to the first set level (STb) at time Tmb.

[0409] Meanwhile, the steering drive unit (652) receives a steering angle change signal through a signal processing device (170), a communication module (EM), an ECU (770), etc., and can change the steering angle according to the first set level (STb) of the steering angle change signal. Consequently, after the Tmb time, forward collision avoidance control can be performed.

[0410] Meanwhile, the processor (175) within the signal processing device (170) can control the first setting level (STb) to be higher than the second setting level (STa) since the steering angle change control starts at the Tmb time point, which is later than the Tma time point.

[0411] Accordingly, the steering angle or steering torque in FIG. 14c may be greater than the steering angle or steering torque in FIG. 13c.

[0412] In summary, FIGS. 13a to 14c show that the processor (175) can vary the output timing of a warning message (1311) for forward collision avoidance in response to the forward gaze level. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0413] Meanwhile, the processor (175) may set the output time of the warning message (1311) for forward collision avoidance to the first output time (Tmb) when the forward gaze level is the first level, and may set the output time of the warning message (1311) for forward collision avoidance to the second output time (Tma) before the first output time (Tmb) when the forward gaze level is the second level lower than the first level. Accordingly, the warning message (1311) may be output adaptively according to the forward gaze level of the driver (OWa).

[0414] Meanwhile, the processor (175) can control the output timing of the warning message (1311) for forward collision avoidance to be delayed as the forward gaze level increases. Accordingly, the warning message (1311) can be output adaptively according to the forward gaze level of the driver (OWa).

[0415] Meanwhile, the processor (175) can vary the steering angle change level for forward collision avoidance in response to the forward gaze level. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0416] Meanwhile, the processor (175) may set the steering angle change level for forward collision avoidance to the first set level (STb) when the forward gaze level is the first level, and may set the steering angle change level for forward collision avoidance to the second set level (STa) lower than the first set level (STb) when the forward gaze level is the second set level (STa) lower than the first set level. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0417] Meanwhile, the processor (175) can control the steering angle change level for forward collision avoidance to increase as the forward gaze level increases. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0418] FIG. 15 is an example of an internal block diagram of a signal processing device according to an embodiment of the present disclosure.

[0419] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure can receive a front image, an internal image, or a sensing signal from a front camera (195a), an internal camera (195i), or a lidar (196), respectively.

[0420] Meanwhile, the signal processing device (170) may include a detection unit (1510) that detects an object or a lane, etc. based on a received image or sensing signal, a motion estimation unit (1520) that estimates motion based on the detected object or lane, etc., a determination unit (1530) that determines vehicle control based on the estimated motion, and an application execution unit (1540) that executes an application based on the determined vehicle control.

[0421] Meanwhile, the detection unit (1510) may include an object detection unit (1512) that detects an object in front of the vehicle from a front image or a driver's face or eyes from an interior image, a lane detection unit (1514) that detects a lane in front of the vehicle from a front image, and a sensor fusion unit (1516) that synthesizes an image signal or a sensing signal.

[0422] Meanwhile, the motion estimation unit (1520) may include an ego motion estimation unit (1522) for estimating ego motion related to estimating the direction of travel of the vehicle (200), a predicted path estimation unit (1524) for estimating the predicted path of the vehicle (200), a target selection unit (1526) for selecting an Autonomous Emergency Steering (AES) ticket, and a gaze level estimation unit (1528) for calculating the driver's forward gaze level.

[0423] Meanwhile, the decision unit (1530) may include a condition decision unit (1532) that determines an automatic emergency steering activity condition based on a signal from an ego motion estimation unit (1522), a prediction path estimation unit (1524), a target selection unit (1526), ​​or a gaze level estimation unit (1528), a state machine decision unit (1534) that determines an automatic emergency steering state machine, and an automatic emergency steering mode decision unit (1536) that determines an automatic emergency steering mode.

[0424] Meanwhile, the application execution unit (1540) may include a warning application (1542) for warning to avoid forward collision or an automatic emergency steering control application (1544) for executing an automatic emergency steering mode based on a signal from a condition determination unit (1532), a state machine determination unit (1534), or an automatic emergency steering mode determination unit (1536).

[0425] Meanwhile, the processor (175) within the signal processing device (170) can execute a driver monitoring application (Ndm) based on the internal image, as shown in FIG. 9, and execute an automatic emergency steering control application (1544) based on the front image.

[0426] The driver monitoring application (Ndm) at this time may include multiple microservices.

[0427] For example, a driver monitoring application (Ndm) may include microservices such as an object detection unit (1512) and a gaze level estimation unit (1528).

[0428] Meanwhile, the processor (175) within the signal processing device (170) can execute an object detection unit (1512), a monitoring level estimation unit (1528), etc. corresponding to the second safety level, ASIL D.

[0429] To this end, the processor (175) within the signal processing device (170) can be controlled to execute the object detection unit (1512), the attention level estimation unit (1528), etc., in a virtual machine (850) corresponding to a second safety level such as ASIL D. Accordingly, the driver monitoring application (Ndm) can be stably executed.

[0430] Meanwhile, the processor (175) within the signal processing device (170) can execute an automatic emergency steering control application (1544).

[0431] The automatic emergency steering control application (1544) at this time may include multiple microservices.

[0432] For example, an automatic emergency steering control application (1544) may include an object detection unit (1512), a lane detection unit (1514), an ego motion estimation unit (1522), a predicted path estimation unit (1524), a target selection unit (1526), ​​a gaze level estimation unit (1528), a condition determination unit (1532), a state machine determination unit (1534), or an automatic emergency steering mode determination unit (1536), which are microservices.

[0433] To this end, the processor (175) within the signal processing device (170) can be controlled to execute an object detection unit (1512), a lane detection unit (1514), an ego motion estimation unit (1522), a predicted path estimation unit (1524), a target selection unit (1526), ​​a gaze level estimation unit (1528), a condition determination unit (1532), a state machine determination unit (1534), or an automatic emergency steering mode determination unit (1536) in a virtual machine (850) corresponding to a second safety level such as ASIL D. Accordingly, the automatic emergency steering control application (1544) can be stably executed.

[0434] Next, the processor (175) within the signal processing device (170) can execute a warning application (1542) for forward collision avoidance based on the internal image and the forward image.

[0435] The warning application (1542) for forward collision avoidance at this time can correspond to the first safety level, ASIL B, or QM.

[0436] To this end, the processor (175) within the signal processing device (170) can control the execution of an alert application (1542) in a virtual machine (830) corresponding to a first safety level such as ASIL B.

[0437] Meanwhile, the safety level of the warning application (1542) may be lower than the safety level of the driver monitoring application (Ndm) or the automatic emergency steering control application (1544).

[0438] Meanwhile, the processor (175) executes a plurality of virtual machines (810, 830, 850) on the hypervisor (505), as shown in FIG. 9, and some virtual machines (850) among the plurality of virtual machines (810, 830, 850) can execute a driver monitoring application (Ndm) based on an internal image and an automatic emergency steering control application (1544) based on a forward image. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0439] Meanwhile, among the plurality of virtual machines (810, 830, 850), some other virtual machines (830) execute a warning application (1542) for forward collision avoidance, and the safety level of the warning application (1542) may be lower than the safety level of the driver monitoring application (Ndm) or the automatic emergency steering control application (1544).

[0440] Meanwhile, among the plurality of virtual machines (810, 830, 850), some virtual machines (850) can execute multiple microservices for an automatic emergency steering control application (1544) based on a forward image, and execute multiple microservices for a driver monitoring application (Ndm) based on an internal image. Accordingly, the automatic emergency steering control application (1544) and the driver monitoring application (Ndm) can be executed stably and efficiently. Furthermore, vehicle control can be efficiently performed using microservices.

[0441] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. A processor that receives and processes internal images from an internal camera; The above processor, Based on the above internal image, the driver's forward gaze level is calculated, A signal processing device that calculates a forward collision time or forward collision avoidance time with an object in front of a vehicle, and varies a forward collision avoidance control time based on the forward collision time or forward collision avoidance time and a forward gaze level.

2. In paragraph 1, The above processor, Receive a front image from the front camera, A signal processing device that calculates a forward collision time or a forward collision avoidance time with an object in front of the vehicle based on the forward image.

3. In paragraph 1, The above processor, If the above forward gaze level is the first level, the forward collision avoidance control time point is set to the first time point, A signal processing device that sets the forward collision avoidance control time point to a second time point that is before the first time point when the forward gaze level is a second level lower than the first level.

4. In paragraph 3, The above processor, A signal processing device that sets the forward collision avoidance control time point to a third time point between the first time point and the second time point when the forward gaze level is a third level between the first level and the second level.

5. In paragraph 1, The above processor, A signal processing device that controls the forward collision avoidance control timing to be delayed as the above forward gaze level increases.

6. In paragraph 1, The above processor, A signal processing device that varies the output timing of a warning message for forward collision avoidance in response to the above forward gaze level.

7. In paragraph 6, The above processor, When the above forward gaze level is the first level, the output time of the above warning message for forward collision avoidance is set to the first output time, A signal processing device that sets the output time of the warning message for forward collision avoidance to a second output time prior to the first output time when the forward gaze level is a second level lower than the first level.

8. In paragraph 6, The above processor, A signal processing device that controls the output timing of a warning message for forward collision avoidance so that the higher the forward gaze level, the later it is.

9. In paragraph 1, The above processor, A signal processing device that varies the steering angle change level for forward collision avoidance in response to the above forward gaze level.

10. In paragraph 9, The above processor, When the above forward gaze level is the first set level, the steering angle change level for the forward collision avoidance is set to the first level, A signal processing device that sets the steering angle change level for forward collision avoidance to a second set level lower than the first set level when the forward gaze level is a second level lower than the first level.

11. In paragraph 9, The above processor, A signal processing device that controls the steering angle change level for forward collision avoidance to increase as the forward gaze level increases.

12. In paragraph 2, The above processor, Based on the above internal image, execute the driver monitoring application, A signal processing device that executes an automatic emergency steering control application based on the above forward image.

13. In paragraph 12, The above processor, Based on the internal image and the front image, a warning application for forward collision avoidance is executed, A signal processing device wherein the safety level of the above warning application is lower than the safety level of the above driver monitoring application or the above automatic emergency steering control application.

14. In paragraph 2, The above processor, Run multiple virtual machines on a hypervisor, Some of the above virtual machines, Based on the above internal image, execute the driver monitoring application, A signal processing device that executes an automatic emergency steering control application based on the above forward image.

15. In paragraph 14, Some other virtual machines among the above multiple virtual machines, Run the above warning application for forward collision avoidance, A signal processing device wherein the safety level of the above warning application is lower than the safety level of the above driver monitoring application or the above automatic emergency steering control application.

16. In paragraph 14, Some of the above virtual machines, Based on the above forward image, execute multiple microservices for the automatic emergency steering control application, A signal processing device that executes multiple microservices for a driver monitoring application based on the above internal image.

17. A processor for receiving and processing an internal image from an internal camera and a front image from a front camera; The above processor, Based on the above internal image, the driver's forward gaze level is calculated, A signal processing device that calculates a forward collision time or forward collision avoidance time with an object in front of the vehicle based on the forward image, and varies a forward collision avoidance control time point or a steering angle change level for the forward collision avoidance based on the forward collision time or forward collision avoidance time and a forward gaze level.

18. A vehicle control device having a signal processing device according to any one of claims 1 to 17.

Images