Signal processing device and vehicle control device including same

Abstract

According to an embodiment of the present disclosure, a signal processing device and a vehicle control device including same comprise a processor for receiving a front image from a camera mounted in a vehicle and processing same, wherein the processor: performs lane detection on the basis of the front image from the camera; executes a lane keeping mode on the basis of the lane detection; according to a first mode during the lane keeping mode, controls a steering driving unit to maintain a first distance from a first lane among the first lane and a second lane adjacent to each other; and according to a second mode during the lane keeping mode, controls the steering driving unit to maintain a second distance from the first lane, the second distance differing from the first distance. Accordingly, a lane distance can be adaptively adjusted by applying a driving situation during operation of the lane keeping mode.

Description

Signal processing device and vehicle control device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle control device having the same, and more specifically, to a signal processing device capable of adaptively adjusting a lane spacing by reflecting a driving situation during lane maintenance mode operation and a vehicle control device having the same.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, for the convenience of vehicle users, a vehicle signal processing device is installed inside the vehicle.

[0004] The signal processing device inside the vehicle receives and processes sensor data from various sensor devices inside the vehicle due to advanced driver assistance systems (ADAS) or autonomous driving.

[0005] Prior art U.S. Patent No. US11840220 relates to a vehicle steering device, and discloses neutral control of the steering wheel to maintain straight driving of the vehicle within the lane.

[0006] However, according to prior literature, there is a disadvantage in that the vehicle steering device is controlled to position the vehicle in the center of the lane without considering the driving situation.

[0007] The problem to be solved by the present disclosure is to provide a signal processing device capable of adaptively adjusting the lane spacing by reflecting the driving situation during lane maintenance mode operation, and a vehicle control device having the same.

[0008] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of adaptively adjusting the lane spacing based on an emergency vehicle approaching from the rear of a vehicle, and a vehicle control device having the same.

[0009] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of adaptively adjusting the lane spacing based on the vehicle side situation and a vehicle control device having the same.

[0010] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of adaptively adjusting the lane spacing based on driving skill or the number of passengers in a vehicle, and a vehicle control device having the same.

[0011] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of adaptively adjusting the lane spacing based on a driver's gaze or emotional information, and a vehicle control device having the same.

[0012] A signal processing device and a vehicle control device having the same according to one embodiment of the present disclosure for solving the above technical problem include a processor that receives and processes a front image from a camera mounted in a vehicle, the processor performs lane detection based on the front image from the camera, and performs a lane keeping mode based on the lane detection, and controls a steering drive unit to maintain a first distance between a first lane among adjacent first and second lanes according to a first mode of the lane keeping mode, and controls a steering drive unit to maintain a second distance different from the first distance between the first lane and the second lane according to a second mode of the lane keeping mode.

[0013] Meanwhile, the processor can detect an emergency vehicle in the rear image based on the rear image from the camera, and control the second mode among the lane keeping modes to be performed based on the emergency vehicle.

[0014] Meanwhile, the processor may control the second mode of the lane keeping mode to be performed based on an emergency vehicle driving behind the vehicle, and may control the distance from the first lane to become smaller as the distance from the emergency vehicle gets closer.

[0015] Meanwhile, the processor may control the second mode of the lane keeping mode to be performed when a barrier or guardrail or a second vehicle larger than a reference size is located on the side of the vehicle based on a front image or a side image from the camera.

[0016] Meanwhile, the processor can detect a second vehicle on the side of the vehicle based on a front image or a side image from the camera, and control the second interval in the second mode of the lane keeping mode to be varied based on the size of the second vehicle.

[0017] Meanwhile, the processor can control the second interval to vary based on the speed of the vehicle in the second mode of the lane maintenance mode.

[0018] Meanwhile, the processor can detect pedestrians on the side of the vehicle based on a front image or a side image from the camera, and control the second interval to vary based on the number of pedestrians detected or the positions of the pedestrians.

[0019] Meanwhile, the processor can vary the second interval in the second mode among the lane keeping modes based on the driver's driving skill or the number of passengers in the vehicle or whether the passenger seat is occupied.

[0020] Meanwhile, the processor can detect the driver's gaze based on an internal image from an internal camera, and vary the second interval in the second mode among the lane keeping modes based on the direction of the driver's gaze.

[0021] Meanwhile, the processor can be controlled to detect the driver's gaze based on an internal image from an internal camera in a manual driving mode, detect a distance from the first lane in the front image, perform learning based on the distance between the driver's gaze and the first lane, and store the learning result in memory.

[0022] Meanwhile, the processor can set a second interval based on the learning result when performing the second mode among the lane maintenance modes.

[0023] Meanwhile, in manual driving mode, the processor can be controlled to detect the driver's emotional information based on an internal image from an internal camera, detect a distance from the first lane in the front image, perform learning based on the driver's emotional information and the distance from the first lane, and store the learning result in memory.

[0024] Meanwhile, the processor, in the manual driving mode, detects the distance from the first lane in the front image, controls the vehicle to store information on the vehicle speed and the distance from the first lane in the memory, and when the second mode is performed among the lane keeping modes, sets the second distance based on the information on the vehicle speed and the distance from the first lane.

[0025] Meanwhile, the processor may run multiple virtual machines on the hypervisor, and some of the multiple virtual machines may run a lane detection application based on a forward image and run multiple microservices for the lane detection application.

[0026] Meanwhile, some other virtual machines among the multiple virtual machines run a notification application for lane keeping mode, and the safety level of the notification application may be lower than the safety level of the lane detection application.

[0027] A signal processing device and a vehicle control device having the same according to another embodiment of the present disclosure include a processor that receives and processes a front image from a camera mounted in a vehicle, the processor performs lane detection based on the front image from the camera, and performs a lane maintenance mode based on the lane detection, and controls a steering drive unit to maintain the center of an adjacent first lane and a second lane according to a first mode of the lane maintenance mode, and controls a steering drive unit to come closer to one of the first lane and the second lane according to a second mode of the lane maintenance mode.

[0028] Meanwhile, the processor may control the steering drive unit to maintain a first distance from the first lane according to a first mode among the lane maintenance modes, and may control the steering drive unit to maintain a second distance different from the first distance from the first lane according to a second mode among the lane maintenance modes.

[0029] A signal processing device and a vehicle control device including the same according to one embodiment of the present disclosure include a processor that receives and processes a forward image from a camera mounted in a vehicle, the processor performs lane detection based on the forward image from the camera, and performs a lane maintenance mode based on the lane detection, and controls a steering drive unit to maintain a first distance between a first lane among adjacent first and second lanes according to a first mode of the lane maintenance mode, and controls a steering drive unit to maintain a second distance different from the first distance between the first lane and the second lane according to a second mode of the lane maintenance mode. Accordingly, the lane distance can be adaptively adjusted by reflecting a driving situation during lane maintenance mode operation.

[0030] Meanwhile, the processor can detect an emergency vehicle within the rear image from the camera and, based on the emergency vehicle, control the execution of the second mode among the lane keeping modes. Accordingly, the lane spacing can be adaptively adjusted based on the emergency vehicle approaching from behind the vehicle.

[0031] Meanwhile, the processor can control the second mode of lane keeping mode to be performed based on the presence of an emergency vehicle behind the vehicle, and control the gap from the first lane to decrease as the distance from the emergency vehicle decreases. Accordingly, the lane gap can be adaptively adjusted based on the presence of an emergency vehicle approaching from behind the vehicle.

[0032] Meanwhile, the processor may control the second mode of the lane keeping mode to be performed when a barrier, guardrail, or a second vehicle larger than a reference size is located on the side of the vehicle based on a forward or side image from the camera. Accordingly, the lane spacing can be adaptively adjusted based on the situation in front of the vehicle or the situation on the side of the vehicle.

[0033] Meanwhile, the processor can detect a second vehicle to the side of the vehicle based on a front or side image from the camera, and control the second interval in the second mode of the lane keeping mode to vary based on the size of the second vehicle. Accordingly, the lane interval can be adaptively adjusted based on the situation ahead of the vehicle or the situation to the side of the vehicle.

[0034] Meanwhile, the processor can control the second gap to vary based on the vehicle speed in the second mode of the lane maintenance mode. Accordingly, the lane gap can be adaptively adjusted based on the vehicle speed.

[0035] Meanwhile, the processor can detect pedestrians on the side of the vehicle based on forward or side images from the camera, and control the second interval to vary based on the number of pedestrians detected or their locations. Accordingly, the lane spacing can be adaptively adjusted based on the situation ahead of the vehicle or the situation on the side of the vehicle.

[0036] Meanwhile, the processor can vary the second interval in the second mode of lane keeping mode based on the driver's driving skill, the number of passengers in the vehicle, or whether the passenger seat is occupied. Accordingly, the lane spacing can be adjusted adaptively based on the driver's skill or the number of passengers in the vehicle.

[0037] Meanwhile, the processor can detect the driver's gaze based on internal images from the internal camera and vary the second interval in the second mode of the lane keeping mode based on the driver's gaze direction. This allows for adaptive lane spacing adjustment based on the driver's gaze direction.

[0038] Meanwhile, in manual driving mode, the processor can detect the driver's gaze based on internal images from the internal camera, detect the distance from the first lane in the forward image, perform learning based on the distance between the driver's gaze and the first lane, and store the learning results in memory. This allows adaptive lane spacing adjustment based on the driver's gaze.

[0039] Meanwhile, the processor can set a second interval based on the learning results when performing the second mode of the lane maintenance mode. Accordingly, the lane interval can be adaptively adjusted based on the learning results.

[0040] Meanwhile, in manual driving mode, the processor can detect the driver's emotional information based on internal images from the internal camera, detect the distance from the first lane in the forward image, perform learning based on the driver's emotional information and the distance from the first lane, and control the processor to store the learning results in memory. Accordingly, the lane distance can be adaptively adjusted based on the driver's emotional information.

[0041] Meanwhile, in manual driving mode, the processor detects the distance from the first lane in the forward image, controls the storage of information about the vehicle's speed and the distance from the first lane in memory, and, when performing the second mode among lane keeping modes, sets the second distance based on the information about the vehicle's speed and the distance from the first lane. Accordingly, the lane distance can be adaptively adjusted based on the information in the manual driving mode.

[0042] Meanwhile, the processor can run multiple virtual machines on the hypervisor, some of which execute a lane detection application based on the forward image, and multiple microservices for the lane detection application. This allows for efficient execution of the lane detection application.

[0043] Meanwhile, some of the other virtual machines among the multiple virtual machines run a notification application for lane keeping mode, and the safety level of the notification application may be lower than that of the lane detection application. This allows the lane detection application to run reliably.

[0044] A signal processing device and a vehicle control device including the same according to another embodiment of the present disclosure include a processor that receives and processes a forward image from a camera mounted in a vehicle, the processor performs lane detection based on the forward image from the camera, and performs a lane maintenance mode based on the lane detection, and controls a steering drive unit to maintain the center of an adjacent first lane and a second lane according to a first mode of the lane maintenance mode, and controls the steering drive unit to come closer to one of the first lane and the second lane according to a second mode of the lane maintenance mode. Accordingly, the lane spacing can be adaptively adjusted to reflect the driving situation during the lane maintenance mode operation.

[0045] Meanwhile, the processor may control the steering drive unit to maintain a first distance from the first lane according to a first mode of the lane maintenance mode, and may control the steering drive unit to maintain a second distance from the first lane, which is different from the first distance, according to a second mode of the lane maintenance mode. Accordingly, the lane distance can be adaptively adjusted to reflect the driving situation during lane maintenance mode operation.

[0046] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0047] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0048] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0049] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0050] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0051] Figures 5a to 5d are drawings showing various examples of vehicle control devices.

[0052] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0053] FIG. 7a is a drawing referenced in the description of a signal processing device related to the present disclosure.

[0054] FIG. 7b is a diagram illustrating an example of execution of a microservice according to an embodiment of the present disclosure.

[0055] FIG. 8 is an example of a signal processing system according to one embodiment of the present disclosure.

[0056] FIG. 9 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0057] FIGS. 10A to 10C are drawings for reference in explaining the operation of a vehicle control device related to the present disclosure.

[0058] FIG. 11A is a flowchart illustrating an operation method of a signal processing device according to one embodiment of the present disclosure.

[0059] FIG. 11b is a flowchart showing an operation method of a signal processing device according to another embodiment of the present disclosure.

[0060] Figures 12a to 18 are drawings for reference in the operation description of Figures 11a to 11b.

[0061] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0062] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0063] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0064] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0065] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0066] Meanwhile, the vehicle (200) may be equipped with multiple displays (180a, 180b) for displaying images, information, etc. inside.

[0067] In Fig. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) is also possible.

[0068] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be called a center information display.

[0069] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0070] Figure 2 is a diagram illustrating the architecture of a signal processing system for a vehicle.

[0071] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0072] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the plurality of zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the plurality of zones (Z1 to Z4).

[0073] Meanwhile, the signal processing device (170a) may further include, in addition to the vehicle communication gateway (GWDa), an autonomous driving control module (ACC), a cockpit control module (CPG), etc.

[0074] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be an HPC (High Performance Computing) gateway.

[0075] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) within a plurality of zones (Z1 to Z4).

[0076] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0077] Referring to the drawing, the interior of the vehicle may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a room mirror display (not shown), etc.

[0078] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0079] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0080] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0081] The signal processing device (170) has a processor (175) therein and can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0082] A second virtual machine (not shown) can operate for the first display (180a), and a third virtual machine (not shown) can operate for the second display (180b).

[0083] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0084] Meanwhile, the first virtual machine (not shown) within the processor (175) shares at least a portion of data with the second virtual machine (not shown) and the third virtual machine (not shown) for data sharing processing. Accordingly, data can be shared and processed among multiple virtual machines for multiple displays within the vehicle.

[0085] Meanwhile, a first virtual machine (not shown) within a processor (175) may receive and process wheel speed sensor data of the vehicle, and transmit the processed wheel speed sensor data to at least one of a second virtual machine (not shown) or a third virtual machine (not shown). Accordingly, the wheel speed sensor data of the vehicle may be shared with at least one virtual machine.

[0086] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information, or images.

[0087] The signal processing device (170) can control the RSE display (180c) by executing a fourth virtual machine (not shown) in addition to the first virtual machine to the third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0088] Accordingly, it is possible to control various displays (180a to 180c) using one signal processing device (170).

[0089] Meanwhile, some of the multiple displays (180a~180c) may operate under Linux OS, while others may operate under Web OS.

[0090] The signal processing device (170) according to the embodiment of the present disclosure can control the same information or the same image to be displayed in synchronization on displays (180a to 180c) operating under various operating systems (OS).

[0091] Meanwhile, in FIG. 3b, a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications and a vehicle speed indicator (212b) and a vehicle interior temperature indicator (213b) are displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) are displayed on a third display (180c).

[0092] Fig. 4 is an example of an internal block diagram of the vehicle of Fig. 1.

[0093] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may include a lamp driving unit (751), a steering driving unit (752), a brake driving unit (753), a power source driving unit (754), a suspension driving unit (756), an air conditioning driving unit (757), a window driving unit (758), a seat driving unit (761), and a signal processing device (170).

[0094] Meanwhile, the vehicle (200) may further include an ECU (770), multiple sensor devices (SN), and multiple communication modules (EMa to EMd).

[0095] Meanwhile, a vehicle (200) according to an embodiment of the present disclosure may further include a vehicle display device (100).

[0096] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa to EMd) for internal communication, a memory (140), a signal processing unit (170), a plurality of displays (180a to 180c), an audio output unit (185), and a power supply unit (190).

[0097] A plurality of communication modules (EMa to EMd) can be arranged, for example, in a plurality of zones (Z1 to Z4) of FIG. 2, respectively.

[0098] Meanwhile, the signal processing device (170) may have a communication switch (736b) for data communication with each communication module (EM1 to EM4) inside.

[0099] Each communication module (EM1 to EM4) can perform data communication with multiple sensor devices (SN) or ECUs (770) or area signal processing devices (170Z).

[0100] Meanwhile, the plurality of sensor devices (SN) may include a camera (195), a lidar (196), a radar (197), or a position sensor (198).

[0101] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0102] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0103] The communication unit (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0104] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0105] The communication unit (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (900). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0106] A plurality of communication modules (EM1 to EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or an area signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0107] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0108] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0109] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0110] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can transmit location information data sensed by a GPS module or location sensor (198) to a signal processing device (170).

[0111] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0112] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0113] For example, the memory (140) may store data regarding a hypervisor, a first virtual machine, a third virtual machine, or the like, for execution within the processor (175).

[0114] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0115] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0116] The signal processing device (170) controls the overall operation of each unit in the vehicle display device (100) or the vehicle (200).

[0117] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0118] The processor (175) can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0119] Among the first virtual machine to the third virtual machine (not shown), the first virtual machine (not shown) may be named a server virtual machine, and the second virtual machine to the third virtual machine (not shown) may be named a guest virtual machine.

[0120] For example, a first virtual machine (not shown) within a processor (175) may receive, process, or output sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data.

[0121] In this way, by performing most of the data processing in the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0122] As another example, a first virtual machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for a second virtual machine or a third virtual machine (not shown).

[0123] And, the first virtual machine (not shown) can transmit processed data to the second virtual machine or the third virtual machine (not shown).

[0124] Accordingly, among the first virtual machine to the third virtual machine (not shown), only the first virtual machine (not shown) receives sensor data, communication data, or external input data from multiple sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtual machines, enabling 1:N data communication, and enabling synchronization when sharing data.

[0125] Meanwhile, the first virtual machine (not shown) can control the second virtual machine (not shown) and the third virtual machine (not shown) to share the same data by writing data to the shared memory (508).

[0126] For example, a first virtual machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in shared memory (508) and control the same data to be shared with a second virtual machine (not shown) and a third virtual machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0127] Ultimately, by performing most of the data processing on the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0128] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown).

[0129] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0130] Meanwhile, the signal processing device (170) of FIG. 4 may be the same as the signal processing device (170, 170a1, 170a2) of the vehicle control device of FIG. 5a or lower.

[0131] Figures 5a to 5d are drawings showing various examples of vehicle control devices.

[0132] FIG. 5A illustrates an example of a vehicle control device according to an embodiment of the present disclosure.

[0133] Referring to the drawings, a vehicle control device (800a) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0134] Meanwhile, the vehicle control device (800a) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0135] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0136] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0137] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0138] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0139] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0140] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0141] For example, sensor data within a vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle exterior radar data, and vehicle exterior lidar data.

[0142] Meanwhile, camera data may include vehicle exterior camera data and vehicle interior camera data.

[0143] Meanwhile, the signal processing device (170a1, 170a2) can execute multiple virtual machines (820, 830, 840) based on safety standards.

[0144] In the drawing, it is illustrated that a processor (175) within a signal processing device (170a) executes a hypervisor (505) and, on the hypervisor (505), executes first to third virtual machines (820 to 840) according to an automotive safety integrity level (Automotive SIL; ASIL).

[0145] The first virtual machine (820) may be a virtual machine corresponding to Quality Management (QM), which is the lowest safety level in the Automotive Safety Integrity Level (ASIL) and is a non-enforceable grade.

[0146] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0147] The second virtual machine (820) may be a virtual machine corresponding to ASIL A or ASIL B, where the sum of severity, exposure, and controllability is 7 or 8 in the automotive safety integrity level (ASIL).

[0148] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0149] The third virtual machine (840) may be a virtual machine corresponding to ASIL C or ASIL D, in which the sum of severity, exposure, and controllability is 9 or 10 in the automotive safety integrity level (ASIL).

[0150] Meanwhile, ASIL D can correspond to the grade that requires the highest safety level.

[0151] The third virtual machine (840) can run a safety operating system (842) and an application (845) on the operating system (842).

[0152] Meanwhile, the third virtual machine (840) may also execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0153] Meanwhile, unlike the drawing, the third virtual machine (840) can also be executed through a separate core rather than the processor (175). This will be described later with reference to FIG. 5b.

[0154] FIG. 5b illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0155] Referring to the drawing, the vehicle control device (800b) according to the embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0156] Meanwhile, the vehicle control device (800b) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0157] The vehicle control device (800b) of FIG. 5b is similar to the vehicle control device (800a) of FIG. 5a, but the signal processing device (170a1) has some differences from the signal processing device (170a1) of FIG. 5a.

[0158] To describe the difference, the signal processing device (170a1) may include a processor (175) and a second processor (177).

[0159] The processor (175) within the signal processing unit (170a1) executes a hypervisor (505), and executes first and second virtual machines (820 to 830) on the hypervisor (505) according to the automotive safety integrity level (Automotive SIL; ASIL).

[0160] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0161] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0162] Meanwhile, the second processor (177) within the signal processing device (170a1) can execute a third virtual machine (840).

[0163] The third virtual machine (840) can execute a safety operating system (842), an auto-execution (845) on the operating system (842), and an application (845) on the auto-execution (845). That is, unlike FIG. 5A, an auto-execution (846) on the operating system (842) can be executed.

[0164] Meanwhile, the third virtual machine (840) may, similarly to FIG. 5a, execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0165] Meanwhile, the third virtual machine (840) requiring a high level of security is preferably executed on a second processor (177), which is a different core or different processor, unlike the first and second virtual machines (820 to 830).

[0166] Meanwhile, in the signal processing devices (170a1, 170a2) of FIGS. 5a and 5b, when the first signal processing device (170a) malfunctions, the second signal processing device (170a2), which is a backup device, can operate.

[0167] Alternatively, it is also possible for the signal processing devices (170a1, 170a2) to operate simultaneously, with the first signal processing device (170a) operating as the main device and the second signal processing device (170a2) operating as the sub device. This will be described with reference to FIGS. 5c and 5d.

[0168] FIG. 5c illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0169] Referring to the drawing, a vehicle control device (800c) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0170] Meanwhile, the vehicle control device (800c) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0171] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0172] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0173] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0174] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0175] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0176] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0177] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175) in the first signal processing device (170a1) executes a hypervisor (505) and can execute a safety virtualization machine (860) and a non-safety virtualization machine (870) on the hypervisor (505), respectively.

[0178] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175b) in the second signal processing device (170a2) executes the hypervisor (505b) and can execute only the safety virtualization machine (880) on the hypervisor (505).

[0179] In this way, since the processing for safety is separated between the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0180] Meanwhile, high-speed network communication can be performed between the first signal processing device (170a1) and the second signal processing device (170a2).

[0181] FIG. 5d illustrates another example of a vehicle control device according to an embodiment of the present disclosure.

[0182] Referring to the drawing, a vehicle control device (800d) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2).

[0183] Meanwhile, the vehicle control device (800d) according to the embodiment of the present disclosure may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0184] The vehicle control device (800d) of FIG. 5d is similar to the vehicle control device (800c) of FIG. 5c, but the second signal processing device (170a2) has some differences from the second signal processing device (170a2) of FIG. 5c.

[0185] The processor (175b) in the second signal processing device (170a2) of FIG. 5d executes a hypervisor (505b) and can execute a safety virtualization machine (880) and a non-safety virtualization machine (890) on the hypervisor (505).

[0186] That is, unlike FIG. 5c, the difference is that the processor (175b) within the second signal processing device (170a2) further executes a non-safety virtualization machine (890).

[0187] In this way, since the processing for safety and non-safety is separated into the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0188] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0189] Referring to the drawing, a vehicle control device (900) according to an embodiment of the present disclosure includes a signal processing device (170).

[0190] The vehicle control device (900) according to the embodiment of the present disclosure may further include at least one display.

[0191] Meanwhile, the vehicle control device (900) according to the embodiment of the present disclosure may further include a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), an ECU (770), or a plurality of sensor devices (SN) of FIG. 4.

[0192] Meanwhile, the vehicle control device (900) according to the embodiment of the present disclosure may further include a lamp driving unit (751), a suspension driving unit (756), an air conditioning driving unit (757), a window driving unit (758), a seat driving unit (761), or a plurality of communication modules (EMa to EMd) of FIG. 4.

[0193] In the drawing, at least one display is illustrated, a cluster display (180a) and an AVN display (180b).

[0194] Meanwhile, the vehicle control device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0195] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having multiple CPUs (175), GPUs (178), NPUs (179), etc., and may be called an HPC (High Performance Computing) signal processing device or a central signal processing device.

[0196] A plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) are connected by wired cables (CB1 to CB4).

[0197] Meanwhile, multiple area signal processing devices (170Z1 to 170Z4) can be connected to each other with wired cables (CBa to CBd).

[0198] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0199] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may be equipped with at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0200] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0201] Meanwhile, sensor data may be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data may be stored in a storage device (925) within the signal processing device (170).

[0202] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0203] In the drawing, it is exemplified that camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2), a third area signal processing device (170Z3), etc.

[0204] Meanwhile, since the data read speed or write speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the plurality of area signal processing devices (170Z1 to 170Z4) to the signal processing device (170), it is preferable that multi-path routing be performed so that a network bottleneck does not occur.

[0205] To this end, the signal processing device (170) according to the embodiment of the present disclosure can perform multi-path routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data from the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, the network configuration can be dynamically changed to transmit data.

[0206] Data communication between a plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) in a vehicle control device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication for high-bandwidth, low-latency communication.

[0207] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can receive an internal image from an internal camera (195i) and perform signal processing on the internal image.

[0208] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can receive a front image from a front camera (195a) and perform signal processing on the front image.

[0209] FIG. 7a is a drawing referenced in the description of a signal processing device related to the present disclosure.

[0210] Referring to the drawing, a signal processing device (170x) related to the present disclosure can execute an application (785) based on sensor data or camera data of a vehicle, and output result data of the application (785) through multiple passes.

[0211] According to this method, the result data of the application (785) is output only after the execution of the application (785) is completed, so inefficiency occurs and a significant amount of time is likely to be required until the execution of the application (785) is completed.

[0212] Accordingly, in this disclosure, a method for sharing intermediate result data of an application, etc., when the application is executed is proposed.

[0213] To this end, the signal processing device (170) according to the embodiment of the present disclosure divides the application into a plurality of micro services, and executes other micro services based on the results of the micro services, etc., thereby efficiently distributing the workload.

[0214] FIG. 7b is a diagram illustrating an example of execution of a microservice according to an embodiment of the present disclosure.

[0215] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure can execute an application (795) based on sensor data or camera data of a vehicle, etc.

[0216] At this time, the signal processing device (170) according to the embodiment of the present disclosure can execute a plurality of micro-services separately for the application (795).

[0217] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute applications or microservices by distinguishing them according to the safety level.

[0218] At this time, the signal processing device (170) according to the embodiment of the present disclosure transmits the result data of the transmitting application or microservice when the security level of the transmitting application or microservice is higher or equal to that of the receiving application or microservice.

[0219] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure prevents the transmission of result data of the transmission application or microservice when the security level of the transmission application or microservice is lower than that of the reception application or microservice.

[0220] In the drawing, based on input data, a first microservice (910) corresponding to the second safety level, ASIL D, is executed, and the result data of the first microservice (910) corresponding to ASIL D can be transmitted to a second microservice (920a) corresponding to the third safety level, QM, a third microservice (920b) corresponding to the first safety level, ASIL B, a fourth microservice (920c) corresponding to the first safety level, ASIL B, and a fifth microservice (920d) corresponding to the third safety level, ASIL D, respectively.

[0221] Since the security level of the first microservice (910) is higher than the security levels of the second microservice (920a), the third microservice (920b), and the fourth microservice (920c), transmission of the result data of the first microservice (910) becomes possible.

[0222] Meanwhile, since the safety level of the first microservice (910) is the same as the safety level of the fifth microservice (920d), transmission of the result data of the first microservice (910) becomes possible.

[0223] Next, the sixth microservice (930a) corresponding to the third safety level, QM, is executed based on the result data of the second microservice (920a), and the result data can be output through the first pass.

[0224] Meanwhile, the seventh microservice (930b) corresponding to the first safety level, ASIL B, is executed based on the result data of the third microservice (920b) and the result data of the fourth microservice (920c), and the result data can be output through the second pass.

[0225] Meanwhile, the 8th microservice (930c) corresponding to the second safety level, ASIL D, is executed based on the result data of the 5th microservice (920d), and the result data can be output through the third pass.

[0226] As shown in the drawing, in addition to outputting the result data of the application (795) through multiple passes, unlike FIG. 7a, by executing and processing the corresponding microservice through each pass within the signal processing device (170), the workload can be efficiently distributed, thereby enabling data processing to be performed efficiently.

[0227] FIG. 8 is an example of a signal processing system according to one embodiment of the present disclosure.

[0228] Referring to the drawing, a signal processing system (1000) according to one embodiment of the present disclosure may include a central signal processing device (170) and an area signal processing device (170z).

[0229] Meanwhile, a signal processing device (170) in a system (1000) according to one embodiment of the present disclosure has a plurality of processor cores (CR1 to CRn, MR).

[0230] Meanwhile, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) may correspond to processor cores in the central processor (CPU) of FIG. 6.

[0231] For example, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) may correspond to application processor cores within the central processor (CPU) of FIG. 6.

[0232] Meanwhile, some (CR1 to CRn) of the multiple processor cores (CR1 to CRn, MR) operate based on a hypervisor (505), and the hypervisor (505) can execute multiple virtual machines (820 to 850).

[0233] Meanwhile, some of the other processor cores (CR1 to CRn, MR) may correspond to M cores or MCUs (micom units).

[0234] Meanwhile, some other processor cores (MR) among the plurality of processor cores (CR1 to CRn, MR) can execute an operating system (805a) corresponding to a second safety level such as ASIL D without executing a hypervisor (505), and execute a fourth virtual machine (840) on the operating system (805a).

[0235] Meanwhile, the fourth virtual machine (840) can execute an application corresponding to a second safety level, such as ASIL D, or a microservice (843) corresponding to an application corresponding to the second safety level. Accordingly, the application or microservice (843) corresponding to the second safety level can be stably performed.

[0236] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the first processor core (CR1) can execute a hypervisor (505), execute an operating system (805b) corresponding to a second safety level such as ASIL D on the hypervisor (505), and execute a first virtual machine (850) on the operating system (805b).

[0237] Meanwhile, the first virtual machine (850) can execute an application corresponding to a first safety level, such as ASIL B, or a microservice (853a, 853b) corresponding to an application corresponding to the first safety level. Accordingly, the application or microservice (853a, 853b) corresponding to the first safety level can be stably performed.

[0238] Meanwhile, unlike the drawing, the first processor core (CR1) among the multiple processor cores (CR1 to CRn, MR) may execute an operating system corresponding to the first safety level, such as ASIL B, on the hypervisor (505).

[0239] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the second processor core (CR2) and the third processor core (CR3) can execute a hypervisor (505), execute an operating system (805c) corresponding to a first safety level such as ASIL B on the hypervisor (505), and execute a second virtualization machine (850) on the operating system (805c).

[0240] Meanwhile, the second virtual machine (850) can execute a third application corresponding to a first safety level, such as ASIL B, or a microservice (833a to 833d) corresponding to the third application corresponding to the first safety level, on an operating system (805c) corresponding to the first safety level. Accordingly, the application or microservice (833a to 833d) corresponding to the first safety level can be stably performed.

[0241] Meanwhile, among the plurality of processor cores (CR1 to CRn, MR), the remaining processor cores (CR4 to CRn) can execute a hypervisor (505), execute an operating system (805d) corresponding to a third safety level such as QM on the hypervisor (505), and execute a third virtualization machine (820) on the operating system (805d).

[0242] Meanwhile, the third virtual machine (820) can execute a fourth application corresponding to the third safety level, such as QM, or a microservice (823a to 823d) corresponding to the fourth application corresponding to the third safety level, on an operating system (805d) corresponding to a third safety level lower than the first safety level. Accordingly, the application or microservice (823a to 823d) corresponding to the third safety level can be stably performed.

[0243] Meanwhile, the area signal processing device (170z) may be equipped with a plurality of application processor cores (CRR1 to CRRm) and an M core (MRb) for executing applications of ASIL D corresponding to the second safety level, which is the highest safety level.

[0244] Meanwhile, among the plurality of processor cores (CRR1 to CRRm, MRb) within the area signal processing device (170z), some (RR1 to CRRm) may execute an operating system (806b) corresponding to a first safety level such as ASIL B, and may execute a virtual machine (830b) corresponding to the first safety level on the operating system (806a).

[0245] Meanwhile, a virtual machine (830b) corresponding to the first safety level can execute an application corresponding to the first safety level, such as ASIL B, or a microservice (830ba to 830bd) corresponding to the application corresponding to the first safety level. Accordingly, the application or microservice (830ba to 830bd) corresponding to the first safety level can be stably performed.

[0246] Meanwhile, among the plurality of processor cores (CRR1 to CRRm, MRb) within the area signal processing device (170z), another part (MRb) may execute an operating system (806a) corresponding to a second safety level such as ASIL D, and may execute a virtual machine (840b) corresponding to a second safety level such as ASIL D on the operating system (806a).

[0247] Meanwhile, a virtual machine (840b) corresponding to the second safety level can execute an application corresponding to the second safety level, such as ASIL D, or a microservice (843b) corresponding to the application corresponding to the second safety level. Accordingly, the application or microservice (843b) corresponding to the second safety level can be stably performed.

[0248] FIG. 9 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0249] Referring to the drawings, a signal processing device (170) in a signal processing system (1000) according to one embodiment of the present disclosure includes a central processor (175) and at least one neural processor (179a to 179c).

[0250] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may further include a graphics processor (178).

[0251] Meanwhile, the central processor (175) according to the embodiment of the present disclosure executes a hypervisor (505).

[0252] Meanwhile, a system (1100) driven by a signal processing device (170) according to an embodiment of the present disclosure executes multiple virtual machines (810, 830, 850) on a hypervisor (505).

[0253] Specifically, a central processor (175) in a signal processing device (170) according to an embodiment of the present disclosure executes a hypervisor (505) and executes a plurality of virtual machines (810, 830, 850) on the hypervisor (505).

[0254] Meanwhile, the central processor (175) in the signal processing device (170) according to the embodiment of the present disclosure executes an application for driving the vehicle.

[0255] Meanwhile, if the central processor (175) determines that the application has failed to operate, it controls a second application corresponding to the application to be executed in another central processor or another signal processing device, and varies the standard fallback guarantee time for the application's operation failure based on the safety level of the application.

[0256] Accordingly, applications for vehicle operation can be performed reliably. In particular, applications for vehicle operation can be performed reliably based on safety levels.

[0257] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a shared memory (508).

[0258] In the drawing, it is illustrated that a hypervisor (505) is executed on a central processor (175) and a shared memory (508) is executed within the hypervisor (505).

[0259] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may receive data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (196), and perform signal processing using at least one of a central processor (175), a graphic processor (178), and a plurality of neural processors (179a to 179c).

[0260] Meanwhile, the sensor device (700) can continuously output sensor data to the signal processing device (170) during vehicle operation.

[0261] The sensor data at this time is data from sensor devices (700) of various vehicles, and may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle internal temperature data, and vehicle internal humidity data.

[0262] Meanwhile, the camera device (195) can continuously output camera data to the signal processing device (170) during vehicle operation.

[0263] Meanwhile, Lidar (196) can continuously output Lidar data to a signal processing device (170) during vehicle operation.

[0264] Meanwhile, the neural processor (179) can detect an object based on camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0265] Meanwhile, the neural processor (179) can receive camera data at a fixed frame rate, detect an object based on the camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0266] Meanwhile, among the multiple virtualization machines (810, 830, 850), the first virtualization machine (810), which is a server virtualization machine, controls the operation of the neural processor (179).

[0267] Meanwhile, among the multiple virtualization machines (810, 830, 850), the second virtualization machine (850) and the third virtualization machine (830), which are guest virtualization machines, can each execute an application.

[0268] In the drawing, the second virtual machine (850) is illustrated as executing an ADAS (Advanced Driver Assistance Systems) application (Nad) or an autonomous driving application or a Driver monitoring system (DMS) application (Ndm), and the third virtual machine (830) is illustrated as executing an augmented reality (AR) application (Nar).

[0269] The first virtual machine (810) sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications running on at least one of the plurality of virtual machines (810, 830, 850), and if the first operation and the third operation can be processed in parallel, the first neural processor (179a) controls the first operation and the third operation to be processed in parallel, and controls the second operation to be processed after the first operation and the third operation are completed. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0270] Meanwhile, if the first virtual machine (810) receives a request for a fourth operation after the third operation has been requested and sharing of the operation layers between the second and fourth operations is possible, the first neural processor (179a) can be controlled to sequentially process the second and fourth operations after the first and third operations are completed. Accordingly, the neural processor can be operated efficiently.

[0271] Meanwhile, the first virtual machine (810) can control the arrangement of data for multiple operations within the internal memory (1805) of the first neural processor (179a) to vary when multiple operations are requested from multiple applications. Accordingly, the neural processor can be operated efficiently.

[0272] Meanwhile, the first virtual machine (810) can execute a neural system service (1110) to control at least one neural processor (179a to 179c).

[0273] Meanwhile, the neural system service (1110) can control the arrangement of data for multiple operations within the internal memory (1805) of the first neural processor (179a) to vary when multiple operations are requested from multiple applications. This enables efficient operation of the neural processor.

[0274] Meanwhile, the neural system service (1110) may execute or be equipped with a neural manager (1113) for managing at least one neural processor (179a to 179c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (179a to 179c), and a neural interface (1118) for interfacing with at least one neural processor (179a to 179c).

[0275] Meanwhile, the neural system service (1110) may further execute or include a model container (509) that performs interface of model parameters related to the operation of the neural processor (179) and version management of learning files.

[0276] The neural manager (1113) can perform artificial intelligence model management, learning model management, camera data management, sensor data management, or command queue management.

[0277] The neural controller (1115) can determine an optimal inference method of at least one neural processor (179a to 179c), perform queue, partition, caching, or scalable coding, or control at least one neural processor (179a to 179c).

[0278] The neural interface (1118) can execute an application program interface (API) related to the accelerator of at least one neural processor (179a-179c).

[0279] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing between the neural system service (1110) and the model container (509) or the neural system service (1110) and the shared memory (508).

[0280] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing to the first virtual machine (810).

[0281] Meanwhile, the interface (522) within the first virtual machine (810) can perform interfacing to a vehicle driving assistance application (Nad) or a driver monitoring system application (Ndm) running within the second virtual machine (850) or an augmented reality application (Nar) running within the third virtual machine (830).

[0282] For example, the interface (522) within the first virtual machine (810) can be controlled to transmit camera data, sensor data, or voice data to the neural processor (179) using the shared memory (508).

[0283] Meanwhile, the interface (522) within the first virtual machine (810) can control the transmission of result data output from the neural processor (179) and recorded in the shared memory (508) to the neural system service (1110).

[0284] Meanwhile, the interface (522) within the first virtual machine (810) can control the transmission of result data output from the neural processor (179) and recorded in the shared memory (508) to a vehicle driving assistance application (Nad) or a driver monitoring system application (Ndm) running within the second virtual machine (850) or an augmented reality application (Nar) running within the third virtual machine (830).

[0285] Meanwhile, the first virtual machine (810) may be executed based on the first operating system (805), the second virtual machine (850) may be executed based on the second operating system (805b) with a high safety level, and the third virtual machine (830) may be executed based on the third operating system (805c).

[0286] That is, multiple virtual machines (810, 830, 850) may be run based on different operating systems, or may be run based on at least two operating systems.

[0287] Meanwhile, the second operating system (805b) may be an operating system corresponding to a second safety level, such as ASIL D, and the third operating system (805c) may be an operating system corresponding to a first safety level, such as ASIL B.

[0288] Meanwhile, the first operating system (805) may be an operating system corresponding to a second safety level, such as ASIL D. However, the present invention is not limited thereto, and the first operating system (805) may also be an operating system corresponding to a first safety level, such as ASIL B.

[0289] Meanwhile, the neural manager (1113) can manage the operating requirements of an artificial neural network-based application, control neural network weight data, and process necessary input data.

[0290] Meanwhile, the neural manager (1113) can sequentially process the optimized command queue through a hardware accelerator and transmit the operation result to the application.

[0291] Driving requirements can include the computational priorities, dependencies, and accuracy of a neural network. Operational priorities are preset values, such as whether the first operation should always be processed before the second, or, in the case of a safety-critical neural network, whether it should be processed first in the command queue before other candidate neural networks.

[0292] Meanwhile, neural network weight data refers to a file in which the element values ​​of each matrix are structured and stored in the process of inferring the results of a neural network calculated through a series of matrix operations.

[0293] Neural network weight data can be stored in advance as a model container (509) within the neural system service (1110) through an API call of the neural system service (1110) during the application installation process.

[0294] Meanwhile, the basic weight data loaded into the model container (509) can be automatically converted and stored in various discretization levels during the system initialization process. For example, if the basic weight is defined as FP32, it can be sub-discretized into INT8, INT16, and FP16, and a total of four weight files can be stored.

[0295] Required input data refers to input signals required for the current neural network to operate, such as vehicle speed, current location, radar, lidar, camera images, and intermediate or final calculation results of the preceding neural network.

[0296] The input data can be transmitted in real time to the shared memory (508) within the hypervisor (505) through an interface operating through the central processor (175) in the server virtualization machine.

[0297] A command queue is a memory buffer of a sequential FIFO data structure that can define a series of orders for processing artificial neural networks through hardware accelerators.

[0298] A single neural network operation request entering the command queue can be transmitted along with metadata such as the application name, the location of the application virtualization machine, the storage destination of the operation result, hardware accelerator control settings, the memory location of the input data, and the memory location information for each discretization level of the weight data.

[0299] The hardware accelerator control settings may include a unique number of the hardware accelerator in charge of the operation, the current target discretization level of the weight data (INT8, INT16, FP16, FP32, etc.), and a target neural network weight location mapping table for each hardware accelerator internal memory address.

[0300] Meanwhile, the neural controller (1115) can schedule an optimized command queue based on the requested artificial neural network operation commands and the availability of current hardware resources, and control the actual hardware accelerator to match the expected operation of the command queue.

[0301] The neural controller (1115) can receive neural network operation requirements from the neural manager (1113) and optimize the command queue.

[0302] The optimization process, in other words, can be calculated through a simulated scheduling that checks the priority, dependency, and accuracy metadata for each slot in the current command queue, and applies various queue optimization techniques (such as Partition, Caching, and Accuracy Coding) to all candidate commands in the current command queue, and finds a combination that maximizes hardware utilization and minimizes the latency of individual operation requests within a unit of time.

[0303] Based on the optimal slot location obtained in this way, a weight file (learning model) can be requested from the neural manager (1113) and loaded into the hardware internal memory.

[0304] If two different neural networks are managed as a single virtual neural network using Partition among optimization techniques and input as hardware operation requests, the start and end positions of the weights of the first operation corresponding to the address of the hardware internal memory can be recorded in a mapping table, and then the start and end positions of the weights of the second operation can be recorded in the mapping table.

[0305] Through this, the hardware accelerator performs the process of parallel processing of a virtual neural network, but the neural controller (1115) can separate the results of the operation into the results of the first operation and the second operation through a mapping table and transmit them separately to individual applications.

[0306] After completing the above initialization process, the neural controller (1115) can receive a sequential processing request of the command queue from the neural manager (1113).

[0307] At this time, the neural controller (1115) can be controlled to extract input data prepared in advance by the neural manager (1113) from the input data queue, pair the neural network weights with the corresponding input data, and perform computational processing through the hardware accelerator API.

[0308] If, unlike the initial driving requirements, the discretization level of the current neural network is changed according to a specific situation, the neural controller (1115) can perform a bitwise concanate operation that concatenates the weight conversion difference value (Delta) of the hardware internal memory to the basic weight of the current internal memory, thereby converting the discretization level of the basic weight of the internal memory in real time.

[0309] Meanwhile, the central processor (175) executes an application for driving the vehicle, and when it is determined that the application has failed to operate, it controls a second application corresponding to the application to be executed in another central processor (175) or another signal processing device (170), and varies the standard fallback guarantee time for the application's operation failure based on the safety level of the application.

[0310] Meanwhile, the above-described fallback guarantee time may mean the time from the fallback start time to the fallback end time.

[0311] Alternatively, the fallback guarantee time may mean the time from when the application determines that the operation has failed or when a failure has been determined to the time when the fallback starts to the time when the fallback ends.

[0312] Meanwhile, the safety level may mean the Automotive Safety Integrity Level (ASIL), the autonomous driving level, or a combination of the Automotive Safety Integrity Level and the autonomous driving level.

[0313] Accordingly, applications for vehicle operation can be performed reliably. In particular, applications for vehicle operation can be performed reliably based on safety levels.

[0314] Meanwhile, the central processor (175) may set the reference fallback guarantee time to a first time period when the safety level of the application is the corresponding first safety level, and may set the reference fallback guarantee time to a second time period longer than the first time period when the safety level of the application is the second safety level higher than the first safety level. Accordingly, the application for vehicle driving can be stably performed.

[0315] For example, the central processor (175) may set the standard fallback guarantee time to a first time, approximately 10 seconds, for a first application corresponding to ASIL D when the autonomous driving level is Level 3, and may set the standard fallback guarantee time to a second time, approximately 30 seconds, for a second application corresponding to ASIL D when the autonomous driving level is Level 4. Accordingly, it is possible to stably perform applications for driving a vehicle based on the safety level.

[0316] As another example, the central processor (175) may set the standard fallback guarantee time to approximately 7 seconds for a third application corresponding to ASIL B when the autonomous driving level is Level 3, and may set the standard fallback guarantee time to approximately 10 seconds for a fourth application corresponding to ASIL D when the autonomous driving level is Level 3. Accordingly, it is possible to stably perform applications for driving the vehicle based on the safety level.

[0317] As another example, the central processor (175) may set the reference fallback guarantee time to a first time period of approximately 10 seconds when the safety level of the augmented reality application (Nar) is the first safety level corresponding to ASIL B, and may set the reference fallback guarantee time to a second time period of approximately 30 seconds when the safety level of the vehicle driving assistance application (Nad) is the second safety level corresponding to ASIL D, which is higher than ASIL B. Accordingly, it is possible to stably perform an application for vehicle driving based on the safety level.

[0318] Meanwhile, the central processor (175) can set the standard fallback guarantee time to a third time shorter than the first time when the safety level of the application is a third safety level lower than the first safety level. Accordingly, the application for vehicle driving can be stably performed.

[0319] For example, the central processor (175) can set the standard fallback guarantee time to approximately 1 second, which is the third time, for the fifth application corresponding to ASIL D or ASIL B when the autonomous driving level is Level 2.

[0320] As another example, the central processor (175) may set the standard fallback guarantee time to approximately 0.7 seconds for the sixth application corresponding to QM when the autonomous driving level is Level 2.

[0321] As another example, the central processor (175) may set the standard fallback guarantee time to the third time, approximately 0.5 seconds, when the safety level of the augmented reality application (Nar) corresponds to a QM lower than ASIL B. Accordingly, the application for driving the vehicle can be stably performed based on the safety level.

[0322] Meanwhile, the central processor (175) can control the standard fallback guarantee time of the application executed in the second virtual machine (850) among the plurality of virtual machines (810, 830, 850) to be greater than the standard fallback guarantee time of the application executed in the third virtual machine, when the second virtual machine (850) executes an application with a higher security level than the third virtual machine (830).

[0323] For example, the central processor (175) can be set so that when the second virtual machine (850) executes a first application with an autonomous driving level of Level 4, the standard fallback guarantee time is approximately 30 seconds, and when the third virtual machine (830) executes a second application with an autonomous driving level of Level 3, the standard fallback guarantee time is approximately 10 seconds. Accordingly, it is possible to stably perform applications for vehicle driving based on a safety level.

[0324] As another example, the central processor (175) may set the standard fallback guarantee time of the vehicle driving assistance application (Nad) to be approximately 30 seconds when the second virtualization machine (850) executes a vehicle driving assistance application (Nad) corresponding to ASIL D, and may set the standard fallback guarantee time of the augmented reality application (Nar) to be approximately 10 seconds when the third virtualization machine (830) executes an augmented reality application (Nar) corresponding to ASIL B. Accordingly, it is possible to stably perform applications for vehicle driving based on a safety level.

[0325] FIGS. 10A to 10C are drawings for reference in explaining the operation of a vehicle control device related to the present disclosure.

[0326] Figure 10a illustrates an example of vehicle driving based on lane keeping mode.

[0327] In particular, FIG. 10a illustrates a vehicle (200) driving between the first lane (LNa) and the second lane (LNb) based on a lane maintenance mode, and a large truck (OBm) driving in the adjacent lane.

[0328] Referring to the drawings, a vehicle control device related to the present disclosure can detect lanes (LNa, LNb) based on images from a camera, and, when performing lane keeping mode, control to maintain a constant state at the center of both lanes (LNa, LNb) or a first gap (DPa) with the first lane (LNa).

[0329] When the vehicle (200) is located in the center of both lanes (LNa, LNb), the distance between the center of the vehicle (200) and the first lane (LNa) may be DPm.

[0330] Meanwhile, when a lane keeping mode is performed while a large truck (OBm) is driving between the second lane (LNb) and the third lane (LNc), which are adjacent lanes, the vehicle control device related to the present disclosure maintains a constant state at a first interval from the center of both lanes (LNa, LNb) or the first lane (LNa).

[0331] However, if the lane spacing is maintained at a constant level according to this lane keeping mode, the driver will feel uneasy due to the large truck (OBm).

[0332] Figure 10b illustrates another example of vehicle driving based on lane keeping mode.

[0333] In particular, FIG. 10b illustrates a vehicle (200) driving between a first lane (LNa) and a second lane (LNb) based on a lane maintenance mode, and a barrier or guard rail (RBm) is positioned next to the first lane (LNa).

[0334] Referring to the drawings, a vehicle control device related to the present disclosure maintains a constant state at a first interval from the center of both lanes (LNa, LNb) or the first lane (LNa) when a lane maintenance mode is performed while a barrier or guardrail (RBm) is positioned next to the first lane (LNa).

[0335] However, if the lane spacing is maintained at a constant level according to this lane keeping mode, the driver feels uneasy due to the barrier or guardrail (RBm).

[0336] Figure 10c illustrates another example of vehicle driving based on lane keeping mode.

[0337] In particular, FIG. 10c illustrates a vehicle (200) driving between the first lane (LNa) and the second lane (LNb) based on a lane maintenance mode, and an emergency vehicle (200f) approaches from behind the vehicle (200).

[0338] Specifically, a plurality of vehicles (200mb, 200mc, 200md) are driven at the rear of a vehicle (200), and an emergency vehicle (200f) is positioned between the vehicle (200) and the plurality of vehicles (200mb, 200mc, 200md).

[0339] Multiple vehicles (200mb, 200mc, 200md) can adjust their direction of travel to increase the distance between vehicles for the progress of the emergency vehicle (200f).

[0340] Meanwhile, the vehicle control device related to the present disclosure maintains a constant state at a first distance from the center of the two lanes (LNa, LNb) or the first lane (LNa) when the lane maintenance mode is performed while an emergency vehicle (200f) is approaching the rear of the vehicle (200).

[0341] However, if the lane spacing is maintained at a constant level according to this lane maintenance mode, there is an inconvenience that there is no lane spacing adjustment for emergency vehicles (200f).

[0342] Accordingly, in the present disclosure, the lane maintenance mode is subdivided so that, according to the first mode of the lane maintenance mode, the lane spacing is maintained at a constant level, and then, in consideration of the vehicle driving conditions or the driver's condition, the second mode of the lane maintenance mode adaptively adjusts the lane spacing. This is described with reference to FIG. 11a and below.

[0343] FIG. 11A is a flowchart illustrating an operation method of a signal processing device according to one embodiment of the present disclosure.

[0344] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure receives a front image from a front camera (195a) and detects a lane based on the front image (S1105).

[0345] Meanwhile, the processor (175) can detect a lane based on location information such as a side image, map information from a memory (140), GPS from a communication unit (120), etc., in addition to a front image from a front camera (195a).

[0346] Meanwhile, the processor (175) can detect various objects, such as surrounding vehicles, signs, pedestrians, traffic lights, and road borders, in addition to lanes, based on a front image or side image from the front camera (195a).

[0347] Meanwhile, the processor (175) can detect various objects such as surrounding vehicles, signs, pedestrians, traffic lights, and road borders based on location information such as map information from the memory (140) and GPS from the communication unit (120), in addition to the front image or side image from the front camera (195a).

[0348] Next, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can be controlled to perform a lane maintenance mode based on lane detection (S1110).

[0349] Meanwhile, the processor (175) in the signal processing device (170) determines whether the lane maintenance mode is the first mode among the lane maintenance modes (S1115), and if so, can perform the first mode among the lane maintenance modes.

[0350] That is, the processor (175) within the signal processing device (170) performs a lane maintenance mode based on lane detection, and controls the steering drive unit (752) to maintain a first gap (DPa) between the first lane (LNa) and the second lane (LNb) among the adjacent first lanes (LNa) and second lanes (LNb) according to the first mode among the lane maintenance modes (S1120).

[0351] Meanwhile, in step 1115 (S1115), if it is not the first mode among the lane maintenance modes, the processor (175) in the signal processing device (170) determines whether it is the second mode among the lane maintenance modes (S1125), and if so, performs the second mode among the lane maintenance modes.

[0352] That is, the processor (175) within the signal processing device (170) performs a lane maintenance mode based on lane detection, and controls the steering drive unit (752) to maintain the first lane (LNa) and a second interval (DPb) different from the first interval (DPa) according to the second mode among the lane maintenance modes (S1130).

[0353] Accordingly, the lane spacing can be adjusted adaptively to reflect the driving situation during lane keeping mode operation.

[0354] FIG. 11b is a flowchart showing an operation method of a signal processing device according to another embodiment of the present disclosure.

[0355] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure receives a front image from a front camera (195a) and detects a lane based on the front image (S1105).

[0356] Next, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can be controlled to perform a lane maintenance mode based on lane detection (S1110).

[0357] Meanwhile, the processor (175) in the signal processing device (170) determines whether the lane maintenance mode is the first mode among the lane maintenance modes (S1115), and if so, can perform the first mode among the lane maintenance modes.

[0358] That is, the processor (175) within the signal processing device (170) performs a lane maintenance mode based on lane detection, and controls the steering drive unit (752) to maintain the center of the adjacent first lane (LNa) and second lane (LNb) according to the first mode among the lane maintenance modes (S1120b).

[0359] Meanwhile, in step 1115 (S1115), if it is not the first mode among the lane maintenance modes, the processor (175) in the signal processing device (170) determines whether it is the second mode among the lane maintenance modes (S1125), and if so, performs the second mode among the lane maintenance modes.

[0360] That is, the processor (175) within the signal processing device (170) performs a lane maintenance mode based on lane detection, and controls the steering drive unit (752) to get closer to one of the first lane (LNa) and the second lane (LNb) according to the second mode among the lane maintenance modes (S1130b).

[0361] Accordingly, the lane spacing can be adjusted adaptively to reflect the driving situation during lane keeping mode operation.

[0362] Meanwhile, the processor (175) in the signal processing device (170) can control the steering drive unit (752) to maintain a first gap (DPa) with the first lane (LNa) in order to maintain the center of the adjacent first lane (LNa) and second lane (LNb) according to the first mode among the lane maintenance modes.

[0363] Meanwhile, the processor (175) within the signal processing device (170) can control the steering drive unit (752) to maintain a second gap (DPb) between the first lane (LNa) and the second lane (LNb) in order to bring the vehicle closer to one of the first lane (LNa) and the second lane (LNb) according to the first mode among the lane maintenance modes. At this time, the second gap (DPb) is preferably different from the first gap (DPa).

[0364] Accordingly, the lane spacing can be adjusted adaptively to reflect the driving situation during lane keeping mode operation.

[0365] Figures 12a to 18 are drawings for reference in the operation description of Figures 11a to 11b.

[0366] First, Fig. 12a illustrates an example of the first mode among the lane maintenance modes.

[0367] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure performs lane detection based on a front image from a camera (195), and performs a lane maintenance mode based on the lane detection.

[0368] In particular, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure controls a steering drive unit (752) to maintain a first gap (DPa) between the first lane (LNa) and the second lane (LNb) among adjacent first lanes (LNa) and second lanes (LNb), according to a first mode among lane maintenance modes.

[0369] Alternatively, the processor (175) in the signal processing device (170) according to another embodiment of the present disclosure controls the steering drive unit (752) to maintain the center of the adjacent first lane (LNa) and second lane (LNb) according to the first mode among the lane maintenance modes.

[0370] Accordingly, the vehicle (200) drives in a direction (DRa) that maintains the center of the adjacent first lane (LNa) and second lane (LNb).

[0371] Meanwhile, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure may control the first mode of the lane maintenance mode to be performed when there is no adjacent vehicle, no barrier or guard rail (RBm), or no emergency vehicle (200f) around the vehicle based on the front image, side image, or rear image from the camera (195).

[0372] Figure 12b illustrates that the driver's gaze is positioned on the vehicle ahead (OBk).

[0373] Referring to the drawing, the processor (175) within the signal processing device (170) can detect the direction of the driver's (OWa) face or the direction of the driver's (OWa) gaze based on the internal image from the internal camera (195i).

[0374] Meanwhile, the processor (175) within the signal processing device (170) can set the forward gaze level to the first level when the driver's (OWa) gaze direction (OPa) is located at the front vehicle (OBk), as shown in FIG. 12b.

[0375] Meanwhile, the processor (175) within the signal processing device (170) can control the first mode of the lane maintenance mode to be performed when the driver's (OWa) gaze direction (OPa) is located at the front vehicle (OBk), or when the forward gaze level is equal to or higher than the reference level as the first level. Accordingly, the first mode of the lane maintenance mode can be performed, as shown in FIG. 12a.

[0376] Meanwhile, the processor (175) within the signal processing device (170) can set the forward gaze level to a second level lower than the first level when, unlike in FIG. 12b, the driver's (OWa) gaze direction (OPa) is not located at the front vehicle (OBk), but at the side of the vehicle, etc.

[0377] Meanwhile, the processor (175) in the signal processing device (170) can control the second mode among the lane maintenance modes to be performed when the driver's (OWa) gaze direction (OPa) is not located at the front vehicle (OBk), or when the forward gaze level is lower than the reference level as the second level.

[0378] Next, Fig. 12c illustrates an example of the second mode among the lane maintenance modes.

[0379] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure performs lane detection based on a front image from a camera (195), and performs a lane maintenance mode based on the lane detection.

[0380] Meanwhile, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can detect a surrounding vehicle object based on a front image or a side image from a camera (195).

[0381] For example, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the second mode among the lane maintenance modes to be performed when a second vehicle (OBm) is located in an adjacent lane, as shown in the drawing.

[0382] As another example, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure may control the execution of a second mode among lane maintenance modes when a second vehicle (OBm) is located in an adjacent lane and the size of the second vehicle (OBm) is greater than or equal to a reference size.

[0383] As another example, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure may control the second mode among the lane maintenance modes to be performed when a second vehicle (OBm) is located in an adjacent lane and the line of sight (OPa) of the driver (OWa) is located toward the second vehicle (OBm).

[0384] As another example, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure may control the second mode among the lane keeping modes to be performed when a second vehicle (OBm) is located in an adjacent lane, the line of sight (OPa) of the driver (OWa) is located toward the second vehicle (OBm), and the emotion of the driver (OWa) is surprise or fear.

[0385] Meanwhile, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure controls the steering drive unit (752) to maintain a second gap (DPb) smaller than the first gap (DPa) between the first lane (LNa) and the second lane (LNb) among the adjacent first lanes (LNa) and second lanes (LNb) according to the second mode among the lane maintenance modes. Accordingly, the lane gap can be adaptively adjusted to reflect the driving situation during the lane maintenance mode operation.

[0386] Alternatively, the processor (175) in the signal processing device (170) according to another embodiment of the present disclosure controls the steering drive unit (752) to move closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb) according to the second mode among the lane maintenance modes.

[0387] That is, the processor (175) in the signal processing device (170) according to another embodiment of the present disclosure can control the distance between the center of the vehicle (200) and the first lane (LNa) to be DPma, which is smaller than DPm, according to the second mode among the lane maintenance modes.

[0388] Specifically, when a second vehicle (OBm) is located on one side, the processor (175) within the signal processing device (170) can control the vehicle (200) to move to the other side, which is opposite to the one side, while maintaining the lane in which it is driving, taking into consideration the driver's (OWa) gaze direction (OPa), emotional state, etc.

[0389] Ultimately, the processor (175) within the signal processing device (170) according to one embodiment of the present disclosure can control the vehicle's driving direction to be DRb rather than DRa, depending on the second mode among the lane maintenance modes. Accordingly, the lane spacing can be adaptively adjusted to reflect the driving situation during lane maintenance mode operation.

[0390] Meanwhile, the processor (175) can detect a second vehicle (OBm) on the side of the vehicle based on a front image or a side image from the camera (195), and control the second gap (DPb) in the second mode among the lane maintenance modes to be varied based on the size of the second vehicle (OBm).

[0391] For example, the processor (175) can control the second gap (DPb) to become smaller as the size of the second vehicle (OBm) increases, or the DPma to become smaller so as to come closer to the first lane (LNa). Accordingly, the lane gap can be adaptively adjusted based on the size of the second vehicle (OBm).

[0392] Meanwhile, the processor (175) can control the second interval (DPb) to vary based on the speed of the vehicle (200) in the second mode among the lane maintenance modes.

[0393] For example, the processor (175) can control the second gap (DPb) to become smaller or closer to the first lane (LNa) as the speed of the vehicle (200) increases. Accordingly, the lane gap can be adaptively adjusted based on the speed of the vehicle (200).

[0394] Meanwhile, the processor (175) can vary the second interval (DPb) in the second mode among the lane maintenance modes based on the driving skill of the driver (OWa) or the number of passengers in the vehicle (200) or whether the passenger seat is occupied.

[0395] For example, the processor (175) can control the second gap (DPb) to become smaller or closer to the first lane (LNa) as the driving skill of the driver (OWa) decreases or the number of passengers in the vehicle (200) increases. Accordingly, the lane gap can be adaptively adjusted based on the driving skill or the number of passengers.

[0396] As another example, the processor (175) can control the second gap (DPb) to be smaller or closer to the first lane (LNa) when the passenger is seated in the passenger seat than when the passenger is not seated in the passenger seat. Accordingly, the lane gap can be adaptively adjusted based on whether the passenger is seated in the passenger seat.

[0397] Meanwhile, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure, when performing the second mode among the lane maintenance modes, can detect pedestrians on the side of the vehicle based on the front image or the side image from the camera (195), and control the second interval (DPb) to vary based on the number of pedestrians detected or the positions of the pedestrians.

[0398] For example, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the second gap (DPb) to become smaller or closer to the first lane (LNa) as the number of pedestrians approaching the second lane (LNb) among the first lane (LNa) and the second lane (LNb) increases or as the distance to the second lane (LNb) becomes closer. Accordingly, the lane gap can be adaptively adjusted based on the detected pedestrians.

[0399] Meanwhile, the processor (175) can detect the driver's (OWa) gaze based on an internal image from an internal camera (195i), and vary the second interval (DPb) in the second mode among the lane maintenance modes based on the direction of the driver's (OWa) gaze.

[0400] For example, the processor (175) within the signal processing device (170) according to one embodiment of the present disclosure can control the second gap (DPb) to become smaller or closer to the first lane (LNa) as the driver's (OWa) gaze direction moves away from the front vehicle (OBk) of FIG. 12b. Accordingly, the lane gap can be adaptively adjusted based on the driver's (OWa) gaze direction.

[0401] Meanwhile, the processor (175) can be controlled to detect the driver's (OWa) gaze based on an internal image from an internal camera (195i) in a manual driving mode other than a lane maintenance mode, detect a distance from the first lane (LNa) in the front image, perform learning based on the distance between the driver's (OWa) gaze and the first lane (LNa), and store the learning result in the memory (140).

[0402] Meanwhile, the processor (175) can set the second interval (DPb) based on the learning result when performing the second mode among the lane maintenance modes.

[0403] For example, in the manual driving mode, when the driver's (OWa) forward gaze level is the first level and the distance from the first lane (LNa) is smaller than DPa, the processor (175) can determine, through learning, that the driving pattern of the driver (OWa) is a first driving pattern that approaches the first lane (LNa) among the first lane (LNa) and the second lane (LNb).

[0404] Meanwhile, the processor (175) can set the second interval (DPb) when performing the second mode among the lane maintenance modes based on the first driving pattern according to the learning result.

[0405] As another example, the processor (175) can, in a manual driving mode, determine, through learning, that the driving pattern of the driver (OWa) is a second driving pattern that is closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb), when the forward gaze level of the driver (OWa) is a second level lower than the first level and the distance from the first lane (LNa) is smaller than DPa.

[0406] Meanwhile, the processor (175) can set the second interval to be smaller than the first driving pattern when performing the second mode of the lane maintenance mode based on the second driving pattern obtained through learning. Accordingly, the lane interval can be adaptively adjusted based on learning.

[0407] As another example, the processor (175) can, in a manual driving mode, determine, through learning, that the driving pattern of the driver (OWa) is a third driving pattern that is closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb), when the driver's (OWa) forward gaze level is the first level, the distance from the first lane (LNa) is smaller than DPa, and smaller than the first driving pattern.

[0408] Meanwhile, the processor (175) can set the second interval to be smaller than the first driving pattern when performing the second mode of the lane maintenance mode based on the third driving pattern obtained through learning. Accordingly, the lane interval can be adaptively adjusted based on learning.

[0409] Meanwhile, the processor (175) can be controlled to detect, in manual driving mode, the emotional information of the driver (OWa) based on an internal image from an internal camera (195i), detect the distance from the first lane (LNa) in the front image, perform learning based on the emotional information of the driver (OWa) and the distance from the first lane (LNa), and store the learning result in memory. Accordingly, the lane distance can be adaptively adjusted based on the emotional information of the driver (OWa).

[0410] For example, the processor (175) can detect the emotional information of the driver (OWa) at the second vehicle position on the side based on the eye movements or facial movements of the driver (OWa) in the manual driving mode.

[0411] Meanwhile, the processor (175), in the manual driving mode, when the fear level of the emotional information of the driver (OWa) at the second vehicle position on the side is the first level and the distance from the first lane (LNa) is smaller than DPa, can determine, through learning, that the driving pattern of the driver (OWa) is a first driving pattern that approaches the first lane (LNa) among the first lane (LNa) and the second lane (LNb).

[0412] Meanwhile, the processor (175) can set the second interval (DPb) when performing the second mode among the lane maintenance modes based on the first driving pattern according to the learning result.

[0413] As another example, the processor (175) can, in a manual driving mode, determine, by learning, that the driver's (OWa) driving pattern is a second driving pattern that is closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb), when the fear level of the driver's (OWa) emotional information at the second vehicle position on the side is a second level higher than the first level and the distance from the first lane (LNa) is smaller than DPa.

[0414] Meanwhile, the processor (175) can set the second interval to be smaller than the first driving pattern when performing the second mode of the lane maintenance mode based on the second driving pattern obtained through learning. Accordingly, the lane interval can be adaptively adjusted based on learning.

[0415] As another example, the processor (175) can, in a manual driving mode, determine, by learning, that the driving pattern of the driver (OWa) is a third driving pattern that is closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb), when the fear level of the emotional information of the driver (OWa) at the second vehicle position on the side is the first level, the distance from the first lane (LNa) is smaller than DPa, and smaller than the first driving pattern.

[0416] Meanwhile, the processor (175) can set the second interval to be smaller than the first driving pattern when performing the second mode among the lane maintenance modes based on the third driving pattern obtained through learning. Accordingly, the lane interval can be adaptively adjusted based on learning.

[0417] Meanwhile, the processor (175) can be controlled to detect, in a manual driving mode, the driver's (OWa) gaze direction and the driver's (OWa) emotion information based on an internal image from an internal camera (195i), detect the distance from the first lane (LNa) in the front image, perform learning based on the driver's (OWa) gaze direction information, the driver's (OWa) emotion information, and the distance from the first lane (LNa), and store the learning result in a memory. Accordingly, the lane distance can be adaptively adjusted based on the driver's (OWa) gaze direction information and emotion information.

[0418] Next, Fig. 12d illustrates another example of the second mode among the lane maintenance modes.

[0419] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can control the second mode among lane maintenance modes to be performed when a second vehicle (OBm) is located in an adjacent lane, as shown in the drawing.

[0420] Meanwhile, compared to FIG. 12c, the processor (175) can control the second distance from the first lane (LNa) to be DPc, which is larger than DPb, or the distance between the center of the vehicle (200) and the first lane (LNa) to be DPmb, which is larger than DPma, as the speed of the vehicle (200) decreases, or the driving skill of the driver (OWa) increases, or the number of passengers in the vehicle (200) decreases, or the number of pedestrians approaching the second lane (LNb) decreases.

[0421] At this time, DPc is preferably smaller than DPa, and DPmb is preferably smaller than DPm. Accordingly, the lane spacing can be adaptively adjusted to reflect driving conditions during lane keeping mode operation.

[0422] Finally, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the driving direction of the vehicle to be DRc rather than DRa, according to the second mode among the lane maintenance modes.

[0423] Meanwhile, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the second mode among the lane maintenance modes to be performed when a barrier or guard rail (RBm) or a second vehicle larger than the reference size is located on the side of the vehicle based on a front image or a side image from the camera (195).

[0424] A second mode of lane maintenance mode related to a barrier or guard rail (RBm) is described with reference to FIGS. 13a to 13d.

[0425] Figure 13a illustrates an example of a first mode among the lane maintenance modes.

[0426] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can control the first mode of the lane maintenance mode to be performed when there is no adjacent vehicle, no barrier or guard rail (RBm), or no emergency vehicle (200f) around the vehicle based on a front image or a side image from a camera (195).

[0427] Figure 13b illustrates that the driver's gaze is positioned at the barrier or guard rail (RBm).

[0428] Referring to the drawing, the processor (175) within the signal processing device (170) can detect the direction of the driver's (OWa) face or the direction of the driver's (OWa) gaze based on the internal image from the internal camera (195i).

[0429] Meanwhile, the processor (175) within the signal processing device (170) can control the second mode of the lane maintenance mode to be performed when the driver's (OWa) line of sight direction (OPb) is located at the barrier or guardrail (RBm), as shown in FIG. 13b. Accordingly, the second mode of the lane maintenance mode can be performed, as shown in FIG. 13c.

[0430] Meanwhile, the processor (175) within the signal processing device (170) can control the second mode among the lane maintenance modes to be performed when the driver's (OWa) line of sight direction (OPb) is located at the barrier or guardrail (RBm), as shown in FIG. 13b, and the driver's (OWa) emotion is surprise or fear.

[0431] Specifically, the processor (175) within the signal processing device (170) can control the vehicle (200) to move to the opposite side of the one side while maintaining the lane in which it is driving by considering the driver's (OWa) line of sight direction (OPa), emotional state, etc., when located at a barrier or guardrail (RBm) on one side.

[0432] Next, Figure 13c illustrates another example of the second mode among the lane maintenance modes.

[0433] Referring to the drawing, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure performs lane detection based on a front image from a camera (195), and performs a lane maintenance mode based on the lane detection.

[0434] Meanwhile, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can detect a surrounding vehicle object, a barrier, a guardrail, etc., based on a front image or a side image from a camera (195).

[0435] For example, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the second mode among the lane maintenance modes to be performed when a barrier or guard rail (RBm) is located on the side of the vehicle, as shown in the drawing.

[0436] Specifically, the processor (175) can control the vehicle (200) to come closer to the second lane (LNb) than to the first lane (LNa) when a barrier or guard rail (RBm) is located close to the first lane (LNa) among the first lane (LNa) and the second lane (LNb).

[0437] That is, the processor (175) can control, when a barrier or guardrail (RBm) is located close to the first lane (LNa) among the first lane (LNa) and the second lane (LNb), the second gap with the first lane (LNa) to be DPd, which is larger than DPa, or the gap between the center of the vehicle (200) and the first lane (LNa) to be DPna, which is larger than DPm. Accordingly, the lane gap can be adaptively adjusted to reflect the driving situation during the lane maintenance mode operation.

[0438] Finally, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the driving direction of the vehicle to be DRd rather than DRa, according to the second mode among the lane maintenance modes.

[0439] Next, Figure 13d illustrates another example of the second mode among the lane maintenance modes.

[0440] Referring to the drawing, Fig. 13d differs from Fig. 13c in that the second vehicle (OBm) is located on the side of the second lane (LNb).

[0441] Meanwhile, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure can vary the distance from the first lane (LNa) when a barrier or guard rail (RBm) is positioned close to the first lane (LNa) and a second vehicle (OBm) is positioned close to the second lane (LNb).

[0442] For example, if the processor (175) determines that the driver (OWa) is more afraid of the barrier or guardrail (RBm) than the second vehicle (OBm), the processor (175) can control the driver to get closer to the second lane (LNb) among the first lane (LNa) and the second lane (LNb), as shown in FIG. 13d.

[0443] As another example, if the processor (175) determines that the driver (OWa) feels more fearful of the second vehicle (OBm) than of the barrier or guardrail (RBm), unlike FIG. 13d, the processor (175) may control the driver to come closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb).

[0444] As shown in the drawing, when a barrier or guardrail (RBm) is positioned close to the first lane (LNa) and a second vehicle (OBm) is positioned close to the second lane (LNb), the processor (175) can control the second gap with the first lane (LNa) to be DPe, which is greater than DPa and smaller than DPd, or the gap between the center of the vehicle (200) and the first lane (LNa) to be DPnb, which is greater than DPm and smaller than DPna. Accordingly, the lane gap can be adaptively adjusted to reflect the driving situation during lane maintenance mode operation.

[0445] Finally, the processor (175) in the signal processing device (170) according to one embodiment of the present disclosure can control the driving direction of the vehicle to be DRe rather than DRa, according to the second mode among the lane maintenance modes.

[0446] Figure 14a illustrates another example of the second mode among the lane maintenance modes.

[0447] Referring to the drawing, the processor (175) can detect emotional information of the driver (OWa) based on an internal image from an internal camera (195i).

[0448] In addition, the processor (175) can control the second mode among the lane keeping modes to be performed when the driver's (OWa) emotional information is detected as surprise or fear.

[0449] For example, the processor (175) may control the center of the vehicle (200) to align with the center line (LNct) between the first lane (LNa) and the second lane (LNb) based on the first mode among the lane maintenance modes.

[0450] At this time, the distance between the center of the vehicle (200) and the first lane (LNa) and the second lane (LNb) may be DPm and DPm, respectively. That is, the distance between the center of the vehicle (200) and the first lane (LNa) and the distance between the center of the vehicle (200) and the second lane (LNb) may be the same.

[0451] Meanwhile, the processor (175) can control the execution of the second mode of the lane maintenance mode when the driver's (OWa) emotional information is surprise or fear during the execution of the first mode of the lane maintenance mode.

[0452] That is, the processor (175) can control the center of the vehicle (200) to come closer to the first lane (LNa) based on the left offset (OFa) according to the second mode among the lane maintenance modes.

[0453] Specifically, the processor (175) can control the distance between the center of the vehicle (200) and the first lane (LNa) to be DPc, which is smaller than DPm, according to the second mode among the lane maintenance modes.

[0454] In addition, the processor (175) can control the distance between the center of the vehicle (200) and the second lane (LNb) to be DPd, which is greater than DPm, according to the second mode among the lane maintenance modes. Accordingly, the lane distance can be adaptively adjusted based on the emotional information of the driver (OWa).

[0455] Meanwhile, in the drawing, the same offset is applied when the emotional information is surprise or fear, but the processor (175) can control the offset to be larger in the case of fear than in the case of surprise.

[0456] For example, in the case where the emotional information is surprise, the processor (175) preferably determines that the distance between the center of the vehicle (200) and the first lane (LNa) is DPc, and in the case where the emotional information is fear, the distance between the center of the vehicle (200) and the first lane (LNa) is preferably smaller than DPc.

[0457] Specifically, the processor (175) can control the vehicle (200) to approach the first lane (LNa) more closely when the emotional information is fear rather than surprise. Accordingly, the lane spacing can be adaptively adjusted based on the emotional information of the driver (OWa).

[0458] Figure 14b is a drawing illustrating the performance of the first mode among the lane maintenance modes and the second mode among the lane maintenance modes.

[0459] Referring to the drawing, the processor (175) can control the center of the vehicle (200) to align with the center line (LNct) between the first lane (LNa) and the second lane (LNb) based on the first mode among the lane maintenance modes.

[0460] At this time, the distance between the vehicle (200) and the first lane (LNa) and the second lane (LNb) may be DPda and DPdb, respectively. Meanwhile, DPda and DPdb may be at the same level.

[0461] Meanwhile, the processor (175) can control the vehicle to be closer to the first lane (LNa) among the first lane (LNa) and the second lane (LNb) according to the second mode among the lane maintenance modes.

[0462] Specifically, the processor (175) can control the distance between the vehicle (200) and the first lane (LNa) to be DPdc, which is smaller than DPda, according to the second mode among the lane maintenance modes.

[0463] In addition, the processor (175) can control the distance between the vehicle (200) and the second lane (LNb) to be DPdd, which is greater than DPdb, according to the second mode among the lane maintenance modes. Accordingly, the lane distance can be adaptively adjusted.

[0464] Figure 15a illustrates an example in which a barrier or guard rail (RBm) is positioned on the first side of the vehicle and a second vehicle is positioned on the second side.

[0465] Referring to the drawing, the processor (175) can control the vehicle to move closer to the first lane (LNa) or further away from the second lane (LNb) in order to maintain a distance from the second vehicle (OBm) according to the second mode among the lane maintenance modes.

[0466] As shown in the drawing, the processor (175) can control the distance DS2 between the vehicle (200) and the second lane (LNb) in the second mode of the lane maintenance mode to be greater than the distance between the vehicle (200) and the second lane (LNb) in the first mode of the lane maintenance mode. In this case, the distance between the vehicle (200) and the second vehicle (OBm) may be DS2b.

[0467] Meanwhile, the processor (175) can control, in accordance with the second mode among the lane maintenance modes, to perform separation control from the barrier or guard rail (RBm) when the barrier or guard rail (RBm) appears during the separation operation with the second vehicle (OBm).

[0468] That is, the processor (175) can control the distance between the vehicle (200) and the second lane (LNb) to be DS1, which is smaller than DS2, according to the second mode among the lane maintenance modes. At this time, the distance between the vehicle (200) and the second vehicle (OBm) may be DS1b.

[0469] Ultimately, the processor (175) can control the offset to be smaller when the object is located on both sides than when the object is located on only one side during the second mode of the lane maintenance mode. Accordingly, the vehicle movement is reduced when the object is located on both sides than when the object is located on only one side.

[0470] Figure 15b illustrates an example where a second vehicle is positioned on the first side of the vehicle.

[0471] Referring to the drawing, the processor (175) can control the vehicle to move closer to the third lane (LNc) or further away from the second lane (LNb) in order to keep a distance from the second vehicle (OBm) adjacent to the second lane (LNb) according to the second mode among the lane maintenance modes.

[0472] As shown in the drawing, the processor (175) can control the distance DS5 between the vehicle (200) and the second lane (LNb) in the second mode of the lane maintenance mode to be greater than the distance between the vehicle (200) and the second lane (LNb) in the first mode of the lane maintenance mode. In this case, the distance between the vehicle (200) and the second vehicle (OBm) may be DS5b.

[0473] Meanwhile, the processor (175) can control, in accordance with the second mode among the lane maintenance modes, to perform separation control with the third vehicle (200c) when a third vehicle (200c) adjacent to the third lane (LNc) appears during a separation operation with the second vehicle (OBm).

[0474] That is, the processor (175) can control the distance between the vehicle (200) and the second lane (LNb) to be DS4, which is smaller than DS5, according to the second mode among the lane maintenance modes. At this time, the distance between the vehicle (200) and the second vehicle (OBm) may be DS4b.

[0475] Ultimately, the processor (175) can control the offset to be smaller when the object is located on both sides than when the object is located on only one side during the second mode of the lane maintenance mode. Accordingly, the vehicle movement is reduced when the object is located on both sides than when the object is located on only one side.

[0476] Figure 16a illustrates the first mode being performed during the lane maintenance mode.

[0477] Referring to the drawing, the processor (175) can control the vehicle to maintain a constant state at the center of both lanes (LNa, LNb) or at the first gap (DPa) with the first lane (LNa) when performing the first mode among the lane maintenance modes.

[0478] That is, when the first mode among the lane maintenance modes is performed, the processor (175) can control the first lane (LNa) to be at a first gap (DPa), or control the center of the vehicle (200) to be at a gap DPm between the first lane (LNa).

[0479] Meanwhile, multiple vehicles (200mb, 200mc, 200md) positioned at the rear of the vehicle (200) can perform lane spacing adjustment for an emergency driving vehicle (200f).

[0480] The emergency vehicle (200f) at this time may be an ambulance, police car, fire truck, etc.

[0481] Meanwhile, the processor (175) can control the vehicle (200) to enter the second mode of the lane maintenance mode when an emergency vehicle (200f) approaches the rear of the vehicle (200) while the vehicle is performing the first mode of the lane maintenance mode.

[0482] Figure 16b illustrates the second mode of lane keeping mode being performed when an emergency vehicle approaches.

[0483] Referring to the drawing, the processor (175) can detect an emergency vehicle (200f) in the rear image based on the rear image from the camera (195).

[0484] Meanwhile, the processor (175) can detect an emergency vehicle (200f) in the rear image based on an ambulance sound or a fire truck sound from the rear of the vehicle, in addition to detecting an object in the rear image from the camera (195).

[0485] Alternatively, the processor (175) may detect an emergency vehicle (200f) in the rear image based on traffic information or map information related to a sound from the rear of the vehicle or a vehicle accident, in addition to object detection in the rear image from the camera (195).

[0486] Meanwhile, the processor (175) can control the second mode among the lane maintenance modes to be performed based on the detected emergency driving vehicle (200f).

[0487] That is, when the second mode of the lane maintenance mode is performed in response to an approaching emergency vehicle, the processor (175) can control the second gap (DPk) to be smaller than the first gap (DPa) between the first lane (LNa), or can control the center of the vehicle (200) to be smaller than the gap DPmk between the first lane (LNa) and DPm. Accordingly, the lane gap can be adaptively adjusted based on an emergency vehicle (200f) approaching from the rear of the vehicle.

[0488] Meanwhile, the processor (175) can control the second mode of the lane maintenance mode to be performed based on the emergency vehicle (200f) at the rear of the vehicle, and can control the distance from the first lane (LNa) to become smaller as the distance from the emergency vehicle (200f) becomes closer.

[0489] In particular, the processor (175) can control the distance from the first lane (LNa) to gradually decrease as the distance from the emergency vehicle (200f) gets closer. Accordingly, the lane distance can be adaptively adjusted based on the emergency vehicle (200f) approaching from behind the vehicle.

[0490] Figure 16c is a diagram illustrating lane spacing offset according to vehicle speed.

[0491] Referring to the drawing, the processor (175) can control the center of the vehicle (200) to align with the center line of the first lane (LNa) and the second lane (LNb) according to the lane maintenance mode while driving at the second speed.

[0492] Meanwhile, the processor (175) can control the center of the vehicle (200) to be closer to either the first lane (LNa) or the second lane (LNb) depending on the lane maintenance mode while driving at a first speed lower than the second speed.

[0493] In particular, the processor (175) can control the vehicle to match the center line, which is the center of the first lane (LNa) and the second lane (LNb), at the second speed, according to the first mode among the lane maintenance modes.

[0494] Meanwhile, the processor (175) can control the vehicle (200) to drive at a first speed lower than the second speed while performing the second mode of the lane maintenance mode, thereby bringing the center of the vehicle (200) closer to either the first lane (LNa) or the second lane (LNb). Accordingly, the lane spacing can be adaptively adjusted to reflect the driving situation during the lane maintenance mode operation.

[0495] Meanwhile, unlike the drawing, the processor (175) can control the second gap (DPb), which is the gap between the first lane (LNa) and the vehicle (200), to become smaller or closer to the first lane (LNa) as the speed of the vehicle (200) increases when the second mode of the lane maintenance mode is performed. Accordingly, the lane gap can be adaptively adjusted based on the speed of the vehicle (200).

[0496] Figure 17a illustrates an example of spacing adjustment with the first lane (LNa) based on approaching rearward vehicles.

[0497] Referring to the drawing, the processor (175) can control the distance from the first lane (LNa) to gradually decrease as the distance from the emergency vehicle (200f) gets closer.

[0498] In the drawing, the gap from the first lane (LNa) is shown as decreasing to zero (0) at Dsc.

[0499] Alternatively, the processor (175) may control the distance from the second lane (LNb) to gradually increase as the distance from the emergency vehicle (200f) gets closer.

[0500] In particular, the processor (175) exemplifies that when an emergency vehicle (200f) approaches the rear right side of the vehicle (200) despite the presence of a barrier or guardrail (RBm) on the left side of the vehicle (200), the distance from the second lane (LNb) increases from DSb to DSa. Accordingly, the lane distance can be adaptively adjusted based on the emergency vehicle (200f) approaching from the rear of the vehicle.

[0501] Figure 17b illustrates another example of spacing adjustment with the first lane (LNa) based on approaching rearward vehicles.

[0502] Referring to the drawing, the processor (175) can control the distance from the first lane (LNa) to gradually decrease based on the approach of the emergency vehicle (200f).

[0503] In particular, the drawing illustrates a vehicle (200) being positioned on the first lane (LNa).

[0504] Meanwhile, the processor (175) can adjust the distance from the barrier or guard rail (RBm) by taking the barrier or guard rail (RBm) into consideration when there is a barrier or guard rail (RBm) on the left side of the vehicle (200).

[0505] That is, the processor (175) can control the vehicle (200) to increase the distance from the barrier or guard rail (RBm) so that DSd becomes greater than DSe when the vehicle (200) is positioned on the first lane (LNa) and the distance from the barrier or guard rail (RBm) is DSe.

[0506] That is, the processor (175) can control the gap between the first lane (LNa) and the left side of the vehicle (200) to become zero (0) through gap adjustment in the DSf state. Accordingly, the lane gap can be adaptively adjusted to reflect the driving situation during lane maintenance mode operation.

[0507] Figure 17c illustrates another example of spacing adjustment based on approaching rearward vehicles.

[0508] Referring to the drawing, the processor (175) can adjust the distance between the vehicle (200) traveling between the second lane (LNb) and the third lane (LNc) based on the approach of the emergency vehicle (200f).

[0509] For example, the processor (175) can control the gap between the vehicle (200) and the third lane (LNc) to become smaller when an emergency vehicle (200f) approaches from behind, a second vehicle (OBm) is located on the right, and a third vehicle (200b) is located on the left.

[0510] In the drawing, when an emergency vehicle (200f) approaches from the rear, a second vehicle (OBm) is located on the right, and a third vehicle (200b) is located on the left, the distance between the vehicle (200) and the third lane (LNc) is DSo, and the distance between the vehicle (200) and the second lane (LNb) is DSn.

[0511] At this time, it is preferable that DSn, which is the distance between the vehicle (200) and the second lane (LNb), is greater than DSo, which is the distance between the vehicle (200) and the third lane (LNc).

[0512] Meanwhile, the processor (175) can adjust the distance between the vehicle (200) and the third lane (LNc) based on the size of the vehicle located on the right or the distance from the vehicle.

[0513] For example, as the size of the vehicle located on the right side decreases or the distance from the vehicle increases, the distance between the vehicle (200) and the third lane (LNc) can be controlled to decrease, or the distance between the vehicle (200) and the second lane (LNb) can be controlled to increase.

[0514] In the drawing, the vehicle located on the right is the fourth vehicle (200c), which is smaller in size than the second vehicle (OBm), and the gap between the fourth vehicle (200c) and the third lane (LNc) is larger than the gap between the second vehicle (OBm) and the third lane (LNc).

[0515] Accordingly, the processor (175) exemplifies that when an emergency vehicle (200f) approaches from behind, a fourth vehicle (200c) is located on the right, and a third vehicle (200b) is located on the left, the distance between the vehicle (200) and the third lane (LNc) is zero (0), and the distance between the vehicle (200) and the second lane (LNb) is DSm, which is greater than DSn.

[0516] Meanwhile, the processor (175) can control the distance between the third vehicle (200b) on the left side to increase when an emergency vehicle (200f) approaches from the rear so that the emergency vehicle (200f) can pass, and then reduce the distance between the third vehicle (200b) after the emergency vehicle (200f) passes.

[0517] Meanwhile, the processor (175) may control the vehicle (200) to be temporarily positioned on the third lane (LNc) when the distance from the third vehicle (200b) on the left increases.

[0518] Figure 17d illustrates an example of spacing adjustment when the left vehicle is positioned.

[0519] Referring to the drawing, the processor (175) can control the vehicle to move closer to the second lane (LNb), which is the right lane among the first lane (LNa) and the second lane (LNb), when the third vehicle (200b) is located on the left.

[0520] That is, the processor (175) can control the vehicle (200) to be positioned on the second lane (LNb), which is the right lane, according to the first mode of the lane maintenance mode, while maintaining the first distance (DSs) with respect to the second lane (LNb), which is the right lane, according to the second mode of the lane maintenance mode, when a third vehicle (200b) is positioned on the left.

[0521] That is, the processor (175) can control the distance between the right side of the vehicle (200) and the second lane (LNb) to be DSr, according to the second mode among the lane maintenance modes.

[0522] Meanwhile, the processor (175) can control the vehicle (200) to move to the left when a barrier or guard rail (RBm) appears on the right after the vehicle (200) has moved to the right based on the third vehicle (200b) on the left.

[0523] That is, when a barrier or guardrail (RBm) appears on the right side while the vehicle (200) is driving on the second lane (LNb), the processor (175) can control the distance from the first lane (LNa) to become DSp. Accordingly, the lane distance can be adaptively adjusted based on the vehicle side situation.

[0524] FIG. 18 is an example of an internal block diagram of a signal processing device according to an embodiment of the present disclosure.

[0525] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure can receive a front image, an internal image, a sensing signal, or a rear image from a front camera (195a), an internal camera (195i), a lidar (196), or a rear camera (195r), respectively.

[0526] Meanwhile, the signal processing device (170) may include a detection unit (1510) that detects an object or a lane, etc. based on a received image or sensing signal, a motion estimation unit (1520) that estimates motion based on the detected object or lane, etc., a determination unit (1530) that determines vehicle control based on the estimated motion, and an application execution unit (1540) that executes an application based on the determined vehicle control.

[0527] Meanwhile, the detection unit (1510) may include an object detection unit (1512) that detects an object in front of the vehicle from a front image or a driver's face or eyes from an interior image, a lane detection unit (1514) that detects a lane in front of the vehicle from a front image, and a sensor fusion unit (1516) that synthesizes an image signal or a sensing signal.

[0528] Meanwhile, the motion estimation unit (1520) may include an ego motion estimation unit (1522) for estimating ego motion related to estimating the direction of travel of the vehicle (200), a predicted path estimation unit (1524) for estimating the predicted path of the vehicle (200), a gaze tracking estimation unit (1526) for tracking the driver's gaze, and an emotion information estimation unit (1528) for calculating the driver's emotion information.

[0529] Meanwhile, the decision unit (1530) may include a condition decision unit (1532) that determines a lane maintenance mode condition based on a signal from an ego motion estimation unit (1522), a prediction path estimation unit (1524), an eye tracking estimation unit (1526), ​​or an emotion information estimation unit (1528), a state machine decision unit (1534) that determines a lane maintenance state machine, and a mode decision unit (1536) that determines a first mode and a second mode among the lane maintenance modes.

[0530] Meanwhile, the application execution unit (1540) may include a notification application (1542) for the lane maintenance mode or an automatic steering control application (1544) for execution of the second mode among the lane maintenance modes, based on a signal from the condition determination unit (1532), the state machine determination unit (1534), or the mode determination unit (1536).

[0531] Meanwhile, the processor (175) within the signal processing device (170) can execute a driver monitoring application (Ndm) based on the internal image, as shown in FIG. 9, and execute an automatic steering control application (1544) based on the front image.

[0532] The driver monitoring application (Ndm) at this time may include multiple microservices.

[0533] For example, a driver monitoring application (Ndm) may include an object detection unit (1512) as a microservice, a gaze tracking estimation unit (1526) for tracking the driver's gaze, an emotion information estimation unit (1528), etc.

[0534] Meanwhile, the processor (175) within the signal processing device (170) can execute an object detection unit (1512), an eye tracking estimation unit (1526), ​​an emotion information estimation unit (1528), etc. corresponding to the second safety level, ASIL D.

[0535] To this end, the processor (175) within the signal processing device (170) can be controlled to execute an object detection unit (1512), an eye tracking estimation unit (1526), ​​an emotion information estimation unit (1528), etc., in a virtual machine (850) corresponding to a second safety level such as ASIL D. Accordingly, the driver monitoring application (Ndm) can be stably executed.

[0536] Meanwhile, the processor (175) within the signal processing device (170) can execute an automatic steering control application (1544).

[0537] The automatic steering control application (1544) at this time may include multiple microservices.

[0538] For example, an automatic steering control application (1544) may include an object detection unit (1512), a lane detection unit (1514), an ego motion estimation unit (1522), a predicted path estimation unit (1524), an eye tracking estimation unit (1526), ​​an emotion information estimation unit (1528), a condition determination unit (1532), a state machine determination unit (1534), or an automatic emergency steering mode determination unit (1536), which are microservices.

[0539] To this end, the processor (175) within the signal processing device (170) can be controlled to execute an object detection unit (1512), a lane detection unit (1514), an ego motion estimation unit (1522), a predicted path estimation unit (1524), an eye tracking estimation unit (1526), ​​an emotion information estimation unit (1528), a condition determination unit (1532), a state machine determination unit (1534), or an automatic emergency steering mode determination unit (1536) in a virtual machine (850) corresponding to a second safety level such as ASIL D. Accordingly, the automatic steering control application (1544) can be stably executed.

[0540] Meanwhile, the processor (175) within the signal processing device (170) can execute a lane detection application based on the front image.

[0541] At this time, the lane detection application may include multiple microservices.

[0542] That is, the processor (175) within the signal processing device (170) can execute a lane detection application including a plurality of micro-services based on the front image.

[0543] For example, a processor (175) within a signal processing device (170) may execute a lane detection application or multiple microservices for a lane detection application in a virtual machine (850) corresponding to a second safety level such as ASIL D.

[0544] Next, the processor (175) within the signal processing device (170) can execute a notification application (1542) for lane maintenance mode based on the internal image and the front image.

[0545] The notification application (1542) for the lane maintenance mode at this time can correspond to the first safety level, ASIL B, or QM.

[0546] To this end, the processor (175) within the signal processing device (170) can control the execution of a notification application (1542) in a virtual machine (830) corresponding to a first safety level such as ASIL B.

[0547] Meanwhile, the safety level of the notification application (1542) may be lower than the safety level of the driver monitoring application (Ndm) or the automatic steering control application (1544) or the lane detection application.

[0548] Meanwhile, the processor (175) executes a plurality of virtual machines (810, 830, 850) on the hypervisor (505), as shown in FIG. 9, and some of the virtual machines (850) among the plurality of virtual machines (810, 830, 850) can execute a driver monitoring application (Ndm) based on an internal image and an automatic steering control application (1544) based on a forward image. Accordingly, adaptive vehicle control can be performed according to the forward gaze level of the driver (OWa).

[0549] Meanwhile, among the plurality of virtual machines (810, 830, 850), some other virtual machines (830) execute a notification application (1542) for lane keeping mode, and the safety level of the notification application (1542) may be lower than the safety level of the driver monitoring application (Ndm) or the automatic steering control application (1544) or the lane detection application.

[0550] Meanwhile, among the plurality of virtual machines (810, 830, 850), some virtual machines (850) may execute multiple micro-services for an automatic steering control application (1544) based on a forward image and execute multiple micro-services for a driver monitoring application (Ndm) based on an internal image.

[0551] Accordingly, the automatic steering control application (1544) and driver monitoring application (Ndm) can be executed stably and efficiently. Furthermore, vehicle control can be efficiently performed using microservices.

[0552] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. Equipped with a processor that receives and processes a front image from a camera installed in the vehicle; The above processor, Lane detection is performed based on the front image from the above camera, and lane keeping mode is performed based on the lane detection. According to the first mode among the above lane maintenance modes, the steering drive unit is controlled to maintain a first distance from the first lane among the adjacent first and second lanes, A signal processing device that controls the steering drive unit to maintain a second interval different from the first interval with respect to the first lane, according to the second mode among the above lane maintenance modes.

2. In paragraph 1, The above processor, A signal processing device that detects an emergency vehicle in the rear image based on a rear image from the camera, and controls the second mode among the lane maintenance modes to be performed based on the emergency vehicle.

3. In paragraph 1, The above processor, A signal processing device that controls the second mode of the lane maintenance mode to be performed based on an emergency vehicle behind the vehicle, and controls the gap with the first lane to become smaller as the distance from the emergency vehicle becomes closer.

4. In paragraph 1, The above processor, A signal processing device that controls the second mode among the lane maintenance modes to be performed when a barrier or guardrail or a second vehicle larger than a reference size is located on the side of the vehicle based on the front image or side image from the camera.

5. In paragraph 1, The above processor, A signal processing device that detects a second vehicle on the side of the vehicle based on the front image or side image from the camera, and controls the second interval in the second mode among the lane maintenance modes to be varied based on the size of the second vehicle.

6. In paragraph 1, The above processor, A signal processing device that controls the second interval to vary based on the speed of the vehicle in the second mode among the above lane maintenance modes.

7. In paragraph 1, The above processor, A signal processing device that detects pedestrians on the side of the vehicle based on the front image or side image from the camera, and controls the second interval to be variable based on the number of pedestrians detected or the positions of the pedestrians.

8. In paragraph 1, The above processor, A signal processing device that varies the second interval in the second mode among the lane maintenance modes based on the driving skill of the driver, the number of passengers in the vehicle, or whether the passenger seat is occupied.

9. In paragraph 1, The above processor, Based on the internal image from the internal camera, the driver's gaze is detected, A signal processing device that varies the second interval in the second mode among the lane maintenance modes based on the driver's gaze direction.

10. In paragraph 1, The above processor, In manual driving mode, based on the internal image from the internal camera, the driver's line of sight is detected, and the distance from the first lane in the front image is detected, A signal processing device that performs learning based on the distance between the driver's line of sight and the first lane and controls the learning result to be stored in memory.

11. In paragraph 10, The above processor, A signal processing device that sets the second interval based on the learning result when performing the second mode among the above lane maintenance modes.

12. In paragraph 1, The above processor, In manual driving mode, based on the internal image from the internal camera, the driver's emotional information is detected, and the distance from the first lane in the front image is detected, A signal processing device that performs learning based on the driver's emotional information and the distance from the first lane, and controls the learning result to be stored in memory.

13. In paragraph 12, The above processor, A signal processing device that sets the second interval based on the learning result when performing the second mode among the above lane maintenance modes.

14. In paragraph 1, The above processor, In manual driving mode, detect the gap with the first lane in the front image, Control to store the speed of the above vehicle and the distance information from the first lane in the memory, A signal processing device that sets the second interval based on information about the speed of the vehicle and the interval from the first lane when performing the second mode among the lane maintenance modes.

15. In paragraph 1, The above processor, Run multiple virtual machines on a hypervisor, Some of the above virtual machines execute a lane detection application based on the forward image, A signal processing device that executes multiple microservices for the above lane detection application.

16. In paragraph 15, Some other virtual machines among the above plurality of virtual machines run a notification application for the lane maintenance mode, A signal processing device wherein the safety level of the above notification application is lower than the safety level of the above lane detection application.

17. A processor that receives and processes a front image from a camera installed in the vehicle; The above processor, Perform lane detection based on the front image from the above camera, and perform lane keeping mode based on the lane detection, In the first mode of the lane keeping mode, the steering drive unit is controlled to maintain the center of the adjacent first and second lanes. A signal processing device that controls the steering drive unit to move closer to either the first lane or the second lane, depending on the second mode among the lane maintenance modes.

18. In paragraph 17, The above processor, According to the first mode among the above lane maintenance modes, the steering drive unit is controlled to maintain the first distance from the first lane, A signal processing device that controls the steering drive unit to maintain a second interval different from the first interval with the first lane according to the second mode among the above lane maintenance modes.

19. A vehicle control device having a signal processing device according to any one of claims 1 to 18.

Images