Signal processing device and vehicle control device having the same

Abstract

The signal processing device and vehicle control device having the same according to an embodiment of the present disclosure have a processor that receives and processes a front image from a camera mounted in a vehicle; the processor performs lane line detection based on the front image from the camera and performs a lane keeping mode based on the lane line detection; the processor controls a steering drive portion to maintain a first interval with a first lane line among adjacent first and second lane lines according to a first mode in the lane keeping mode; and the processor controls the steering drive portion to maintain a second interval different from the first interval with the first lane line according to a second mode in the lane keeping mode. Thus, the lane line interval can be adaptively adjusted to reflect the driving conditions during operation of the lane keeping mode.

Description

Technical Field

[0001] This disclosure relates to a signal processing apparatus and a vehicle control apparatus having the same, and more specifically, to a signal processing apparatus and a vehicle control apparatus having the same capable of reflecting driving conditions and adaptively adjusting lane spacing during operation in lane-keeping mode. Background Technology

[0002] A vehicle is a device that moves in the direction desired by the user. A car is a typical example.

[0003] On the other hand, for the convenience of users of the vehicle, a vehicle signal processing device is installed inside the vehicle.

[0004] Due to vehicle driver assistance systems (ADAS) or autonomous driving, the vehicle's internal signal processing unit receives and processes sensor data from various internal sensor devices.

[0005] The existing U.S. patent US11840220 relates to a vehicle steering device, disclosing neutral steering wheel control for keeping a vehicle traveling in a straight line within a lane.

[0006] However, existing literature has the following drawback: it controls the vehicle steering mechanism to keep the vehicle in the center of the lane without taking into account driving conditions. Summary of the Invention

[0007] The problem to be solved

[0008] The problem to be solved by this disclosure is to provide a signal processing device and a vehicle control device having the same, which can reflect driving conditions and adaptively adjust lane spacing during operation in lane keeping mode.

[0009] Another issue to be addressed by this disclosure is to provide a signal processing device and a vehicle control device having the same, capable of adaptively adjusting lane spacing based on an emergency vehicle approaching from behind the vehicle.

[0010] Another issue to be addressed by this disclosure is to provide a signal processing device capable of adaptively adjusting lane spacing based on the side conditions of a vehicle, and a vehicle control device having the same.

[0011] Another issue to be addressed by this disclosure is to provide a signal processing device and a vehicle control device having the same, capable of adaptively adjusting lane spacing based on driving proficiency or the number of occupants in the vehicle.

[0012] Another issue to be addressed by this disclosure is to provide a signal processing device and a vehicle control device having the same, capable of adaptively adjusting lane spacing based on driver's line of sight or emotional information.

[0013] Technical solutions to the problem

[0014] A signal processing apparatus and a vehicle control device having the present disclosure, according to an embodiment of the present disclosure, for solving the above-mentioned problems, include a processor that receives and processes a forward image from a camera installed in the vehicle; the processor performs lane line detection based on the forward image from the camera and performs a lane keeping mode based on the lane line detection; the processor controls the steering drive unit to maintain a first interval with the first lane line of a first lane line and a second lane line adjacent to the lane keeping mode, according to a first mode of the lane keeping mode; the processor controls the steering drive unit to maintain a second interval with the first lane line that is different from the first interval, according to a second mode of the lane keeping mode.

[0015] On the other hand, the processor can be controlled to detect emergency vehicles in the rear image based on the rear image from the camera, and execute the second mode in the lane keeping mode based on the emergency vehicles.

[0016] On the other hand, the processor can be controlled to execute the second mode of lane keeping mode based on emergency vehicles behind the vehicle, and the closer the distance to the emergency vehicles becomes, the smaller the interval with the first lane line becomes.

[0017] On the other hand, if a second vehicle, at a road boundary or guardrail or above a reference size, is located to the side of the vehicle based on a front or side image from the camera, the processor can control the execution of the second mode in the lane-keeping mode.

[0018] On the other hand, the processor can be controlled to detect a second vehicle on the side of the vehicle based on a front or side image from the camera, and to change the second interval in the second mode of the lane keeping mode based on the size of the second vehicle.

[0019] On the other hand, the processor can control the second interval to be changed based on the vehicle's speed in the second mode of lane keeping mode.

[0020] On the other hand, the processor can be controlled to detect pedestrians on the side of the vehicle based on front or side images from the camera, and to change the second interval based on the number of pedestrians detected or their positions.

[0021] On the other hand, the processor can change the second interval in the second mode of lane keeping mode based on the driver's driving skill or the number of passengers in the vehicle or whether the front passenger is seated.

[0022] On the other hand, the processor can detect the driver's gaze based on internal images from the internal camera and change the second interval in the second mode of lane keeping mode based on the direction of the driver's gaze.

[0023] On the other hand, the processor can be controlled to detect the driver's gaze based on the internal image from the internal camera in manual driving mode, detect the distance between the driver's gaze and the first lane line in the image ahead, learn based on the driver's gaze and the distance between the driver and the first lane line, and store the learning results in memory.

[0024] On the other hand, the processor can set a second interval based on the learning results when executing the second mode in lane keeping mode.

[0025] On the other hand, the processor can be controlled to detect the driver's emotional information based on internal images from the internal camera in manual driving mode, detect the distance between the driver and the first lane line in the image ahead, learn based on the driver's emotional information and the distance between the driver and the first lane line, and store the learning results in memory.

[0026] On the other hand, the processor can be controlled to detect the distance to the first lane line in the image ahead in manual driving mode, and store the vehicle speed and the distance information to the first lane line in memory; the processor can set the second distance based on the vehicle speed and the distance information to the first lane line when executing the second mode in lane keeping mode.

[0027] On the other hand, the processor can run multiple virtual machines on the hypervisor, and some of these virtual machines can run lane detection applications based on the image ahead, running multiple microservices for the lane detection applications.

[0028] On the other hand, another portion of the virtual machines among the plurality of virtual machines can run a notification application for lane keeping mode; the security level of the notification application can be lower than that of the lane line detection application.

[0029] Another embodiment of the signal processing apparatus and vehicle control device having the present disclosure includes a processor that receives and processes forward images from a camera installed in the vehicle; the processor performs lane line detection based on the forward images from the camera and performs a lane keeping mode based on the lane line detection; the processor controls the steering drive unit to remain in the center of adjacent first and second lane lines according to a first mode of the lane keeping mode; the processor controls the steering drive unit to move closer to either the first or second lane line according to a second mode of the lane keeping mode.

[0030] On the other hand, the processor can control the steering drive unit to maintain a first interval with the first lane line according to the first mode in the lane keeping mode; the processor can control the steering drive unit to maintain a second interval with the first lane line according to the second mode in the lane keeping mode, which is different from the first interval.

[0031] Invention Effects

[0032] A signal processing apparatus and a vehicle control device having the present disclosure, according to an embodiment, include a processor that receives and processes forward images from a camera installed in the vehicle; the processor performs lane line detection based on the forward images from the camera, and performs a lane keeping mode based on the lane line detection; the processor controls the steering drive unit to maintain a first distance from the first lane line of a first lane line and a second lane line, according to a first mode in the lane keeping mode; the processor controls the steering drive unit to maintain a second distance from the first lane line, different from the first distance, according to a second mode in the lane keeping mode. Thus, the lane line spacing can be adaptively adjusted to reflect driving conditions during lane keeping mode operation.

[0033] On the other hand, the processor can be controlled to detect emergency vehicles in the rear image based on the rear image from the camera, and execute the second mode in the lane-keeping mode based on the emergency vehicles. Thus, the lane spacing can be adaptively adjusted based on emergency vehicles approaching from behind the vehicle.

[0034] On the other hand, the processor can control the second mode of lane keeping mode to be executed based on emergency vehicles approaching from behind the vehicle, and control the spacing between the vehicle and the first lane line to become smaller as the distance to the emergency vehicle becomes closer. Thus, the lane line spacing can be adaptively adjusted based on emergency vehicles approaching from behind the vehicle.

[0035] On the other hand, if, based on the front or side images from the camera, a second vehicle at or above the road boundary, guardrail, or reference size is located beside the vehicle, the processor can control the system to execute the second mode of lane-keeping mode. This allows for adaptive adjustment of lane spacing based on the situation in front of or beside the vehicle.

[0036] On the other hand, the processor can be controlled to detect a second vehicle on the side of the vehicle based on the front or side image from the camera, and to change the second lane spacing in the second mode of the lane keeping mode based on the size of the second vehicle. Thus, the lane spacing can be adaptively adjusted based on the situation in front of or to the side of the vehicle.

[0037] On the other hand, the processor can control the lane keeping mode to change the second lane spacing based on the vehicle's speed in the second mode. This allows for adaptive adjustment of lane spacing based on vehicle speed.

[0038] On the other hand, the processor can be controlled to detect pedestrians on the side of the vehicle based on front or side images from the camera, and to change the second interval based on the number or position of the detected pedestrians. Thus, the lane spacing can be adaptively adjusted based on the situation in front of or to the side of the vehicle.

[0039] On the other hand, the processor can change the second lane spacing in the second mode of lane keeping mode based on the driver's driving skill, the number of occupants in the vehicle, or whether the front passenger is seated. Thus, lane spacing can be adaptively adjusted based on driving skill or the number of occupants in the vehicle.

[0040] On the other hand, the processor can detect the driver's gaze based on internal images from the internal camera and adjust the second lane spacing in the second mode of lane keeping mode based on the direction of the driver's gaze. Thus, the lane spacing can be adaptively adjusted based on the direction of the driver's gaze.

[0041] On the other hand, the processor can be controlled to detect the driver's gaze based on internal images from the internal camera in manual driving mode, detect the distance between the driver's gaze and the first lane line in the image ahead, learn based on the driver's gaze and the distance between the driver's gaze and the first lane line, and store the learning results in memory. Thus, the lane line spacing can be adaptively adjusted based on the driver's gaze.

[0042] On the other hand, when executing the second mode in lane keeping mode, the processor can set a second interval based on the learning results. This allows for adaptive adjustment of lane spacing based on the learning outcomes.

[0043] On the other hand, the processor can be controlled to detect the driver's emotional information based on internal images from the internal camera in manual driving mode, detect the distance between the driver and the first lane line in the image ahead, learn based on the driver's emotional information and the distance to the first lane line, and store the learning results in memory. Thus, the lane line spacing can be adaptively adjusted based on the driver's emotional information.

[0044] On the other hand, the processor can be controlled to detect the distance to the first lane line in the image ahead in manual driving mode, and store the vehicle speed and the distance information to the first lane line in memory; when executing the second mode in lane keeping mode, the processor can set a second distance based on the vehicle speed and the distance information to the first lane line. Thus, the lane line spacing can be adaptively adjusted based on the information in manual driving mode.

[0045] On the other hand, the processor can run multiple virtual machines on the hypervisor, and some of these virtual machines can run lane detection applications based on the image ahead, running multiple microservices for the lane detection applications. This enables the efficient operation of lane detection applications.

[0046] On the other hand, another portion of the multiple virtual machines can run a notification application for lane-keeping mode; the security level of the notification application can be lower than that of the lane detection application. This allows the lane detection application to run stably.

[0047] Another embodiment of the signal processing apparatus and vehicle control device having the present disclosure includes a processor that receives and processes forward images from a camera installed in the vehicle; the processor performs lane line detection based on the forward images from the camera, and performs a lane keeping mode based on the lane line detection; the processor controls the steering drive unit to remain centered on adjacent first and second lane lines according to a first mode of the lane keeping mode; the processor controls the steering drive unit to move closer to either the first or second lane line according to a second mode of the lane keeping mode. Thus, it is possible to reflect driving conditions and adaptively adjust lane line spacing during lane keeping mode operation.

[0048] On the other hand, the processor can control the steering drive to maintain a first interval with the first lane line according to the first mode in the lane keeping mode; the processor can also control the steering drive to maintain a second interval with the first lane line, different from the first interval, according to the second mode in the lane keeping mode. Thus, it is possible to reflect driving conditions and adaptively adjust the lane line interval during lane keeping mode operation. Attached Figure Description

[0049] Figure 1 This is a diagram showing an example of the exterior and interior of a vehicle.

[0050] Figure 2 This is a diagram showing the architecture of a vehicle signal processing system.

[0051] Figure 3a This is a diagram showing an example of the arrangement of a vehicle display device inside a vehicle.

[0052] Figure 3b This is another example of the arrangement of vehicle display devices inside a vehicle.

[0053] Figure 4 yes Figure 1 An example of an internal block diagram of a vehicle.

[0054] Figures 5a to 5d This is a diagram showing various examples of vehicle control devices.

[0055] Figure 6 This is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0056] Figure 7a These figures are referenced when illustrating the signal processing apparatus in relation to this disclosure.

[0057] Figure 7b This is a diagram illustrating an example of the operation of a microservice according to an embodiment of this disclosure.

[0058] Figure 8 This is an example of a signal processing system according to an embodiment of the present disclosure.

[0059] Figure 9 This is a diagram illustrating an example of a system driven in a signal processing apparatus according to an embodiment of the present disclosure.

[0060] Figures 10a to 10c This is a diagram used to illustrate the operation of the vehicle control device related to this disclosure.

[0061] Figure 11a This is a flowchart illustrating the operation method of a signal processing apparatus according to an embodiment of the present disclosure.

[0062] Figure 11b This is a flowchart illustrating an operation method of a signal processing apparatus according to another embodiment of the present disclosure.

[0063] Figures 12a to 18 This is an explanation Figures 11a to 11b The action is referenced in the diagram. Detailed Implementation

[0064] The present disclosure will now be described in detail with reference to the accompanying drawings.

[0065] The suffixes “module” and “section” used in the following description for constituent elements are merely for the convenience of writing the specification and do not have any particularly important meaning or function in themselves. Therefore, “module” and “section” can be used interchangeably.

[0066] Figure 1 This is a diagram showing an example of the exterior and interior of a vehicle.

[0067] Referring to the accompanying drawings, the vehicle 200 is driven by a plurality of wheels 103FR, 103FL, 103RL, ... and a steering wheel 150. The plurality of wheels 103FR, 103FL, 103RL, ... are rotated by a power source, and the steering wheel 150 is used to adjust the direction of travel of the vehicle 200.

[0068] On the other hand, the vehicle 200 may also have a camera 195 for acquiring images of the front of the vehicle.

[0069] On the other hand, multiple displays 180a and 180b for displaying images, information, etc. can be installed inside the vehicle 200.

[0070] exist Figure 1 In the example, the instrument cluster display 180a and the AVN (Audio Video Navigation) display 180b are examples of multiple displays 180a and 180b. In addition, they could also be HUD (Head-Up Display), etc.

[0071] On the other hand, the AVN (Audio Video Navigation) display 180b can also be named the Central Information Display.

[0072] On the other hand, the concept of vehicle 200 described in this specification may encompass vehicles that have an engine as a power source, hybrid vehicles that have both an engine and an electric motor as a power source, electric vehicles that have an electric motor as a power source, etc.

[0073] Figure 2 This is a diagram showing the architecture of a vehicle signal processing system.

[0074] Referring to the attached diagram, the architecture 300a of the vehicle signal processing system can correspond to a zone-based architecture.

[0075] Therefore, sensor devices and processors inside the vehicle can be arranged in the plurality of zones Z1 to Z4 respectively, and a signal processing device 170a including a vehicle communication gateway GWDa can be arranged in the central area of ​​the plurality of zones Z1 to Z4.

[0076] On the other hand, in addition to the vehicle communication gateway GWDa, the signal processing device 170a may also include an autonomous driving control module ACC and a cockpit control module CPG.

[0077] The vehicle communication gateway GWDa within this signal processing device 170a can be an HPC (High Performance Computing) gateway.

[0078] Right now, Figure 2 The signal processing device 170a can function as an integrated HPC to exchange data with an external communication module (not shown) or processors (not shown) in multiple zones Z1 to Z4.

[0079] Figure 3a This is a diagram showing an example of the arrangement of a vehicle display device inside a vehicle.

[0080] Referring to the attached diagram, the vehicle interior may be equipped with an instrument cluster display 180a, an AVN (Audio Video Navigation) display 180b, a rear seat entertainment display 180c, 180d, and a rearview mirror display (not shown).

[0081] Figure 3b This is another example of the arrangement of vehicle display devices inside a vehicle.

[0082] The vehicle display device 100 of this disclosure embodiment may have a plurality of displays 180a to 180b and a signal processing device 170, wherein the signal processing device 170 performs signal processing for displaying images, information, etc. on the plurality of displays 180a to 180b and outputs image signals to at least one display 180a to 180b.

[0083] The first display 180a among the plurality of displays 180a to 180b may be an instrument cluster display 180a for displaying driving status, action information, etc., and the second display 180b may be an AVN (Audio Video Navigation) display 180b for displaying vehicle driving information, navigation maps, various entertainment information or images.

[0084] The signal processing device 170 may have a processor 175 internally configured, and may run a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor 175.

[0085] The second virtual machine (not shown) can act for the first display 180a, and the third virtual machine (not shown) can act for the second display 180b.

[0086] On the other hand, the first virtual machine (not shown) within the processor 175 can be controlled to set up a shared memory 508 based on the hypervisor 505 to transmit the same data to the second virtual machine (not shown) and the third virtual machine (not shown). Thus, the same information or the same image can be displayed synchronously on the first display 180a and the second display 180b within the vehicle.

[0087] On the other hand, for data sharing, a first virtual machine (not shown) within the processor 175 shares at least a portion of the data with a second virtual machine (not shown) and a third virtual machine (not shown). Thus, data processing can be shared among multiple virtual machines used for multiple displays within the vehicle.

[0088] On the other hand, the first virtual machine (not shown) within the processor 175 can receive and process the vehicle's wheel speed sensor data, and transmit the processed wheel speed sensor data to at least one of the second virtual machine (not shown) and the third virtual machine (not shown). Thus, the vehicle's wheel speed sensor data can be shared with at least one virtual machine, etc.

[0089] On the other hand, the vehicle display device 100 of the present disclosure embodiment may also have a rear seat entertainment (RSE) display 180c for displaying driving status information, simple navigation information, various entertainment information or images.

[0090] In addition to the first to third virtual machines (not shown), the signal processing device 170 can control the RSE display 180c by additionally running a fourth virtual machine (not shown) on a hypervisor (not shown) within the processor 175.

[0091] Therefore, a signal processing device 170 can be used to control various displays 180a to 180c.

[0092] On the other hand, some of the multiple displays 180a-180c can operate based on Linux OS (operating system), while others can operate based on Internet OS.

[0093] The signal processing apparatus 170 of this embodiment can be controlled to synchronously display the same information or the same image even when the displays 180a to 180c operate under various operating systems (OS).

[0094] on the other hand, Figure 3bExample: A vehicle speed indicator 212a and a vehicle interior temperature indicator 213a are displayed on a first display 180a; a main screen 222 including a plurality of applications and the vehicle speed indicator 212b and the vehicle interior temperature indicator 213b is displayed on a second display 180b; and a second main screen 222b including a plurality of applications and the vehicle interior temperature indicator 213c is displayed on a third display 180c.

[0095] Figure 4 yes Figure 1 An example of an internal block diagram of a vehicle.

[0096] Referring to the accompanying drawings, the vehicle 200 of this embodiment may include a light drive unit 751, a steering drive unit 752, a braking drive unit 753, a power source drive unit 754, a suspension drive unit 756, an air conditioning drive unit 757, a window drive unit 758, a seat drive unit 761, and a signal processing device 170.

[0097] On the other hand, vehicle 200 may also have ECU 770, multiple sensor devices SN, and multiple communication modules EMa to EMd.

[0098] On the other hand, the vehicle 200 of this embodiment may also have a vehicle display device 100.

[0099] The vehicle display device 100 of this embodiment may include an input unit 110, a communication unit 120 for communicating with external devices, a plurality of communication modules EMa to EMD for internal communication, a memory 140, a signal processing device 170, a plurality of displays 180a to 180c, an audio output unit 185, and a power supply unit 190.

[0100] Multiple communication modules EMa to EMD can be arranged separately, for example, in... Figure 2 The multiple zones Z1 to Z4.

[0101] On the other hand, a communication switch 736b for data communication with each communication module EM1 to EM4 may be provided inside the signal processing device 170.

[0102] Each communication module EM1 to EM4 can communicate with multiple sensor devices SN or ECU (Electronic Control Unit) 770 or area signal processing device 170Z.

[0103] On the other hand, the plurality of sensor devices SN may include a camera 195, a lidar 196, a radar 197, or a position sensor 198.

[0104] The input unit 110 may be equipped with physical buttons, tablets, etc., for key input, touch input, etc.

[0105] On the other hand, the input unit 110 may be equipped with a microphone (not shown) for user voice input.

[0106] The communication unit 120 can exchange data wirelessly with the mobile terminal 800 or the server 900.

[0107] In particular, the communication unit 120 can exchange data wirelessly with the vehicle driver's mobile terminal. Various wireless data communication methods can be used, such as Bluetooth, WiFi (Wireless High Fidelity), WiFi Direct, and APiX.

[0108] The communication unit 120 can receive weather information, road traffic information, such as TPEG (Transport Protocol Expert Group) information, from the mobile terminal 800 or the server 900. For this purpose, the communication unit 120 may include a mobile communication module (not shown).

[0109] Multiple communication modules EM1 to EM4 can receive sensor data from ECU770, sensor device SN, or area signal processing device 170Z, and transmit the received sensor data to signal processing device 170.

[0110] Here, sensor data may include at least one of the following: vehicle orientation data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, headlight data, vehicle interior temperature data, and vehicle interior humidity data.

[0111] This sensor data can be acquired from heading sensors, yaw sensors, gyroscope sensors, position modules, vehicle forward / reverse sensors, wheel sensors, vehicle speed sensors, vehicle tilt sensors, battery sensors, fuel sensors, tire sensors, steering sensors based on steering wheel rotation, vehicle interior temperature sensors, and vehicle interior humidity sensors.

[0112] On the other hand, the positioning module may include a GPS module or a position sensor 198 for receiving GPS (Global Positioning System) information.

[0113] On the other hand, at least one of the plurality of communication modules EM1 to EM4 can transmit location information data detected by the GPS module or the position sensor 198 to the signal processing device 170.

[0114] On the other hand, at least one of the plurality of communication modules EM1 to EM4 can receive frontal image data, side image data, rear image data, and distance information of obstacles around the vehicle from the camera 195, lidar 196, or radar 197, and transmit the received information to the signal processing device 170.

[0115] The memory 140 can store various data for the overall operation of the vehicle display device 100, such as programs for processing or controlling the signal processing device 170.

[0116] For example, memory 140 may store data about a hypervisor, a first virtual machine, to a third virtual machine, used to run on processor 175.

[0117] The audio output unit 185 converts the electrical signal from the signal processing device 170 into an audio signal and outputs it. For this purpose, a speaker or the like can be provided.

[0118] The power supply unit 190 can supply the power required for the operation of each component according to the control of the signal processing device 170. In particular, the power supply unit 190 can receive power from the battery or the like inside the vehicle.

[0119] The signal processing device 170 controls the overall operation of the vehicle display device 100 or the various units within the vehicle 200.

[0120] For example, signal processing device 170 may include processor 175 that performs signal processing for vehicle displays 180a, 180b.

[0121] Processor 175 can run a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within processor 175.

[0122] The first virtual machine (not shown) among the first to third virtual machines (not shown) can be named the Server Virtual Machine, and the second to third virtual machines (not shown) can be named the Guest Virtual Machine.

[0123] For example, a first virtual machine (not shown) within processor 175 can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or manipulate the data before outputting it.

[0124] As described above, data sharing in a 1:N manner can be achieved by performing most of the data processing in the first virtual machine (not shown).

[0125] As another example, the first virtual machine (not shown) can directly receive and process CAN (Controller Area Network) data, Ethernet data, audio data, radio data, USB (Universal Serial Bus) data, and wireless communication data for the second to third virtual machines (not shown).

[0126] In addition, the first virtual machine (not shown) can transfer the processed data to the second virtual machine to the third virtual machine (not shown).

[0127] Therefore, by having only the first virtual machine (not shown) among the first to third virtual machines (not shown) receive sensor data, communication data, or external input data from multiple sensor devices and perform signal processing, the signal processing burden in other virtual machines can be reduced, 1:N data communication can be achieved, and thus synchronization during data sharing can be realized.

[0128] On the other hand, the first virtual machine (not shown) can be controlled to write data to the shared memory 508, thereby sharing the same data with the second virtual machine (not shown) and the third virtual machine (not shown).

[0129] For example, the first virtual machine (not shown) can be controlled to write vehicle sensor data, the location information data, the camera image data, or the touch input data into the shared memory 508, thereby sharing the same data with the second virtual machine (not shown) and the third virtual machine (not shown). This enables 1:N data sharing.

[0130] Ultimately, by performing most of the data processing in the first virtual machine (not shown), 1:N data sharing can be achieved.

[0131] On the other hand, the first virtual machine (not shown) within the processor 175 can be controlled to set up a shared memory 508 based on the hypervisor 505 to transfer the same data to the second virtual machine (not shown) and the third virtual machine (not shown).

[0132] On the other hand, the signal processing device 170 can process various signals such as audio signals, video signals, and data signals. Therefore, the signal processing device 170 can be implemented as a system on chip (SOC).

[0133] on the other hand, Figure 4 The signal processing device 170 can be with Figure 5a The signal processing devices 170, 170a1, and 170a2 of the vehicle control device shown in the following figures are the same.

[0134] Figures 5a to 5d This is a diagram showing various examples of vehicle control devices.

[0135] Figure 5a An example of a vehicle control device according to an embodiment of the present disclosure is shown.

[0136] Referring to the accompanying drawings, the vehicle control device 800a of this disclosure embodiment includes signal processing devices 170a1 and 170a2.

[0137] On the other hand, the vehicle control device 800a of the present disclosure embodiment may also include a plurality of regional signal processing devices 170Z1 to 170Z4.

[0138] On the other hand, the accompanying drawings illustrate two signal processing devices 170a1 and 170a2, but this is an example for backup purposes, and it could also be a single signal processing device.

[0139] On the other hand, signal processing devices 170a1 and 170a2 can also be named HPC (High Performance Computing) signal processing devices.

[0140] Multiple signal processing devices 170Z1 to 170Z4 can be arranged in each region Z1 to Z4 and transmit sensor data to signal processing devices 170a1 and 170a2.

[0141] Signal processing devices 170a1 and 170a2 receive data from a plurality of regional signal processing devices 170Z1 to 170Z4 or communication device 120 via wired connection.

[0142] Although the accompanying drawings illustrate the exchange of data between signal processing devices 170a1, 170a2 and multiple regional signal processing devices 170Z1 to 170Z4 based on wired communication, and the exchange of data between signal processing devices 170a1, 170a2 and server 400 based on wireless communication, it is also possible for communication device 120 and server 400 to exchange data based on wireless communication, while signal processing devices 170a1, 170a2 and communication device 120 can exchange data based on wired communication.

[0143] On the other hand, the data received by the signal processing devices 170a1 and 170a2 may include camera data or sensor data.

[0144] For example, sensor data inside the vehicle may include at least one of the following: wheel speed data, vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, headlight data, vehicle interior temperature data, vehicle interior humidity data, vehicle exterior radar data, and vehicle exterior lidar data.

[0145] On the other hand, camera data can include data from both external and internal vehicle cameras.

[0146] On the other hand, signal processing devices 170a1 and 170a2 can run multiple virtual machines 820, 830, and 840 according to the safety level.

[0147] The accompanying drawings illustrate a scenario where the processor 175 within the signal processing device 170a runs a management program 505, and on the management program 505, the first virtual machine to the third virtual machine 820 to 840 are run according to the Automotive Safety Integrity Level (ASIL).

[0148] The first virtual machine 820 may be a virtual machine corresponding to QM (Quality Management), which is the lowest safety level in the Automotive Safety Integrity Level (ASIL) and is not a mandatory level.

[0149] The first virtual machine 820 can run operating system 822, container runtime 824 on operating system 822, and containers 827 and 829 on container runtime 824.

[0150] The second virtual machine 820 can be a virtual machine corresponding to ASIL A or ASIL B, which are automotive safety integrity levels (ASIL) with a sum of 7 or 8 in severity, exposure, and controllability.

[0151] The second virtual machine 820 can run operating system 832, container runtime 834 on operating system 832, and containers 837 and 839 on container runtime 834.

[0152] The third virtual machine 840 can be a virtual machine corresponding to ASIL C or ASIL D, which are the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) of 9 or 10.

[0153] On the other hand, ASIL D can correspond to the level that requires the highest level of security.

[0154] The third virtual machine 840 can run the secure operating system 842 and the application 845 on the operating system 842.

[0155] On the other hand, the third virtual machine 840 can also run a secure operating system 842, a container runtime 844 on the secure operating system 842, and a container 847 on the container runtime 844.

[0156] On the other hand, unlike the attached diagram, the third virtual machine 840 can also run using an additional core instead of processor 175. This will be discussed later. Figure 5b Please provide an explanation.

[0157] Figure 5b Another example of a vehicle control device according to an embodiment of this disclosure is shown.

[0158] Referring to the accompanying drawings, the vehicle control device 800b of this disclosure embodiment includes signal processing devices 170a1 and 170a2.

[0159] On the other hand, the vehicle control device 800b of the present disclosure embodiment may also include a plurality of regional signal processing devices 170Z1 to 170Z4.

[0160] Figure 5b Vehicle control unit 800b and Figure 5a The vehicle control unit 800a is similar, but the signal processing unit 170a1 is... Figure 5a The signal processing device 170a1 has some differences.

[0161] Focusing on this difference, the signal processing device 170a1 may include a processor 175 and a second processor 177.

[0162] The processor 175 within the signal processing device 170a1 runs a management program 505 and runs a first virtual machine 820 to a second virtual machine 830 on the management program 505 according to the Automotive Safety Integrity Level (ASIL).

[0163] The first virtual machine 820 can run operating system 822, container runtime 824 on operating system 822, and containers 827 and 829 on container runtime 824.

[0164] The second virtual machine 820 can run operating system 832, container runtime 834 on operating system 832, and containers 837 and 839 on container runtime 834.

[0165] On the other hand, the second processor 177 within the signal processing device 170a1 can run the third virtual machine 840.

[0166] The third virtual machine 840 can run the secure operating system 842, the AUTOSAR (Automotive Open System Architecture) 845 on the operating system 842, and applications 845 on the AUTOSAR 845. That is, it is compatible with... Figure 5a In contrast, it can also run AUTOSAR 846 on the 842 operating system.

[0167] On the other hand, the third virtual machine 840 can also be used with Figure 5a Similarly, secure operating system 842, container runtime 844 on secure operating system 842, and container 847 on container runtime 844 are run.

[0168] On the other hand, preferably, unlike the first virtual machine 820 to the second virtual machine 830, the third virtual machine 840 with a high security level is required to run on other cores or as a second processor 177 of other processors.

[0169] On the other hand, Figure 5a and Figure 5b In the signal processing devices 170a1 and 170a2, when the first signal processing device 170a malfunctions, the second signal processing device 170a2, which serves as a backup, can operate.

[0170] Alternatively, signal processing devices 170a1 and 170a2 may operate simultaneously, with the first signal processing device 170a operating as the main device and the second signal processing device 170a2 operating as the auxiliary device. For this, refer to... Figure 5c and Figure 5d Please provide an explanation.

[0171] Figure 5c Another example of a vehicle control device according to an embodiment of the present disclosure is shown.

[0172] Referring to the accompanying drawings, the vehicle control device 800c of this disclosure embodiment includes signal processing devices 170a1 and 170a2.

[0173] On the other hand, the vehicle control device 800c of the present disclosure embodiment may also include a plurality of regional signal processing devices 170Z1 to 170Z4.

[0174] On the other hand, the accompanying drawings illustrate two signal processing devices 170a1 and 170a2, but this is an example for backup purposes, etc., and it could also be a single signal processing device.

[0175] On the other hand, signal processing devices 170a1 and 170a2 can also be named HPC (High Performance Computing) signal processing devices.

[0176] Multiple signal processing devices 170Z1 to 170Z4 can be arranged in each region Z1 to Z4 and transmit sensor data to signal processing devices 170a1 and 170a2.

[0177] Signal processing devices 170a1 and 170a2 receive data from a plurality of regional signal processing devices 170Z1 to 170Z4 or communication device 120 via wired connection.

[0178] Although the accompanying drawings illustrate the exchange of data between signal processing devices 170a1, 170a2 and multiple regional signal processing devices 170Z1 to 170Z4 based on wired communication, and the exchange of data between signal processing devices 170a1, 170a2 and server 400 based on wireless communication, it is also possible for communication device 120 and server 400 to exchange data based on wireless communication, while signal processing devices 170a1, 170a2 and communication device 120 can exchange data based on wired communication.

[0179] On the other hand, the data received by the signal processing devices 170a1 and 170a2 may include camera data or sensor data.

[0180] On the other hand, the processor 175 in the first signal processing device 170a1 of the signal processing devices 170a1 and 170a2 can run the management program 505, and a safe virtual machine 860 and a non-safe virtual machine 870 can be run on the management program 505 respectively.

[0181] On the other hand, the processor 175b in the second signal processing device 170a2 of the signal processing devices 170a1 and 170a2 can run the hypervisor 505b, and the hypervisor 505b can run only the safety virtual machine 880.

[0182] In this way, the processing for safety is divided into a first signal processing device 170a1 and a second signal processing device 170a2, thereby improving stability and processing speed.

[0183] On the other hand, high-speed network communication can be performed between the first signal processing device 170a1 and the second signal processing device 170a2.

[0184] Figure 5d Another example of a vehicle control device according to an embodiment of the present disclosure is shown.

[0185] Referring to the accompanying drawings, the vehicle control device 800d of this disclosure embodiment includes signal processing devices 170a1 and 170a2.

[0186] On the other hand, the vehicle control device 800d of the present disclosure embodiment may also include a plurality of regional signal processing devices 170Z1 to 170Z4.

[0187] Figure 5d Vehicle control unit 800d and Figure 5c The vehicle control unit 800c is similar, but the second signal processing unit 170a2 is... Figure 5c There are some differences in the second signal processing device 170a2.

[0188] Figure 5d The processor 175b within the second signal processing device 170a2 can run a hypervisor 505b, and run a safe virtual machine 880 and a non-safe virtual machine 890 on the hypervisor 505b respectively.

[0189] That is, the difference lies in, with Figure 5c In contrast, the processor 175b within the second signal processing device 170a2 also runs a non-safety virtual machine 890.

[0190] According to this method, the processing for safety and non-safety is divided into a first signal processing device 170a1 and a second signal processing device 170a2, thereby improving stability and processing speed.

[0191] Figure 6 This is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0192] Referring to the accompanying drawings, the vehicle control device 900 of this disclosure embodiment includes a signal processing device 170.

[0193] The vehicle control device 900 of this disclosure embodiment may also have at least one display.

[0194] On the other hand, the vehicle control device 900 of this disclosure embodiment may also have Figure 4 Steering drive unit 752, braking drive unit 753, power source drive unit 754, ECU 770 or multiple sensor devices SN, etc.

[0195] On the other hand, the vehicle control device 900 of this disclosure embodiment may also have Figure 4 The light drive unit 751, suspension drive unit 756, air conditioning drive unit 757, window drive unit 758, seat drive unit 761, or multiple communication modules EMa to EMD, etc.

[0196] In the accompanying drawings, instrument cluster display 180a and AVN display 180b are shown as at least one display.

[0197] On the other hand, the vehicle control device 900 may also be equipped with a plurality of area signal processing devices 170Z1 to 170Z4.

[0198] At this time, the signal processing device 170, as a high-performance central signal processing and control device with multiple CPUs 175, GPUs 178, NPUs 179, etc., can be named an HPC (High Performance Computing) signal processing device or a central signal processing device.

[0199] The multiple area signal processing devices 170Z1 to 170Z4 and the signal processing device 170 are connected by wired cables CB1 to CB4.

[0200] On the other hand, the multiple regional signal processing devices 170Z1 to 170Z4 can be connected to each other using wired cables CBa to CBd.

[0201] At this time, the wired cables CBa to CBd may include CAN communication cables, Ethernet communication cables, or PCI Express (peripheral component interconnect express) cables.

[0202] On the other hand, the signal processing apparatus 170 of the present disclosure embodiment may be provided with at least one processor 175, 178, 177 and a large-capacity storage device 925.

[0203] For example, the signal processing apparatus 170 of this disclosure embodiment may include a central processing unit 175, 177, a graphics processor 178, and a neural processor 179.

[0204] On the other hand, sensor data can be transmitted to signal processing device 170 from at least one of the plurality of region signal processing devices 170Z1 to 170Z4. In particular, the sensor data can be stored in storage device 925 within signal processing device 170.

[0205] The sensor data at this time may include at least one of the following: camera data, lidar data, radar data, vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, headlight data, vehicle interior temperature data, and vehicle interior humidity data.

[0206] The accompanying drawings illustrate a scenario where camera data from camera 195a and lidar data from lidar sensor 196 are input to a first area signal processing unit 170Z1, and the camera data and lidar data are transmitted to a signal processing unit 170 via a second area signal processing unit 170Z2 and a third area signal processing unit 170Z3, etc.

[0207] On the other hand, since the speed of reading data from or writing data to the storage device 925 is faster than the network speed when transmitting sensor data from at least one of the plurality of regional signal processing devices 170Z1 to 170Z4 to the signal processing device 170, it is preferable to perform multi-path routing to prevent network bottlenecks.

[0208] Therefore, the signal processing apparatus 170 of this embodiment can perform multi-path routing based on Software-Defined Network (SDN). This ensures a stable network environment for data reading or writing to the storage device 925. Furthermore, since multi-path transmission to the storage device 925 is possible, data transmission can be achieved by dynamically changing the network configuration.

[0209] For high-bandwidth, low-latency communication, the data communication between the plurality of regional signal processing devices 170Z1 to 170Z4 within the vehicle control device 900 of this disclosure and the signal processing device 170 is preferably Peripheral Component Interconnect Express (PCIEX) communication.

[0210] On the other hand, the signal processing apparatus 170 of this embodiment can receive internal images from the internal camera 195i and perform signal processing on the internal images.

[0211] On the other hand, the signal processing apparatus 170 of this embodiment can receive a front image from the front camera 195a and perform signal processing on the front image.

[0212] Figure 7a These figures are referenced when illustrating the signal processing apparatus in relation to this disclosure.

[0213] Referring to the accompanying drawings, the signal processing apparatus 170x related to this disclosure can run application 785 based on vehicle sensor data or camera data, and output the result data of application 785 through multiple paths.

[0214] In this way, the result data of application 785 is only output after application 785 has finished running. Therefore, the efficiency is low and the time consumption is likely to increase before application 785 has finished running.

[0215] Therefore, this disclosure proposes a scheme for sharing intermediate result data of an application while the application is running.

[0216] Therefore, the signal processing apparatus 170 of this embodiment divides the application into a plurality of microservices and runs other microservices based on the results of the microservices, thereby efficiently distributing the workload.

[0217] Figure 7b This is a diagram illustrating an example of the operation of a microservice according to an embodiment of this disclosure.

[0218] Referring to the accompanying drawings, the signal processing apparatus 170 of this disclosure embodiment can run application 795 based on vehicle sensor data or camera data, etc.

[0219] At this time, the signal processing apparatus 170 of this embodiment of the present disclosure can divide and run a plurality of microservices for application 795.

[0220] On the other hand, the signal processing apparatus 170 of this disclosure embodiment can be divided and run applications or microservices according to various security levels.

[0221] At this time, if the security level of the transmitting application or microservice is higher than or equal to the security level of the receiving application or microservice, the signal processing apparatus 170 of this embodiment of the disclosure sends the result data of the transmitting application or microservice.

[0222] On the other hand, if the security level of the transmitting application or microservice is lower than the security level of the receiving application or microservice, the signal processing apparatus 170 of this embodiment of the disclosure prevents the transmission of result data of the transmitting application or microservice.

[0223] Referring to the accompanying drawings, a first microservice 910 corresponding to ASIL D (the second security level) can be run based on input data. The result data of the first microservice 910 corresponding to ASIL D can be transmitted to the second microservice 920a (the third security level QM), the third microservice 920b (the first security level ASIL B), the fourth microservice 920c (the first security level ASIL B), and the fifth microservice 920d (the third security level ASIL D), respectively.

[0224] Since the security level of the first microservice 910 is higher than that of the second microservice 920a, the third microservice 920b, and the fourth microservice 920c, the result data of the first microservice 910 can be transmitted.

[0225] On the other hand, since the security level of the first microservice 910 is the same as that of the fifth microservice 920d, the result data of the first microservice 910 can be transmitted.

[0226] Next, the sixth microservice 930a, which corresponds to QM as the third security level, can run based on the result data of the second microservice 920a, and its result data can be output through the first path.

[0227] On the other hand, the seventh microservice 930b, which corresponds to ASIL B as the first security level, can run based on the result data of the third microservice 920b and the result data of the fourth microservice 920c, and its result data can be output through the second path.

[0228] On the other hand, the eighth microservice 930c, which corresponds to ASIL D as the second security level, can run based on the result data of the fifth microservice 920d, and its result data can be output through the third path.

[0229] As shown in the figure, in addition to using multiple paths to output the result data of application 795, it also... Figure 7a Unlike other systems, each microservice runs and processes its own microservice through its own path within the signal processing device 170, thereby efficiently distributing the workload and enabling efficient data processing.

[0230] Figure 8 This is an example of a signal processing system according to an embodiment of the present disclosure.

[0231] Referring to the accompanying drawings, a signal processing system 1000 according to an embodiment of the present disclosure may have a central signal processing device 170 and a regional signal processing device 170z.

[0232] On the other hand, the signal processing device 170 in the system 1000 of one embodiment of the present disclosure has a plurality of processor cores CR1 to CRn and MR.

[0233] On the other hand, a subset of the multiple processor cores CR1 to CRn, and a portion of CR1 to CRn in the MR, can interact with... Figure 6 The processor core within the central processing unit (CPU) corresponds to this.

[0234] For example, a plurality of processor cores CR1 to CRn, and a subset of CR1 to CRn in MR, can be coupled with... Figure 6 It corresponds to the application processor core within the central processing unit (CPU).

[0235] On the other hand, a plurality of processor cores CR1 to CRn, and a portion of CR1 to CRn in MR, can run on hypervisor 505, which can run a plurality of virtual machines 820 to 850.

[0236] On the other hand, another part of the multiple processor cores CR1 to CRn and MR can correspond to the M core or MCU (micom unit).

[0237] On the other hand, another part of the plurality of processor cores CR1 to CRn and MR can run an operating system 805a corresponding to the second security level, such as ASIL D, without the operation of the hypervisor 505, and run a fourth virtual machine 840 on the operating system 805a.

[0238] On the other hand, the fourth virtual machine 840 can run applications corresponding to the second security level, such as ASIL D, or microservices 843 corresponding to applications corresponding to the second security level. Therefore, applications or microservices 843 corresponding to the second security level can be executed stably.

[0239] On the other hand, the first processor core CR1 among the plurality of processor cores CR1 to CRn and MR can run a hypervisor 505, which runs an operating system 805b corresponding to the second security level, such as ASIL D, and runs a first virtual machine 850 on the operating system 805b.

[0240] On the other hand, the first virtual machine 850 can run applications corresponding to the first security level, such as ASIL B, or microservices 853a and 853b corresponding to the applications corresponding to the first security level. Therefore, applications or microservices 853a and 853b corresponding to the first security level can run stably.

[0241] On the other hand, unlike the attached figure, the first processor core CR1 among the plurality of processor cores CR1 to CRn and MR can also run an operating system such as ASIL B corresponding to the first security level on the hypervisor 505.

[0242] On the other hand, the plurality of processor cores CR1 to CRn, the second processor core CR2 and the third processor core CR3 in MR can run a hypervisor 505, on which an operating system 805c corresponding to the first security level, such as ASIL B, can run, and on the operating system 805c, a second virtual machine 850 can run.

[0243] On the other hand, the second virtual machine 850 can run a third application corresponding to the first security level, such as ASIL B, or microservices 833a to 833d corresponding to the third application corresponding to the first security level, on the operating system 805c corresponding to the first security level. Therefore, the application or microservices 833a to 833d corresponding to the first security level can run stably.

[0244] On the other hand, the remaining processor cores CR4 to CRn in the plurality of processor cores CR1 to CRn can run a hypervisor 505, on which an operating system 805d corresponding to the third security level, such as QM, can run, and on the operating system 805d, a third virtual machine 820 can run.

[0245] On the other hand, the third virtual machine 820 can run a fourth application corresponding to the third security level, such as QM, or microservices 823a to 823d corresponding to the fourth application corresponding to the third security level, on an operating system 805d that corresponds to a third security level lower than the first security level. Therefore, applications or microservices 823a to 823d corresponding to the third security level can run stably.

[0246] On the other hand, the regional signal processing device 170z may have a plurality of application processor cores CRR1 to CRRm and an M core MRb for running applications that correspond to ASIL D, which is the highest security level.

[0247] On the other hand, a portion of the multiple processor cores CRR1 to CRRm and MRb in the regional signal processing device 170z, RR1 to CRRm, can run an operating system 806b corresponding to the first security level, such as ASIL B, and run a virtual machine 830b corresponding to the first security level on the operating system 806a.

[0248] On the other hand, the virtual machine 830b corresponding to the first security level can run applications corresponding to the first security level, such as ASIL B, or microservices 830ba to 830bd corresponding to applications corresponding to the first security level. Therefore, applications or microservices 830ba to 830bd corresponding to the first security level can run stably.

[0249] On the other hand, another part of the multiple processor cores CRR1 to CRRm and MRb in the regional signal processing device 170z, MRb, can run an operating system 806a corresponding to the second security level, such as ASIL D, and run a virtual machine 840b corresponding to the second security level, such as ASIL D, on the operating system 806a.

[0250] On the other hand, the virtual machine 840b corresponding to the second security level can run applications corresponding to the second security level, such as ASIL D, or microservices 843b corresponding to applications corresponding to the second security level. Therefore, applications or microservices 843b corresponding to the second security level can run stably.

[0251] Figure 9 This is a diagram illustrating an example of a system driven in a signal processing apparatus according to an embodiment of the present disclosure.

[0252] Referring to the accompanying drawings, a signal processing device 170 within a signal processing system 1000 according to an embodiment of the present disclosure includes a central processing unit 175 and at least one neural processor 179a to 179c.

[0253] On the other hand, the signal processing apparatus 170 of the present disclosure embodiment may also have a graphics processor 178.

[0254] On the other hand, the central processing unit 175 of this embodiment runs the management program 505.

[0255] On the other hand, the system 1100 driven in the signal processing apparatus 170 of this disclosure embodiment runs a plurality of virtual machines 810, 830, 850 on the hypervisor 505.

[0256] Specifically, the central processing unit 175 in the signal processing apparatus 170 of this disclosure runs a management program 505 and runs a plurality of virtual machines 810, 830, and 850 on the management program 505.

[0257] On the other hand, the central processing unit 175 within the signal processing apparatus 170 of this disclosure embodiment runs an application for driving the vehicle.

[0258] On the other hand, if the central processing unit 175 determines that the application's operation has failed, it controls the operation to run a second application corresponding to the application on another central processing unit or other signal processing device, and changes the baseline rollback assurance time for the application's operation failure based on the application's security level.

[0259] This enables the stable execution of applications used for vehicle operation. In particular, it enables the stable execution of applications used for vehicle operation based on safety levels.

[0260] On the other hand, the signal processing apparatus 170 of one embodiment of the present disclosure may also include a shared memory 508.

[0261] The accompanying drawings illustrate a scenario where a hypervisor 505 runs on a central processing unit 175, and a shared memory 508 runs within the hypervisor 505.

[0262] On the other hand, the signal processing apparatus 170 of the present disclosure embodiment can receive data from the camera device 195, the sensor device 700, the communication device 120 or the lidar device 196, and perform signal processing using at least one of the central processing unit 175, the graphics processor 178, and the plurality of neural processors 179a to 179c.

[0263] On the other hand, the sensor device 700 can continuously output sensor data to the signal processing device 170 during vehicle operation.

[0264] The sensor data at this time comes from various sensor devices 700 of the vehicle, and may include at least one of the following: vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle light data, vehicle interior temperature data, and vehicle interior humidity data.

[0265] On the other hand, the camera device 195 can continuously output camera data to the signal processing device 170 during vehicle movement.

[0266] On the other hand, the lidar 196 can continuously output lidar data to the signal processing device 170 during vehicle operation.

[0267] On the other hand, the neural processor 179 can detect objects based on camera data and act on the objects at a variable frame rate or output result data including the objects.

[0268] On the other hand, the neural processor 179 can receive camera data at a fixed frame rate and detect objects based on the camera data, and then act on the objects at a variable frame rate or output result data including the objects.

[0269] On the other hand, the first virtual machine 810, which is a server virtual machine among the plurality of virtual machines 810, 830, and 850, controls the actions of the neural processor 179.

[0270] On the other hand, among the plurality of virtual machines 810, 830, and 850, the second virtual machine 850 and the third virtual machine 830, which serve as guest virtual machines, can each run applications.

[0271] The accompanying diagram illustrates a scenario where a second virtual machine 850 runs a vehicle driver assistance system (ADAS) application Nad or an autonomous driving application or a driver monitoring system (DMS) application Ndm, and a third virtual machine 830 runs an augmented reality (AR) application Nar.

[0272] If, in a state where multiple applications running from at least one of the multiple virtual machines 810, 830, and 850 sequentially receive requests for a first operation, a second operation, and a third operation, and the first and third operations can be processed in parallel, then the first virtual machine 810 controls the first neural processor 179a to process the first and third operations in parallel, and processes the second operation after the first and third operations are completed. This enables the neural processor to operate efficiently. Furthermore, it reduces power consumption.

[0273] On the other hand, if the second and fourth operations can share the same computation layer when a request for a fourth operation is received after the third operation request, then the first virtual machine 810 can control the first neural processor 179a to continuously process the second and fourth operations after completing the first and third operations. This enables the neural processor to operate efficiently.

[0274] On the other hand, if multiple applications request multiple operations, the first virtual machine 810 can be controlled to change the arrangement of data for the multiple operations in the internal memory 1805 within the first neural processor 179a. This enables the neural processor to operate efficiently.

[0275] On the other hand, the first virtual machine 810 can run neural system services 1110 for the control of at least one neural processor 179a to 179c.

[0276] On the other hand, when multiple applications request multiple operations, the neural system service 1110 can control the arrangement of data for the multiple operations in the internal memory 1805 within the first neural processor 179a. This enables the neural processor to operate efficiently.

[0277] On the other hand, the neural system service 1110 may operate or have a neural manager 1113 for managing at least one neural processor 179a to 179c, a neural controller 1115 for determining or controlling the reasoning mode of at least one neural processor 179a to 179c, and a neural interface 1118 for connecting to the interface of at least one neural processor 179a to 179c.

[0278] On the other hand, the neural system service 1110 may also run or have an interface for executing model parameters related to the actions of the neural processor 179, and a model container 509 for version management of learning files.

[0279] The Neural Manager 1113 can perform artificial intelligence model management, learning model management, camera data management, sensor data management, or command queue management.

[0280] The neural controller 1115 can determine the optimal reasoning mode for at least one neural processor 179a-179c, or execute queues, partitions, caches, or scalable codes, or control at least one neural processor 179a-179c.

[0281] The neural interface 1118 can run an application programming interface (API) associated with the accelerator of at least one neural processor 179a-179c.

[0282] On the other hand, the interface 522 within the first virtual machine 810 can perform interface connections between the neural system service 1110 and the model container 509 or between the neural system service 1110 and the shared memory 508.

[0283] On the other hand, interface 522 within the first virtual machine 810 can perform interface connections for the first virtual machine 810.

[0284] On the other hand, interface 522 within the first virtual machine 810 can perform interface connections for the vehicle driving assistance application Nad or the driver monitoring system application Ndm running in the second virtual machine 850 or the augmented reality application Nar running in the third virtual machine 830.

[0285] For example, the interface 522 within the first virtual machine 810 can be controlled to transmit camera data, sensor data, or voice data to the neural processor 179 using the shared memory 508.

[0286] On the other hand, the interface 522 within the first virtual machine 810 can be controlled to transmit the result data output from the neural processor 179 and recorded in the shared memory 508 to the neural system service 1110.

[0287] On the other hand, the interface 522 within the first virtual machine 810 can be controlled to transmit the result data output from the neural processor 179 and recorded in the shared memory 508 to the vehicle driving assistance application Nad or the driver monitoring system application Ndm running in the second virtual machine 850 or the augmented reality application Nar running in the third virtual machine 830.

[0288] On the other hand, the first virtual machine 810 can run on the first operating system 805, the second virtual machine 850 can run on the second operating system 805b with a high security level, and the third virtual machine 830 can run on the third operating system 805c.

[0289] That is, multiple virtual machines 810, 830, and 850 can run on different operating systems, or they can run on at least two operating systems.

[0290] On the other hand, the second operating system 805b can be an operating system corresponding to the second security level, such as ASIL D, and the third operating system 805c can be an operating system corresponding to the first security level, such as ASIL B.

[0291] On the other hand, the first operating system 805 can be an operating system corresponding to the second security level, such as ASIL D. However, it is not limited to this; the first operating system 805 can also be an operating system corresponding to the first security level, such as ASIL B.

[0292] On the other hand, the neural manager 1113 can manage the driving requirements of applications based on artificial neural networks, control neural network weight data, and process the required input data.

[0293] On the other hand, the neural manager 1113 processes the optimized command queue sequentially through a hardware accelerator and transmits the computation results to the application.

[0294] The driving requirements can include the computational priority, dependencies, and accuracy of the neural network. Computational priority refers to the relationship that the first operation must always be processed before the second operation, or, in the case of a safety-critical neural network, it must be processed first in the command queue than other candidate neural networks, and this is a pre-defined value.

[0295] On the other hand, neural network weight data refers to the file that structures and stores the element values ​​of each matrix during the reasoning process of the neural network results calculated through a series of matrix operations.

[0296] Neural network weight data can be pre-stored in model container 509 within neural system service 1110 via API calls during application installation.

[0297] On the other hand, the basic weight data loaded into model container 509 can be automatically converted and stored during system initialization at various discretization levels. For example, if the basic weights are defined as FP32, they can be further discretized into INT8, INT16, and FP16, thus storing a total of four weight files.

[0298] The required input data can refer to vehicle speed, current position, radar, lidar, camera images, and intermediate to final computational results of the preceding neural network, etc., which are the input signals required for the current neural network's actions.

[0299] Input data can be transmitted in real time from the server virtual machine to the shared memory 508 within the hypervisor 505 via an interface that is activated according to the central processing unit 175.

[0300] A command queue is a memory buffer of a sequential FIFO data structure that can define a series of sequences for processing artificial neural networks via hardware accelerators.

[0301] A neural network computation request entering the command queue can be sent along with the following information: application name, application virtual machine location, metadata such as the storage destination of the computation results, hardware accelerator control settings, memory location of the input data, and memory location information of the weight data at each discretization level.

[0302] Hardware accelerator control settings may include the inherent number of the hardware accelerator responsible for computation, the current object discretization level of the weight data (INT8, INT16, FP16, FP32, etc.), and the object neural network weight position mapping table according to the memory address of each hardware accelerator.

[0303] On the other hand, the neural controller 1115 can be controlled to schedule an optimized command queue based on the requested artificial neural network operation command and the availability of current hardware resources, and match the actual hardware accelerator with the expected action of the command queue.

[0304] The neural controller 1115 can receive the necessary information for driving the neural network from the neural manager 1113 and optimize the command queue.

[0305] The optimization process, in other words, involves determining the priority, dependency, and accuracy metadata of all slots in the current command queue, and applying various queue optimization techniques (Partition, Caching, Accuracy Coding, etc.) to all candidate commands in the current command queue. This allows for the simulation of scheduling to calculate a combination of directions that maximize hardware utilization and minimize the latency of individual computation requests per unit time.

[0306] Based on the optimal slot position thus determined, the weight file (learning model) can be requested from the neural manager 1113 and loaded into the internal hardware memory.

[0307] If we use partitioning techniques in optimization to manage two different neural networks like a virtual neural network and use them as inputs for hardware computation requests, we can record the start and end positions of the weights of the first operation, corresponding to the addresses in the internal hardware memory, in a mapping table, and then record the start and end positions of the weights of the second operation in the mapping table.

[0308] Thus, although the hardware accelerator appears to be performing parallel processing on a virtual neural network, the neural controller 1115 can use a mapping table to further separate the results of the operation into the results of the first operation and the results of the second operation and send them to individual applications.

[0309] After the initialization process is completed, the neural controller 1115 can receive sequential processing requests from the command queue from the neural manager 1113.

[0310] At this time, the neural controller 1115 can be controlled to retrieve the input data prepared in advance by the neural manager 1113 from the input data queue, pair the neural network weights with their corresponding input data, and perform computational processing through the hardware accelerator API.

[0311] If, unlike the initial requirements, the discretization level of the current neural network changes due to a specific condition, the neural controller 1115 can perform a bitwise concanate operation by concatenating the weight conversion difference (Delta) of the hardware internal memory to the basic weights of the current internal memory, thereby converting the discretization level of the basic weights of the internal memory in real time.

[0312] On the other hand, the central processing unit 175 runs an application for driving the vehicle, and if it determines that the application has failed, it controls the operation to run a second application corresponding to the application on another central processing unit 175 or other signal processing device 170, and changes the baseline rollback guarantee time for the application's failure based on the application's security level.

[0313] On the other hand, the aforementioned rollback guarantee time can refer to the period from the start time of the rollback to the end time of the rollback.

[0314] Alternatively, the rollback assurance time can refer to the period from the point when the application's action is judged to have failed or malfunctioned, through the rollback start time, to the rollback end time.

[0315] On the other hand, safety level can refer to Automotive Safety Integrity Level (ASIL) or Autonomous Driving Level or a combination of Automotive Safety Integrity Level and Autonomous Driving Level.

[0316] This enables the stable execution of applications used for vehicle operation. In particular, it enables the stable execution of applications used for vehicle operation based on safety levels.

[0317] On the other hand, the central processing unit 175 can set the reference rollback assurance time to a first time when the application's security level is the corresponding first security level, and set the reference rollback assurance time to a second time that is longer than the first time when the application's security level is a second security level that is higher than the first security level. This allows for the stable execution of applications used for vehicle operation.

[0318] For example, the central processing unit 175 can set the baseline backoff assurance time to approximately 10 seconds as the first time for a first application corresponding to ASILD when the autonomous driving level is Level 3, and set the baseline backoff assurance time to approximately 30 seconds as the second time for a second application corresponding to ASILD when the autonomous driving level is Level 4. This allows for the stable execution of applications for vehicle operation based on the safety level.

[0319] As another example, the central processing unit 175 can set the baseline backoff assurance time to approximately 7 seconds for a third application corresponding to ASIL B when the autonomous driving level is Level 3, and set the baseline backoff assurance time to approximately 10 seconds for a fourth application corresponding to ASIL D when the autonomous driving level is Level 3. This allows for the stable execution of applications for vehicle operation based on the safety level.

[0320] As another example, the central processing unit 175 can set the baseline rollback assurance time to approximately 10 seconds as the first time when the safety level of the augmented reality application Nar is the first safety level corresponding to ASIL B, and set the baseline rollback assurance time to approximately 30 seconds as the second time when the safety level of the vehicle driving assistance application Nad is the second safety level corresponding to ASIL D, which is higher than ASIL B. Thus, applications for vehicle driving can be executed stably based on the safety level.

[0321] On the other hand, the central processing unit 175 can set the baseline backoff assurance time to a third time that is shorter than the first time when the application's security level is a third security level, which is lower than the first security level. This enables stable operation of applications used for vehicle operation.

[0322] For example, the central processing unit 175 can set the baseline backoff assurance time to approximately 1 second as the third time for the fifth application corresponding to ASILD or ASIL B in a state of autonomous driving level 2.

[0323] As another example, the central processing unit 175 can set the baseline rollback assurance time to approximately 0.7 seconds for the sixth application corresponding to QM when the autonomous driving level is Level 2.

[0324] As another example, the central processing unit 175 can set the baseline backoff assurance time to approximately 0.5 seconds, which is the third safety level corresponding to QM, which is lower than ASIL B, when the safety level of the augmented reality application Nar is the third safety level. This allows for stable execution of applications for vehicle operation based on the safety level.

[0325] On the other hand, if the second virtual machine 850 among the plurality of virtual machines 810, 830, and 850 runs an application with a higher security level than the third virtual machine 830, the central processing unit 175 can control the baseline rollback assurance time of the application running on the second virtual machine 850 to be greater than the baseline rollback assurance time of the application running on the third virtual machine.

[0326] For example, the central processing unit 175 can be configured such that when the second virtual machine 850 is running a first application with an autonomous driving level of Level 4, the baseline rollback assurance time is approximately 30 seconds, and when the third virtual machine 830 is running a second application with an autonomous driving level of Level 3, the baseline rollback assurance time is approximately 10 seconds. This allows for the stable execution of applications for vehicle operation based on safety levels.

[0327] As another example, the central processing unit 175 can be configured such that, when the second virtual machine 850 is running the vehicle driving assistance application Nad corresponding to ASIL D, the baseline rollback guarantee time for Nad is approximately 30 seconds, and when the third virtual machine 830 is running the augmented reality application Nar corresponding to ASIL B, the baseline rollback guarantee time for Nar is approximately 10 seconds. This allows for the stable execution of applications for vehicle driving based on safety levels.

[0328] Figures 10a to 10c This is a diagram used to illustrate the operation of the vehicle control device related to this disclosure.

[0329] Figure 10a An example of vehicle driving based on lane-keeping mode is shown.

[0330] especially, Figure 10a An example is given of a scenario where, based on lane keeping mode, vehicle 200 is traveling between the first lane line LNa and the second lane line LNb, and a large truck OBm is traveling in the adjacent lane.

[0331] Referring to the accompanying drawings, the vehicle control device related to this disclosure can detect lane lines LNa and LNb based on images from a camera, and control the vehicle to maintain a constant state of keeping the vehicle in the center of the two lane lines LNa and LNb or at a first interval Dpa with the first lane line LNa when executing lane keeping mode.

[0332] When vehicle 200 is located in the center of two lane lines LNa and LNb, the distance between the center of vehicle 200 and the first lane line LNa can be DPm.

[0333] On the other hand, if the large truck OBM executes lane keeping mode while traveling between the second lane line LNb and the third lane line LNc, which are adjacent lanes, the vehicle control device related to this disclosure continues to maintain the vehicle in a constant state of being centered on the two lane lines LNa and LNb or maintaining a first interval with the first lane line LNa.

[0334] However, in situations where lane spacing is kept constant according to lane keeping mode, drivers feel uneasy due to large trucks (OBm).

[0335] Figure 10b This shows another example of vehicle driving based on lane-keeping mode.

[0336] especially, Figure 10b An example is given of a scenario where, based on lane keeping mode, vehicle 200 is traveling between the first lane line LNa and the second lane line LNb, and the road boundary or guardrail RBm is located next to the first lane line LNa.

[0337] Referring to the accompanying drawings, when the road boundary or guardrail RBm is located next to the first lane line LNa, if the lane keeping mode is executed, the vehicle control device related to this disclosure continues to maintain the vehicle in a constant state where it is located in the center of the two lane lines LNa and LNb or at a first interval from the first lane line LNa.

[0338] However, if the lane spacing is kept constant according to this lane keeping pattern, the driver may feel uneasy due to the road boundaries or guardrail RBm.

[0339] Figure 10c This provides another example of vehicle driving based on lane-keeping mode.

[0340] especially, Figure 10c An example is given of a situation where, based on lane keeping mode, vehicle 200 is traveling between the first lane line LNa and the second lane line LNb, and emergency vehicle 200f approaches the rear of vehicle 200.

[0341] Specifically, an example is given of a scenario where multiple vehicles 200mb, 200mc, and 200md are traveling behind vehicle 200, and emergency vehicle 200f is located between vehicle 200 and the multiple vehicles 200mb, 200mc, and 200md.

[0342] Multiple vehicles 200mb, 200mc, and 200md can adjust their direction of travel to increase the interval between vehicles in order to facilitate the movement of emergency vehicle 200f.

[0343] On the other hand, if lane keeping mode is executed while an emergency vehicle 200f is approaching from behind vehicle 200, the vehicle control device related to this disclosure continues to maintain the vehicle in a constant state where it is located in the center of the two lane lines LNa and LNb or at a first interval from the first lane line LNa.

[0344] However, if the lane keeping mode is to maintain a constant lane spacing, there is an inconvenience that lane spacing cannot be adjusted for emergency vehicles 200f.

[0345] Therefore, this disclosure further subdivides the lane-keeping modes, thereby enabling the execution of a second lane-keeping mode that adaptively adjusts the lane spacing while maintaining a constant lane spacing according to the first lane-keeping mode, taking into account vehicle driving conditions or driver status. This will be discussed later. Figure 11a Please provide an explanation.

[0346] Figure 11a This is a flowchart illustrating the operation method of a signal processing apparatus according to an embodiment of the present disclosure.

[0347] Referring to the accompanying drawings, in one embodiment of the present disclosure, the processor 175 in the signal processing apparatus 170 receives a frontal image from the front camera 195a and detects lane lines based on the frontal image (S1105).

[0348] On the other hand, in addition to the front image from the front camera 195a, the processor 175 can also detect lane lines based on side images, map information from the memory 140, and location information such as GPS from the communication unit 120.

[0349] On the other hand, the processor 175 can detect objects outside the lane lines based on the front or side images from the front camera 195a, as well as various objects such as surrounding vehicles, signs, pedestrians, traffic lights, and road borders.

[0350] On the other hand, in addition to front or side images from the front camera 195a, the processor 175 can also detect various objects such as surrounding vehicles, signs, pedestrians, traffic lights, and road borders based on map information from the memory 140 and location information such as GPS from the communication unit 120.

[0351] Next, the processor 175 within the signal processing apparatus 170 of an embodiment of the present disclosure can be controlled to execute a lane keeping mode based on lane line detection (S1110).

[0352] On the other hand, the processor 175 within the signal processing device 170 can determine whether it is the first mode in the lane keeping mode (S1115), and execute the first mode in the lane keeping mode if it is.

[0353] That is, the processor 175 in the signal processing device 170 executes the lane keeping mode based on lane line detection, and controls the steering drive unit 752 to maintain a first distance DPa with the first lane line LNa in the adjacent first lane line LNa and second lane line LNb according to the first mode in the lane keeping mode (S1120).

[0354] On the other hand, in step 1115 (S1115), if it is not the first mode in the lane keeping mode, the processor 175 in the signal processing device 170 determines whether it is the second mode in the lane keeping mode (S1125), and if so, executes the second mode in the lane keeping mode.

[0355] That is, the processor 175 in the signal processing device 170 executes the lane keeping mode based on lane line detection, and controls the steering drive unit 752 to maintain a second interval DPb different from the first interval Dpa according to the second mode in the lane keeping mode (S1130).

[0356] Therefore, it can reflect driving conditions and adaptively adjust lane spacing during lane keeping mode operation.

[0357] Figure 11b This is a flowchart illustrating an operation method of a signal processing apparatus according to another embodiment of the present disclosure.

[0358] Referring to the accompanying drawings, in one embodiment of the present disclosure, the processor 175 in the signal processing apparatus 170 receives a frontal image from the front camera 195a and detects lane lines based on the frontal image (S1105).

[0359] Next, the processor 175 within the signal processing apparatus 170 of an embodiment of the present disclosure can be controlled to execute a lane keeping mode based on lane line detection (S1110).

[0360] On the other hand, the processor 175 within the signal processing device 170 can determine whether it is the first mode in the lane keeping mode (S1115), and execute the first mode in the lane keeping mode if it is.

[0361] That is, the processor 175 in the signal processing device 170 executes the lane keeping mode based on lane line detection, and controls the steering drive unit 752 to keep the vehicle in the center of the adjacent first lane line LNa and second lane line LNb according to the first mode in the lane keeping mode (S1120b).

[0362] On the other hand, in step 1115 (S1115), if it is not the first mode in the lane keeping mode, the processor 175 in the signal processing device 170 determines whether it is the second mode in the lane keeping mode (S1125), and if so, executes the second mode in the lane keeping mode.

[0363] That is, the processor 175 in the signal processing device 170 executes the lane keeping mode based on lane line detection, and controls the steering drive unit 752 to get closer to either the first lane line LNa or the second lane line LNb according to the second mode in the lane keeping mode (S1130b).

[0364] Therefore, it can reflect driving conditions and adaptively adjust lane spacing during lane keeping mode operation.

[0365] On the other hand, the processor 175 within the signal processing device 170 can control the steering drive unit 752 to maintain a first distance Dpa from the first lane line LNa, so as to remain in the center of the adjacent first lane line LNa and second lane line LNb, according to the first mode in the lane keeping mode.

[0366] On the other hand, the processor 175 within the signal processing device 170 can control the steering drive unit 752 to maintain a second interval DPb with the first lane line LNa, based on the first mode in the lane keeping mode, so as to get closer to either the first lane line LNa or the second lane line LNb. Preferably, the second interval DPb is different from the first interval Dpa at this time.

[0367] Therefore, it can reflect driving conditions and adaptively adjust lane spacing during lane keeping mode operation.

[0368] Figures 12a to 18 This is an explanation Figures 11a to 11b The action is referenced in the diagram.

[0369] first, Figure 12a This example illustrates the first mode of lane keeping mode.

[0370] Referring to the accompanying drawings, in one embodiment of the signal processing apparatus 170, the processor 175 performs lane line detection based on a forward image from the camera 195, and performs a lane keeping mode based on the lane line detection.

[0371] In particular, according to a first mode in the lane keeping mode, the processor 175 in the signal processing device 170 of this disclosure controls the steering drive unit 752 to maintain a first distance Dpa with the first lane line LNa in the adjacent first lane line LNa and second lane line LNb.

[0372] Alternatively, in another embodiment of the present disclosure, the processor 175 within the signal processing device 170 controls the steering drive unit 752 to remain centered on the adjacent first lane line LNa and second lane line LNb according to a first mode in the lane keeping mode.

[0373] Thus, vehicle 200 travels along DRa in the direction of maintaining the center of the adjacent first lane line LNa and second lane line LNb.

[0374] On the other hand, if based on the front, side, or rear images from the camera 195, there are no adjacent vehicles around the vehicle, or no road boundaries or guardrails RBm, or no emergency-moving vehicles 200f, then the processor 175 within the signal processing apparatus 170 of an embodiment of this disclosure can be controlled to execute the first mode in the lane-keeping mode.

[0375] Figure 12b This example illustrates a scenario where the driver's line of sight is positioned at the vehicle OBk in front.

[0376] Referring to the accompanying drawings, the processor 175 within the signal processing device 170 can detect the direction of the driver OWA's face or the direction of the driver OWA's gaze based on the internal image from the internal camera 195i.

[0377] On the other hand, such as Figure 12b As shown, if the driver Owa's line of sight OPa is located in front of the vehicle OBk, the processor 175 in the signal processing device 170 can set the forward gaze level to the first level.

[0378] On the other hand, if the driver Owa's gaze direction OPa is located on the vehicle OBk ahead, or if the forward gaze level is Level 1 and above the baseline level, the processor 175 within the signal processing device 170 can control the system to execute the first mode of lane-keeping mode. Thus, as... Figure 12a As shown, the first mode in lane keeping mode can be executed.

[0379] On the other hand, if with Figure 12b In contrast, if the driver Owa's gaze direction Opa is not located at the vehicle OBk in front, but at the side of the vehicle, the processor 175 in the signal processing device 170 can set the forward gaze level to a second level, which is lower than the first level.

[0380] On the other hand, if the driver Owa's gaze direction Opa is not located on the vehicle OBk in front, or if the forward gaze level is level two and lower than the baseline level, the processor 175 within the signal processing device 170 can control the execution of the second mode in the lane keeping mode.

[0381] then, Figure 12c This example illustrates the second mode in lane keeping mode.

[0382] Referring to the accompanying drawings, in one embodiment of the signal processing apparatus 170, the processor 175 performs lane line detection based on a forward image from the camera 195, and performs a lane keeping mode based on the lane line detection.

[0383] On the other hand, the processor 175 within the signal processing apparatus 170 of one embodiment of the present disclosure can detect surrounding vehicle objects based on a front image or a side image from the camera 195.

[0384] For example, as shown in the figure, if the second vehicle OBm is located in the adjacent lane, the processor 175 in the signal processing device 170 of an embodiment of the present disclosure can be controlled to execute the second mode in the lane keeping mode.

[0385] As another example, if the second vehicle OBm is located in an adjacent lane and the size of the second vehicle OBm is greater than the reference size, the processor 175 within the signal processing apparatus 170 of an embodiment of the present disclosure can be controlled to execute the second mode in the lane keeping mode.

[0386] As another example, if the second vehicle OBm is in the adjacent lane and the driver Owa's line of sight Opa is in the direction of the second vehicle OBm, the processor 175 in the signal processing device 170 of an embodiment of the present disclosure can be controlled to execute the second mode in the lane keeping mode.

[0387] As another example, if the second vehicle OBm is in the adjacent lane and the driver Owa's line of sight Opa is in the direction of the second vehicle OBm, and the driver Owa's emotion is surprise or fear, then the processor 175 in the signal processing device 170 of an embodiment of the present disclosure can be controlled to execute the second mode in the lane keeping mode.

[0388] On the other hand, in one embodiment of the signal processing apparatus 170, the processor 175 controls the steering drive unit 752 to maintain a second distance DPb smaller than the first distance Dpa with respect to the first lane line LNa in the lane keeping mode, according to the second mode in the lane keeping mode. Thus, it is possible to reflect the driving situation and adaptively adjust the lane spacing during lane keeping mode operation.

[0389] Alternatively, in another embodiment of this disclosure, the processor 175 within the signal processing device 170 controls the steering drive unit 752 to move closer to the first lane line LNa in the first lane line LNa and the second lane line LNb according to the second mode in the lane keeping mode.

[0390] That is, the processor 175 in the signal processing device 170 of another embodiment of the present disclosure can control the second mode in the lane keeping mode to make the interval between the center of the vehicle 200 and the first lane line LNa smaller than DPm DPma.

[0391] Specifically, if the second vehicle OBM is located on one side, the processor 175 in the signal processing device 170 can take into account the driver Owa's line of sight OPa, emotional state, etc., and control the vehicle 200 to move slightly to the opposite side while maintaining the driving lane.

[0392] Ultimately, the processor 175 within the signal processing device 170 of one embodiment of this disclosure can control the vehicle's travel direction to be like DRb, rather than DRa, according to the second mode in the lane keeping mode. Thus, it is possible to reflect driving conditions and adaptively adjust lane spacing during lane keeping mode operation.

[0393] On the other hand, the processor 175 can detect the second vehicle OBm on the side of the vehicle based on the front or side image from the camera 195, and change the second interval DPb in the second mode of the lane keeping mode based on the size of the second vehicle OBm.

[0394] For example, the processor 175 can control the lane spacing so that the larger the size of the second vehicle OBm, the smaller the second lane interval DPb, or the smaller the DPma, the closer the lane spacing is to the first lane line LNa. Thus, the lane spacing can be adaptively adjusted based on the size of the second vehicle OBm.

[0395] On the other hand, the processor 175 can be controlled to change the second interval DPb based on the speed of the vehicle 200 in the second mode of lane keeping mode.

[0396] For example, the processor 175 can control the lane spacing so that the higher the speed of the vehicle 200, the smaller the second lane interval DPb, or the closer it is to the first lane line LNa. Thus, the lane spacing can be adaptively adjusted according to the speed of the vehicle 200.

[0397] On the other hand, the processor 175 can change the second interval DPb in the second mode of the lane keeping mode based on the driver Owa's driving proficiency, the number of occupants in the vehicle 200, or whether there is anyone sitting in the front passenger seat.

[0398] For example, the processor 175 can control the lane spacing to be smaller or closer to the first lane line LNa if the driver Owa has a lower driving skill level or if there are more passengers in the vehicle 200. Thus, the lane spacing can be adaptively adjusted based on driving skill level or the number of passengers.

[0399] As another example, the processor 175 can be controlled to make the second lane spacing DPb smaller or closer to the first lane line LNa when someone is sitting in the passenger seat, compared to when no one is sitting in the passenger seat. Thus, the lane spacing can be adaptively adjusted based on whether someone is sitting in the passenger seat.

[0400] On the other hand, the processor 175 in the signal processing apparatus 170 of one embodiment of the present disclosure can be controlled to detect pedestrians on the side of the vehicle based on the front image or side image from the camera 195 when executing the second mode in the lane keeping mode, and change the second interval DPb based on the number of pedestrians detected or the position of the pedestrians.

[0401] For example, in one embodiment of the signal processing apparatus 170 of this disclosure, the processor 175 can be controlled to reduce the second lane interval DPb or make the pedestrians closer to the first lane line LNa and the second lane line LNb closer to the second lane line LNb, as the number of pedestrians approaching LNa or the distance from the pedestrians to LNb increases. Thus, the lane interval can be adaptively adjusted based on the detected pedestrians.

[0402] On the other hand, the processor 175 can detect the driver Owa's gaze based on the internal image from the internal camera 195i, and change the second interval DPb in the second mode of the lane keeping mode based on the direction of the driver Owa's gaze.

[0403] For example, the processor 175 within the signal processing apparatus 170 of one embodiment of this disclosure can be controlled such that the driver OWa's line of sight is further away from the driver's line of sight. Figure 12b The closer the second lane spacing (DPb) is to the vehicle ahead (OBk), or the closer it is to the first lane line (LNa), the more adaptively the lane spacing can be adjusted based on the driver's (OWa) line of sight.

[0404] On the other hand, in manual driving mode instead of lane keeping mode, the processor 175 can control the driver's line of sight (OWa) to detect the distance between the driver's line of sight (OWa) and the first lane line (LNa) in the forward image based on the internal image from the internal camera 195i, and learn based on the driver's line of sight (OWa) and the distance between the driver's line of sight (OWa) and the first lane line (LNa), and then store the learning results in the memory 140.

[0405] On the other hand, the processor 175 can set the second interval DPb based on the learning results when executing the second mode in the lane keeping mode.

[0406] For example, if in manual driving mode, the driver OWA's forward gaze level is at level one and the distance between the driver and the first lane line LNa is less than DPa, then the processor 175 can learn to master the driver OWA's driving pattern as a first driving pattern that approaches the first lane line LNa in the first lane line LNa and the second lane line LNb.

[0407] On the other hand, the processor 175 can set a second interval DPb when executing the second mode in the lane keeping mode based on a first driving behavior mode according to the learning results.

[0408] As another example, if in manual driving mode the driver OWA's forward gaze level is a second level lower than the first level, and the distance from the first lane line LNa is less than DPa, then the processor 175 can learn to master the driver OWA's driving behavior pattern as a second driving behavior pattern that approaches the first lane line LNa in the first lane line LNa and the second lane line LNb.

[0409] On the other hand, the processor 175 can set the second lane spacing to be smaller than the first lane spacing when executing the second mode of the lane keeping mode, based on the second driving behavior mode according to the learning results. Thus, the lane spacing can be adaptively adjusted based on learning.

[0410] As another example, in manual driving mode, if the driver OWA's forward gaze level is at level one, the distance between the driver and the first lane line LNa is less than DPa, and smaller than the first driving behavior mode, then the processor 175 can learn to master the driver OWA's driving behavior mode as a third driving behavior mode that is close to the first lane line LNa in the second lane line LNb.

[0411] On the other hand, the processor 175 can set the second lane spacing to be smaller than the first lane spacing when executing the second mode in the lane keeping mode, based on the third driving behavior mode learned from the learning results. Thus, the lane spacing can be adaptively adjusted based on learning.

[0412] On the other hand, in manual driving mode, the processor 175 can detect the driver OWA's emotional information based on internal images from the internal camera 195i, detect the distance between the driver OWA and the first lane line LNa in the forward image, and learn based on the driver OWA's emotional information and the distance between the driver OWA and the first lane line LNa, and then store the learning results in memory. Thus, the lane spacing can be adaptively adjusted based on the driver OWA's emotional information.

[0413] For example, processor 175 can detect driver OWA's emotional information when the second vehicle is located to the side, based on the driver OWA's eye movements or facial movements, in manual driving mode.

[0414] On the other hand, in manual driving mode, if the fear level of driver OWA when the second vehicle is on the side is level 1 and the distance between driver OWA and first lane line LNa is less than DPa, then processor 175 can learn to master driver OWA's driving behavior pattern as the first driving behavior pattern of approaching first lane line LNa in the first lane line LNa and second lane line LNb.

[0415] On the other hand, the processor 175 can set a second interval DPb when executing the second mode in the lane keeping mode based on a first driving behavior mode according to the learning results.

[0416] As another example, in manual driving mode, if the fear level of driver OWA when the second vehicle is on the side is a second level higher than the first level, and the distance between driver OWA and the first lane line LNa is less than DPa, then processor 175 can learn to master driver OWA's driving behavior pattern as a second driving behavior pattern that approaches the first lane line LNa in the first lane line LNa and the second lane line LNb.

[0417] On the other hand, the processor 175 can set the second lane spacing to be smaller than the first lane spacing when executing the second mode of the lane keeping mode, based on the second driving behavior mode according to the learning results. Thus, the lane spacing can be adaptively adjusted based on learning.

[0418] As another example, in manual driving mode, if the fear level of driver OWA when the second vehicle is on the side is level one, the distance between driver OWA and first lane line LNa is less than DPa and smaller than the first driving behavior mode, then processor 175 can learn to master driver OWA's driving behavior mode as a third driving behavior mode that is close to first lane line LNa in the first lane line LNa and second lane line LNb.

[0419] On the other hand, the processor 175 can set the second lane spacing to be smaller than the first lane spacing when executing the second mode in the lane keeping mode, based on the third driving behavior mode learned from the learning results. Thus, the lane spacing can be adaptively adjusted based on learning.

[0420] On the other hand, in manual driving mode, the processor 175 can be controlled to detect the driver OWA's gaze direction and emotional information based on internal images from the internal camera 195i, detect the distance between the driver OWA and the first lane line LNa in the forward image, and learn based on the driver OWA's gaze direction information, emotional information, and distance between the driver OWA and the first lane line LNa, and then store the learning results in memory. Thus, the lane spacing can be adaptively adjusted based on the driver OWA's gaze direction information and emotional information.

[0421] then, Figure 12d This example illustrates another instance of the second mode in lane-keeping mode.

[0422] Referring to the accompanying drawings, the processor 175 within the signal processing apparatus 170 of one embodiment of the present disclosure can be controlled to execute the second mode of the lane keeping mode when the second vehicle OBm is located in the adjacent lane as shown in the figure.

[0423] On the other hand, with Figure 12c In contrast, the processor 175 can control the vehicle 200 to have a lower speed, or the driver OWA to have a higher driving skill, or the vehicle 200 to have fewer passengers, or the number of pedestrians near the second lane line LNb to have fewer, so that the second interval between the vehicle 200 and the first lane line LNa becomes DPc, which is greater than DPb, or the interval between the center of the vehicle 200 and the first lane line LNa becomes DPmb, which is greater than DPma.

[0424] Preferably, DPc is less than DPa and DPmb is less than DPm at this point. This allows the system to reflect driving conditions and adaptively adjust lane spacing during lane-keeping mode operation.

[0425] Ultimately, the processor 175 within the signal processing apparatus 170 of one embodiment of this disclosure can control the vehicle's driving direction to become like DRc instead of DRa according to the second mode in the lane keeping mode.

[0426] On the other hand, if a second vehicle of a road boundary or guardrail RBm or a reference size is located on the side of the vehicle based on a frontal or side image from camera 195, then the processor 175 within the signal processing apparatus 170 of an embodiment of this disclosure can be controlled to execute the second mode in the lane keeping mode.

[0427] The second lane-keeping mode in the lane-keeping modes related to road boundaries or guardrails (RBm) will refer to... Figures 13a to 13d Please provide an explanation.

[0428] Figure 13a This example illustrates the first mode of lane keeping mode.

[0429] Referring to the accompanying drawings, if, based on a frontal or side image from camera 195, there are no adjacent vehicles, or no road boundaries or guardrails RBm, or no emergency-moving vehicles 200f, then the processor 175 within the signal processing apparatus 170 of an embodiment of this disclosure can be controlled to execute the first mode in the lane-keeping mode.

[0430] Figure 13b This example illustrates a scenario where the driver's line of sight is at the road edge or guardrail RBm.

[0431] Referring to the accompanying drawings, the processor 175 within the signal processing device 170 can detect the direction of the driver OWA's face or the direction of the driver OWA's gaze based on the internal image from the internal camera 195i.

[0432] On the other hand, such as Figure 13b As shown, if the driver OWA's line of sight OPb is located at the road boundary or guardrail RBm, the processor 175 within the signal processing unit 170 can control the system to execute the second mode of lane-keeping mode. Thus, as... Figure 13c As shown, the second mode in lane keeping mode can be executed.

[0433] On the other hand, such as Figure 13b As shown, if the driver OWA's line of sight OPb is located at the road boundary or guardrail RBm, and the driver OWA's emotion is surprise or fear, the processor 175 in the signal processing device 170 can control the execution of the second mode in the lane keeping mode.

[0434] Specifically, if the road boundary or guardrail RBm is located on one side, the processor 175 in the signal processing device 170 can take into account the driver OWA's line of sight OPa, emotional state, etc., and control the vehicle 200 to move slightly to the opposite side while maintaining the driving lane.

[0435] then, Figure 13c This is yet another example of the second mode in lane keeping mode.

[0436] Referring to the accompanying drawings, in one embodiment of the signal processing apparatus 170, the processor 175 performs lane line detection based on a forward image from the camera 195, and performs a lane keeping mode based on the lane line detection.

[0437] On the other hand, the processor 175 in the signal processing apparatus 170 of one embodiment of the present disclosure can detect surrounding vehicle objects, road boundaries, or guardrails based on the front or side images from the camera 195.

[0438] For example, as shown in the figure, if the road boundary or guardrail RBm is located on the side of the vehicle, the processor 175 in the signal processing device 170 of an embodiment of the present disclosure can be controlled to execute the second mode in the lane keeping mode.

[0439] Specifically, if the road boundary or guardrail RBm is located close to the first lane line LNa in the first lane line LNa and the second lane line LNb, the processor 175 can control the vehicle 200 to be closer to the second lane line LNb than the first lane line LNa.

[0440] That is, if the road boundary or guardrail RBm is located close to the first lane line LN a in the first lane line LNa and the second lane line LNb, the processor 175 can control the second interval with the first lane line LNa to be greater than DPd, or the interval between the center of the vehicle 200 and the first lane line LNa to be greater than DPm. Thus, it is possible to reflect driving conditions and adaptively adjust the lane spacing during lane keeping mode operation.

[0441] Ultimately, the processor 175 within the signal processing apparatus 170 of one embodiment of this disclosure can control the vehicle's driving direction to be like DRd instead of DRa according to the second mode in the lane keeping mode.

[0442] then, Figure 13d This is yet another example of the second mode in lane keeping mode.

[0443] Referring to the attached diagram, Figure 13d and Figure 13c The difference is that the second vehicle, OBM, is located to the side of the second lane line, LNb.

[0444] On the other hand, if the road boundary or guardrail RBm is located close to the first lane line LNa, and the second vehicle OBm is located close to the second lane line LNb, the processor 175 in the signal processing device 170 of an embodiment of the present disclosure can change the interval with the first lane line LNa.

[0445] For example, if it is determined that driver OWa is more afraid of road boundaries or guardrails than the second vehicle OBm, then Figure 13d As shown, the processor 175 can be controlled to be closer to the second lane line LNb in the first lane line LNa and the second lane line LNb.

[0446] As another example, if it is determined that driver OWa is more afraid of the second vehicle OBm than the road boundary or guardrail RBm, then processor 175 can... Figure 13d The control is different, with the first lane line LNa being closer to the first lane line LNa and the second lane line LNb.

[0447] As shown in the figure, if the road boundary or guardrail RBm is located close to the first lane line LNa, and the second vehicle OBm is located close to the second lane line LNb, then the processor 175 can control the second interval between the vehicle and the first lane line LNa to be DPe, which is greater than DPa and less than DPd, or the interval between the center of the vehicle 200 and the first lane line LNa to be DPnb, which is greater than DPm and less than DPna. Thus, the lane spacing can be adaptively adjusted to reflect driving conditions during lane-keeping mode operation.

[0448] Ultimately, the processor 175 within the signal processing apparatus 170 of one embodiment of this disclosure can control the vehicle's driving direction to become like DRe instead of DRa according to the second mode in the lane keeping mode.

[0449] Figure 14a This is yet another example of the second mode in lane keeping mode.

[0450] Referring to the accompanying drawings, the processor 175 can detect the driver OWa's emotional information based on internal images from the internal camera 195i.

[0451] Furthermore, if the driver OWA's emotional information is detected as surprise or fear, the processor 175 can control the execution of the second mode in the lane-keeping mode.

[0452] For example, the processor 175 can control the lane keeping mode to ensure that the center line LNct between the first lane line LNa and the second lane line LNb is aligned with the center of the vehicle 200, based on the first mode in the lane keeping mode.

[0453] At this point, the distances between the center of vehicle 200 and the first lane line LNa and the second lane line LNb can be DPm and DPm, respectively. That is, the distances between the center of vehicle 200 and the first lane line LNa and the distances between the center of vehicle 200 and the second lane line LNb can be the same.

[0454] On the other hand, if the driver Owa's emotional information is surprise or fear during the execution of the first mode of lane keeping mode, the processor 175 can control the execution of the second mode of lane keeping mode.

[0455] That is, the processor 175 can control the center of the vehicle 200 to be closer to the first lane line LNa based on the left offset OFa, according to the second mode in the lane keeping mode.

[0456] Specifically, the processor 175 can control the second mode in the lane keeping mode to make the interval between the center of the vehicle 200 and the first lane line LNa smaller than DPm, DPc.

[0457] Furthermore, the processor 175 can control the lane spacing according to the second mode in the lane keeping mode, such that the distance between the center of the vehicle 200 and the second lane line LNb becomes DPd, which is larger than DPm. This allows for adaptive adjustment of the lane spacing based on the driver OWA's emotional information.

[0458] On the other hand, in the attached figure, the same offset is used when the emotional information is surprise or fear, but the processor 175 can control the offset to be larger when it is fear than when it is surprise.

[0459] For example, preferably, the processor 175 controls the distance between the center of the vehicle 200 and the first lane line LNa when the emotional information is surprise to be DPc, while the distance between the center of the vehicle 200 and the first lane line LNa when the emotional information is fear is less than DPc.

[0460] Specifically, the processor 175 can control the vehicle 200 to move closer to the first lane line LNa when the emotional information is fear rather than surprise. This allows for adaptive adjustment of lane spacing based on the driver OWA's emotional information.

[0461] Figure 14b This is a diagram illustrating the first and second modes of lane keeping operation.

[0462] Referring to the attached diagram, the processor 175 can control the vehicle 200 to align with the center line LNct between the first lane line LNa and the second lane line LNb based on the first mode of the lane keeping mode.

[0463] At this point, the intervals between vehicle 200 and the first lane line LNa and the second lane line LNb can be DPda and DPdb, respectively. Alternatively, DPda and DPdb can be of the same level.

[0464] On the other hand, the processor 175 can control the lane keeping mode to be closer to the first lane line LNa in the second lane keeping mode.

[0465] Specifically, the processor 175 can control the lane keeping mode so that the interval between the vehicle 200 and the first lane line LNa becomes a smaller DPdc than DPda, according to the second mode in the lane keeping mode.

[0466] Furthermore, the processor 175 can control the lane spacing according to the second mode in the lane keeping mode, such that the interval between the vehicle 200 and the second lane line LNb becomes DPdd, which is larger than DPdb. This allows for adaptive adjustment of the lane spacing.

[0467] Figure 15a This is an example where the road boundary or guardrail RBm is located on the first side of the vehicle, and the second vehicle is located on the second side of the vehicle.

[0468] Referring to the attached diagram, the processor 175 can control the vehicle to move closer to the first lane line LNa or further away from the second lane line LNb, according to the second mode in the lane keeping mode, so as to separate from the second vehicle OBM.

[0469] As shown in the figure, the processor 175 can control the interval between vehicle 200 and the second lane line LNb in the first mode of lane keeping mode to be smaller than DS2, which is the interval between vehicle 200 and the second lane line LNb in the second mode of lane keeping mode. In this case, the interval between vehicle 200 and the second vehicle OBm can be DS2b.

[0470] On the other hand, the processor 175 can control the second mode in the lane keeping mode to perform separation control with the road boundary or guardrail RBm if a road boundary or guardrail RBm appears during the separation action with the second vehicle OBm.

[0471] That is, the processor 175 can control the lane keeping mode so that the interval between vehicle 200 and the second lane line LNb becomes DS1, which is smaller than DS2. At this time, the interval between vehicle 200 and the second vehicle OBm can be DS1b.

[0472] Ultimately, the processor 175 can be controlled such that, when executing the second mode of lane-keeping mode, the offset when the object is on both sides is smaller than the offset when the object is only on one side. Therefore, the vehicle moves less when it is on both sides than when it is on one side.

[0473] Figure 15b An example is given where the second vehicle is located on the first side of the vehicle.

[0474] Referring to the attached diagram, the processor 175 can control the vehicle to move closer to the third lane line LNc or ​​further away from the second lane line LNb, in order to move away from the second vehicle OBm adjacent to the second lane line LNb, according to the second mode in the lane keeping mode.

[0475] As shown in the figure, the processor 175 can control the interval between vehicle 200 and the second lane line LNb in the first mode of lane keeping mode to be smaller than DS5, which is the interval between vehicle 200 and the second lane line LNb in the second mode of lane keeping mode. In this case, the interval between vehicle 200 and the second vehicle OBm can be DS5b.

[0476] On the other hand, if a third vehicle 200c adjacent to the third lane line LNc appears during the separation action from the second vehicle OBm in the second mode of the lane keeping mode, the processor 175 can control the separation control from the third vehicle 200c.

[0477] That is, the processor 175 can control the lane keeping mode so that the interval between vehicle 200 and the second lane line LNb becomes DS4, which is smaller than DS5. At this time, the interval between vehicle 200 and the second vehicle OBm can be DS4b.

[0478] Ultimately, the processor 175 can be controlled such that, when executing the second mode of lane keeping mode, the offset when the object is located on only one side is greater than the offset when the object is located on both sides. Therefore, the vehicle moves less when located on both sides than when located on only one side.

[0479] Figure 16a This example illustrates the scenario where the first mode of lane keeping is executed.

[0480] Referring to the accompanying drawings, the processor 175 can control the vehicle to remain in the center of the two lane lines LNa and LNb or maintain a constant distance Dpa from the first lane line LNa when executing the first mode of the lane keeping mode.

[0481] That is, when executing the first mode in the lane keeping mode, the processor 175 can control the first lane line LNa to have a first interval DPa, or the interval between the center of the vehicle 200 and the first lane line LNa to have a first interval DPm.

[0482] On the other hand, the plurality of vehicles 200mb, 200mc, and 200md arranged behind vehicle 200 can perform lane spacing adjustments for emergency vehicle 200f.

[0483] At this time, the emergency vehicle 200f can be an ambulance, police car, fire truck, etc.

[0484] On the other hand, if an emergency vehicle 200f approaches the rear of vehicle 200 during the execution of the first mode of lane keeping mode, the processor 175 can control the system to enter the second mode of lane keeping mode.

[0485] Figure 16b This example illustrates a scenario where the second mode of lane keeping is activated when an emergency vehicle approaches.

[0486] Referring to the accompanying drawings, the processor 175 can detect an emergency-moving vehicle 200f in the rear image based on the rear image from the camera 195.

[0487] On the other hand, in addition to object detection in the rear image from the camera 195, the processor 175 can also detect emergency vehicles 200f in the rear image based on sounds such as ambulances or fire trucks coming from behind the vehicle.

[0488] Alternatively, in addition to object detection in the rear image from the camera 195, the processor 175 can also detect emergency-moving vehicles 200f in the rear image based on sound from behind the vehicle or traffic or map information related to vehicle accidents.

[0489] On the other hand, the processor 175 can be controlled to execute the second mode in the lane keeping mode based on the detected emergency driving vehicle 200f.

[0490] That is, when executing the second mode of the lane keeping mode based on the approach of an emergency vehicle, the processor 175 can control the lane spacing to be a second spacing DPk smaller than the first spacing Dpa, or control the spacing between the center of the vehicle 200 and the first lane line LNa to be a DPmk smaller than DPm. Thus, the lane spacing can be adaptively adjusted based on an emergency vehicle 200f approaching from behind the vehicle.

[0491] On the other hand, the processor 175 can be controlled to execute the second mode in the lane keeping mode based on the emergency vehicle 200f behind the vehicle, and controlled to be that the smaller the distance to the emergency vehicle 200f, the smaller the interval with the first lane line LNa.

[0492] In particular, the processor 175 can control the spacing between itself and the first lane line LNa to decrease in stages as the distance to the emergency vehicle 200f becomes smaller. Thus, the lane spacing can be adaptively adjusted based on the approach of the emergency vehicle 200f from behind.

[0493] Figure 16c This is a graph illustrating lane spacing offsets that vary depending on vehicle speed.

[0494] Referring to the accompanying drawings, the processor 175 can control the vehicle 200 to align with the center line that serves as the center of the first lane line LNa and the second lane line LNb during the second speed driving mode.

[0495] On the other hand, the processor 175 can control the vehicle 200 to be closer to either the first lane line LNa or the second lane line LNb during the first speed when driving at a speed lower than the second speed, according to the lane keeping mode.

[0496] In particular, the processor 175 can be controlled according to the first mode in the lane keeping mode to align with the center line that is the center of the first lane line LNa and the second lane line LNb at a second speed.

[0497] On the other hand, when executing the second mode of lane keeping mode, the processor 175 can control the vehicle to reduce its speed so that it travels at a first speed lower than the second speed, while simultaneously bringing the center of the vehicle 200 closer to either the first lane line LNa or the second lane line LNb. Thus, it is possible to reflect driving conditions and adaptively adjust lane spacing during lane keeping mode operation.

[0498] On the other hand, the processor 175 can be controlled differently from the attached figure, such that when the second mode of the lane keeping mode is executed, the more the speed of the vehicle 200 increases, the smaller the second interval DPb, which is the interval between the first lane line LNa and the vehicle 200, or the closer it is to the first lane line LNa. Thus, the lane spacing can be adaptively adjusted based on the speed of the vehicle 200.

[0499] Figure 17a An example is given of adjusting the spacing with the first lane line LNa based on the approach of vehicles behind.

[0500] Referring to the attached diagram, the processor 175 can be controlled such that the closer the distance to the emergency vehicle 200f becomes, the smaller the interval with the first lane line LNa will become in stages.

[0501] The attached figure illustrates the case where the distance from the first lane line LNa decreases from Dsc to zero (0).

[0502] Alternatively, the processor 175 can be controlled such that the closer the distance to the emergency-moving vehicle 200f becomes, the greater the interval with the second lane line LNb will gradually increase.

[0503] In particular, an example is given where, even if there is a road boundary or guardrail RBm on the left side of vehicle 200, if an emergency-moving vehicle 200f approaches the rear right side of vehicle 200, the processor 175 will still increase the lane spacing with the second lane line LNb from DSb to Dsa. Thus, the lane spacing can be adaptively adjusted based on the emergency-moving vehicle 200f approaching from behind the vehicle.

[0504] Figure 17b This illustrates another example of adjusting the spacing with the first lane line LNa based on the approach of vehicles behind.

[0505] Referring to the attached diagram, the processor 175 can, based on the approach control of the emergency-driving vehicle 200f, progressively reduce the interval between itself and the first lane line LNa.

[0506] In particular, the accompanying drawings illustrate a scenario where vehicle 200 is located on the first lane line LNa.

[0507] On the other hand, if there is a road boundary or guardrail RBm on the left side of vehicle 200, processor 175 can take the road boundary or guardrail RBm into account to adjust the spacing with the road boundary or guardrail RBm.

[0508] That is, when the vehicle 200 is located on the first lane line LNa and the distance between it and the road boundary or guardrail RBm is DSe, the processor 175 can control the distance between it and the road boundary or guardrail RBm to increase to a greater than DSe, DSd.

[0509] That is, when the interval between the first lane line LNa and the left side of the vehicle 200 is DSf, the processor 175 can adjust the interval to zero (0). Thus, it is possible to reflect the driving situation and adaptively adjust the lane line interval during lane keeping mode operation.

[0510] Figure 17c This provides another example of adjusting the spacing based on the approach of vehicles behind.

[0511] Referring to the accompanying drawings, the processor 175 can adjust the spacing between the vehicle 200 traveling between the second lane line LNb and the third lane line LNc and the third lane line LNc based on the approach of the emergency vehicle 200f.

[0512] For example, if an emergency vehicle 200f approaches from behind, and a second vehicle OBm is on the right and a third vehicle 200b is on the left, the processor 175 can control the spacing between vehicle 200 and the third lane line LNc to be smaller.

[0513] The attached diagram illustrates a scenario where, with an emergency vehicle 200f approaching from behind, a second vehicle OBm is on the right, and a third vehicle 200b is on the left, the distance between vehicle 200 and the third lane line LNc is DSo, and the distance between vehicle 200 and the second lane line LNb is DSn.

[0514] In this case, preferably, DSn, which is the interval between vehicle 200 and the second lane line LNb, is larger than DSo, which is the interval between vehicle 200 and the third lane line LNc.

[0515] On the other hand, the processor 175 can adjust the spacing between vehicle 200 and the third lane line LNc based on the size of the vehicle on the right or the distance between them.

[0516] For example, if the size of a vehicle on the right becomes smaller or the distance between vehicles becomes larger, it can be controlled to reduce the distance between vehicle 200 and the third lane line LNc, or to increase the distance between vehicle 200 and the second lane line LNb.

[0517] The accompanying diagram illustrates a scenario where the vehicle on the right is the fourth vehicle 200c, which is smaller than the second vehicle OBm, and the distance between the fourth vehicle 200c and the third lane line LNc is greater than the distance between the second vehicle OBm and the third lane line LNc.

[0518] Thus, an example is given where, in the case where an emergency vehicle 200f approaches from behind, a fourth vehicle 200c is on the right, and a third vehicle 200b is on the left, the processor 175 controls the spacing between vehicle 200 and the third lane line LNc to be zero (0), while the spacing between vehicle 200 and the second lane line LNb is DSm, which is greater than DSn.

[0519] On the other hand, when the emergency vehicle 200f approaches from behind, the processor 175 can control the processor to increase the distance to the third vehicle 200b on the left to allow the emergency vehicle 200f to pass, and then decrease the distance to the third vehicle 200b after the emergency vehicle 200f has passed.

[0520] On the other hand, the processor 175 can also be controlled to temporarily position vehicle 200 on the third lane line LNc when increasing the distance with the third vehicle 200b on the left.

[0521] Figure 17d An example of adjusting the spacing is shown when there is a vehicle on the left.

[0522] Referring to the attached diagram, if the third vehicle 200b is located on the left, the processor 175 can control it to move closer to the second lane line LNb, which is the right lane line in the first lane line LNa and the second lane line LNb.

[0523] That is, the processor 175 can control the vehicle 200 to maintain a first interval DSs with the second lane line LNb, which is the right lane, according to the first mode in the lane keeping mode, and then, when the third vehicle 200b is on the left, to position the vehicle 200 on the second lane line LNb, which is the right lane, according to the second mode in the lane keeping mode.

[0524] That is, the processor 175 can control the lane keeping mode to make the right side of the vehicle 200 and the second lane line LNb be separated by DSR.

[0525] On the other hand, if vehicle 200 moves to the right based on the third vehicle 200b on the left, and a road boundary or guardrail RBm appears on the right, then processor 175 can control vehicle 200 to move to the left.

[0526] That is, if a road boundary or guardrail RBm appears on the right side while the vehicle 200 is traveling on the second lane line LNb, the processor 175 can control the lane spacing with the first lane line LNa to be DSp. Thus, the lane spacing can be adaptively adjusted based on the vehicle's side profile.

[0527] Figure 18 This is an example of an internal block diagram of a signal processing apparatus according to an embodiment of the present disclosure.

[0528] Referring to the accompanying drawings, the signal processing apparatus 170 of this embodiment can receive front images, internal images, or sensing signals or rear images from the front camera 195a, the internal camera 195i, the lidar 196, or the rear camera 195r, respectively.

[0529] On the other hand, the signal processing device 170 may include: a detection unit 1510 that detects objects or lanes based on received images or sensing signals; a motion estimation unit 1520 that estimates motion based on the detected objects or lanes; a determination unit 1530 that determines vehicle control based on the estimated motion; and an application operation unit 1540 that runs an application based on the determined vehicle control.

[0530] On the other hand, the detection unit 1510 may include: an object detection unit 1512, which detects objects in front of the vehicle from a frontal image or detects the driver's face or eyes from an interior image; a lane line detection unit 1514, which detects lane lines in front of the vehicle from a frontal image; and a sensor fusion unit 1516, which synthesizes image signals or sensing signals.

[0531] On the other hand, the motion estimation unit 1520 may include: a vehicle motion estimation unit 1522, which is used for ego motion estimation related to the travel direction estimation of the vehicle 200; a prediction path estimation unit 1524, which is used for prediction path estimation of the vehicle 200; an eye-tracking estimation unit 1526, which is used for eye-tracking of the driver; and an emotion information estimation unit 1528, which is used for calculating the driver's emotion information.

[0532] On the other hand, the determination unit 1530 may include: a condition determination unit 1532, which determines lane keeping mode conditions based on signals from the vehicle motion estimation unit 1522, the predicted path estimation unit 1524, the eye tracking estimation unit 1526, or the emotion information estimation unit 1528; a state machine determination unit 1534, which determines the lane keeping state machine; and a mode determination unit 1536, which determines the first mode and the second mode in the lane keeping mode.

[0533] On the other hand, the application operation unit 1540 may include an automatic steering control application 1544, which is used to run a notification application 1542 for lane keeping mode or a second mode in lane keeping mode based on signals from a condition determination unit 1532, a state machine determination unit 1534 or a mode determination unit 1536.

[0534] On the other hand, such as Figure 9 As shown, the processor 175 within the signal processing device 170 can run the driver monitoring application Ndm based on the internal image and the automatic steering control application 1544 based on the forward image.

[0535] At this point, the driver monitoring application Ndm can include multiple microservices.

[0536] For example, a driver monitoring application Ndm may include an object detection unit 1512 as a microservice, an eye tracking estimation unit 1526 for eye tracking of the driver, an emotion information estimation unit 1528, etc.

[0537] On the other hand, the processor 175 in the signal processing device 170 can run the object detection unit 1512, the eye tracking estimation unit 1526, the emotion information estimation unit 1528, etc., which are corresponding to the second security level ASILD.

[0538] Therefore, the processor 175 within the signal processing device 170 can be controlled to run the object detection unit 1512, the gaze tracking estimation unit 1526, and the emotion information estimation unit 1528, etc., in a virtual machine 850 corresponding to the second safety level, such as ASIL D. This enables the stable operation of the driver monitoring application Ndm.

[0539] On the other hand, the processor 175 within the signal processing device 170 can run the automatic steering control application 1544.

[0540] At this point, the automatic steering control application 1544 may include multiple microservices.

[0541] For example, the automatic steering control application 1544 may include an object detection unit 1512, a lane detection unit 1514, a vehicle motion estimation unit 1522, a predicted path estimation unit 1524, a gaze tracking estimation unit 1526, an emotion information estimation unit 1528, a condition determination unit 1532, a state machine determination unit 1534, or an automatic emergency steering mode determination unit 1536, etc., as microservices.

[0542] Therefore, the processor 175 within the signal processing device 170 can be controlled to run the object detection unit 1512, lane detection unit 1514, vehicle motion estimation unit 1522, predicted path estimation unit 1524, gaze tracking estimation unit 1526, emotion information estimation unit 1528, condition determination unit 1532, state machine determination unit 1534, or automatic emergency steering mode determination unit 1536, etc., in a virtual machine 850 corresponding to the second safety level, such as ASIL D. This enables the stable operation of the automatic steering control application 1544.

[0543] On the other hand, the processor 175 within the signal processing device 170 can run lane detection applications based on the image ahead.

[0544] At this point, lane detection applications can include multiple microservices.

[0545] That is, the processor 175 within the signal processing device 170 can run a lane detection application that includes multiple microservices based on the forward image.

[0546] For example, the processor 175 within the signal processing device 170 can run a lane detection application in a virtual machine 850 corresponding to a second security level such as ASIL D, or run multiple microservices for the lane detection application.

[0547] Next, the processor 175 within the signal processing device 170 can run the notification application 1542 for lane keeping mode based on the internal image and the forward image.

[0548] The notification application 1542 for lane keeping mode at this time can correspond to ASIL B or QM as the first safety level.

[0549] For this purpose, the processor 175 within the signal processing device 170 can be controlled to run the notification application 1542 in a virtual machine 830, such as ASIL B, which corresponds to the first security level.

[0550] On the other hand, the safety level of notification application 1542 can be lower than that of driver monitoring application Ndm, automatic steering control application 1544, or lane detection application.

[0551] On the other hand, such as Figure 9 As shown, processor 175 can run a plurality of virtual machines 810, 830, and 850 on hypervisor 505. A portion of these virtual machines 850 can run the driver monitoring application Ndm based on internal images and the automatic steering control application 1544 based on forward images. Thus, adaptive vehicle control can be performed according to the driver Owa's forward gaze level.

[0552] On the other hand, another part of the plurality of virtual machines 810, 830, 850, virtual machine 830 can run a notification application 1542 for lane keeping mode, the security level of the notification application 1542 can be lower than the security level of the driver monitoring application Ndm or the automatic steering control application 1544 or the lane line detection application.

[0553] On the other hand, a portion of the plurality of virtual machines 810, 830, and 850, virtual machine 850, can run a plurality of microservices for the automatic steering control application 1544 based on the forward image and a plurality of microservices for the driver monitoring application Ndm based on the internal image.

[0554] This enables the stable and efficient operation of the automatic steering control application 1544 and the driver monitoring application Ndm. Furthermore, it allows for the efficient execution of vehicle control using microservices.

[0555] The preferred embodiments of the present disclosure have been illustrated and described above. However, the present disclosure is not limited to the specific embodiments described above. Various modifications can be made by those skilled in the art without departing from the spirit of the present disclosure as claimed in the claims. Such modifications should not be understood separately from the technical concept or prospect of the present disclosure.

Claims

1. A signal processing apparatus, wherein, It has a processor that receives and processes images from a camera installed inside the vehicle; The processor performs lane line detection based on the forward image from the camera, and performs lane keeping mode based on the lane line detection; The processor controls the steering drive to maintain a first distance from the first lane line in the adjacent first lane line and second lane line according to the first mode in the lane keeping mode. The processor controls the steering drive to maintain a second interval with the first lane line, different from the first interval, according to the second mode in the lane keeping mode.

2. The signal processing apparatus according to claim 1, wherein, The processor is controlled to detect emergency vehicles in the rear image based on the rear image from the camera, and to execute the second mode in the lane keeping mode based on the emergency vehicles.

3. The signal processing apparatus according to claim 1, wherein, The processor controls itself to execute the second mode of the lane keeping mode based on the emergency vehicle behind it, and controls itself to become closer to the emergency vehicle and smaller in distance from the first lane line.

4. The signal processing apparatus according to claim 1, wherein, If, based on the front or side image from the camera, a second vehicle above the road boundary, guardrail, or reference size is located to the side of the vehicle, the processor controls the execution of the second mode in the lane-keeping mode.

5. The signal processing apparatus according to claim 1, wherein, The processor controls the detection of a second vehicle on the side of the vehicle based on the front or side image from the camera, and adjusts the second interval in the second mode of the lane keeping mode based on the size of the second vehicle.

6. The signal processing apparatus according to claim 1, wherein, The processor controls the second interval to change based on the vehicle's speed in the second mode of the lane keeping mode.

7. The signal processing apparatus according to claim 1, wherein, The processor controls the detection of pedestrians on the side of the vehicle based on the front or side image from the camera, and adjusts the second interval based on the number or position of the detected pedestrians.

8. The signal processing apparatus according to claim 1, wherein, The processor adjusts the second interval in the second mode of the lane keeping mode based on the driver's driving skill, the number of occupants in the vehicle, or whether the front passenger is seated.

9. The signal processing apparatus according to claim 1, wherein, The processor detects the driver's gaze based on internal images from an internal camera and adjusts the second interval in the second mode of the lane-keeping mode based on the direction of the driver's gaze.

10. The signal processing apparatus according to claim 1, wherein, The processor is controlled to detect the driver's gaze based on internal images from an internal camera in manual driving mode, detect the distance between the driver's gaze and the first lane line in the forward image, learn based on the driver's gaze and the distance between the driver's gaze and the first lane line, and store the learning results in memory.

11. The signal processing apparatus according to claim 10, wherein, When the processor executes the second mode in the lane keeping mode, it sets the second interval based on the learning results.

12. The signal processing apparatus according to claim 1, wherein, The processor is controlled to detect the driver's emotional information based on internal images from an internal camera in manual driving mode, detect the distance between the driver and the first lane line in the forward image, learn based on the driver's emotional information and the distance between the driver and the first lane line, and store the learning results in memory.

13. The signal processing apparatus according to claim 12, wherein, When the processor executes the second mode in the lane keeping mode, it sets the second interval based on the learning results.

14. The signal processing apparatus according to claim 1, wherein, The processor is controlled to detect the distance between the vehicle and the first lane line in the forward image in manual driving mode, and store the vehicle's speed and the distance information between the vehicle and the first lane line in memory. When the processor executes the second mode in the lane keeping mode, it sets the second interval based on the vehicle's speed and the interval information with respect to the first lane line.

15. The signal processing apparatus according to claim 1, wherein, The processor runs multiple virtual machines on the hypervisor. A subset of the plurality of virtual machines runs a lane detection application based on the foreground image, and runs a plurality of microservices for the lane detection application.

16. The signal processing apparatus according to claim 15, wherein, Another portion of the plurality of virtual machines runs a notification application for the lane keeping mode; The security level of the notification application is lower than that of the lane detection application.

17. A signal processing apparatus, wherein, It has a processor that receives and processes images from a camera installed inside the vehicle; The processor performs lane line detection based on the forward image from the camera, and performs lane keeping mode based on the lane line detection; The processor controls the steering drive to remain centered on the adjacent first lane line and second lane line according to the first mode in the lane keeping mode. The processor controls the steering drive to move closer to either the first lane line or the second lane line according to the second mode in the lane keeping mode.

18. The signal processing apparatus according to claim 17, wherein, The processor controls the steering drive to maintain a first interval with the first lane line according to the first mode in the lane keeping mode. The processor controls the steering drive unit to maintain a second interval with the first lane line, different from the first interval, according to the second mode in the lane keeping mode.

19. A vehicle control device, wherein, A signal processing apparatus having any one of claims 1 to 18.

Images