Signal processing device, and vehicle display device comprising same

Abstract

A signal processing device and a vehicle display device comprising same, according to an embodiment of the present disclosure, comprise a processor for executing a hypervisor, wherein the processor: executes, on the hypervisor, a recognition service for performing recognition on the basis of sensor data and a safety monitor service for monitoring the recognition service; and performs a recovery mode by using a redundant recognition service when the recognition service is executed. Accordingly, the stability of the recognition service can be improved.

Description

Signal processing device and vehicle display device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle display device having the same, and more particularly, to a signal processing device capable of improving the stability of a recognition service and a vehicle display device having the same.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, for the convenience of vehicle users, a vehicle signal processing device is installed inside the vehicle.

[0004] The signal processing device inside the vehicle receives and processes sensor data from various sensor devices inside the vehicle.

[0005] Meanwhile, as the types and number of sensors installed in vehicles increase due to advanced driver assistance systems (ADAS) and autonomous driving, the amount of data that needs to be processed is also increasing.

[0006] Meanwhile, in relation to vehicle driver assistance systems (ADAS) or autonomous driving, there is a possibility of malfunction due to defects or failures when running applications, etc., and the possibility of vehicle accidents occurring due to such malfunctions increases.

[0007] The problem to be solved by the present disclosure is to provide a signal processing device capable of improving the stability of a recognition service and a vehicle display device equipped with the same.

[0008] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of improving the usability of a sensor data-based recognition service and a vehicle display device equipped with the same.

[0009] Another problem that the present disclosure seeks to solve is to provide a signal processing device capable of efficiently performing data processing using microservices and a vehicle display device equipped with the same.

[0010] A signal processing device and a vehicle display device having the same according to one embodiment of the present disclosure for solving the above technical problem include a processor executing a hypervisor, and the processor executes, on the hypervisor, a recognition service that performs recognition based on sensor data and a safety monitor service that monitors the recognition service, and, when executing the recognition service, performs a recovery mode using a redundant recognition service.

[0011] Meanwhile, the safety monitor service controls the operation of a redundant recognition service during the operation of the recognition service, and when a recognition service failure occurs, it can send a recovery command to the recognition service and an activity command to the redundant recognition service.

[0012] Meanwhile, the safety monitor service can control the operation of the redundant recognition service to be temporarily suspended during the operation of the recognition service, and, when a recognition service failure occurs, can send a recovery command to the recognition service and a start command to the redundant recognition service.

[0013] Meanwhile, the safety monitor service can control the switching of operations between the recognition service or the redundant recognition service based on monitoring data or analysis data from the recognition service or the redundant recognition service.

[0014] Meanwhile, the safety monitor service may vary the operation switching criteria between the recognition service or the redundant recognition service based on monitoring data or analysis data from the recognition service or the redundant recognition service.

[0015] Meanwhile, the safety monitor service can control updating of a monitoring module or management module within the recognition service or redundant recognition service.

[0016] Meanwhile, the processor can vary the mode for planning or controlling the vehicle based on the confidence level.

[0017] Meanwhile, the processor can control whether normal mode, degradation mode, emergency mode, or manual driving mode is performed based on the confidence level.

[0018] Meanwhile, the processor can perform recognition services for autonomous driving, vehicle assisted driving, or driver monitoring based on sensor data.

[0019] Meanwhile, the processor can perform recognition services for active steering, lane keeping, emergency braking, automatic lane change, or traffic jam pilot based on sensor data.

[0020] Meanwhile, the processor can execute localization based on vehicle driving status or weather information and sensor data, execute a recognition service based on the localization, execute planning based on the recognition service, and output a vehicle control signal based on the planning.

[0021] Meanwhile, the recognition service can be any one of multiple microservices for running the application.

[0022] Meanwhile, the recognition service can correspond to a microservice for executing object detection or object tracking or lane detection or lane maintenance.

[0023] Meanwhile, the processor can control switching from normal mode to degradation mode based on the recognition service, or performing emergency mode or manual driving mode according to a switching request while performing degradation mode.

[0024] Meanwhile, the processor can control the vehicle to stop after performing the emergency mode based on a transition request while performing the degradation mode based on the recognition service.

[0025] Meanwhile, the processor can perform object detection or object tracking based on the recognition service.

[0026] Meanwhile, the processor can be controlled to perform a fail-safe mode, a degradation mode, or a fail-operation mode based on the recognition service.

[0027] Meanwhile, the processor may perform a fail-safe mode if a monitoring signal is not received while performing autonomous driving mode, perform a degradation mode if the fail-safe mode fails, and perform an alternative operation or control the vehicle to stop after the degradation mode.

[0028] Meanwhile, the processor can reset the component in fail-safe mode.

[0029] Meanwhile, the processor can control the vehicle to reduce speed or perform steering adjustments in degradation mode.

[0030] A signal processing device and a vehicle display device including the same according to one embodiment of the present disclosure include a processor executing a hypervisor, wherein the processor executes, on the hypervisor, a recognition service that performs recognition based on sensor data and a safety monitor service that monitors the recognition service, and, when the recognition service is executed, performs a recovery mode using a redundant recognition service. Accordingly, the stability of the sensor data-based recognition service can be improved. Furthermore, the usability of the sensor data-based recognition service can be improved.

[0031] Meanwhile, the safety monitor service controls the operation of a redundant recognition service during recognition service operation. In the event of a recognition service failure, it can send recovery commands to the recognition service and action commands to the redundant recognition service. This improves the stability of sensor data-based recognition services.

[0032] Meanwhile, the safety monitor service can control the suspension of the redundant recognition service during recognition service operation. In the event of a recognition service failure, it can send a recovery command to the recognition service and a start command to the redundant recognition service. This improves the stability of sensor data-based recognition services.

[0033] Meanwhile, the safety monitoring service can control the transition between the recognition service or the redundant recognition service based on monitoring or analysis data from the recognition service or the redundant recognition service. This improves the stability of the sensor data-based recognition service.

[0034] Meanwhile, the safety monitoring service can vary the operational transition criteria between the recognition service or redundant recognition service based on monitoring or analysis data from the recognition service or redundant recognition service. This can improve the stability of sensor data-based recognition services.

[0035] Meanwhile, the safety monitoring service can control updates to the monitoring or management modules within the recognition service or redundant recognition service. This improves the stability of sensor data-based recognition services.

[0036] Meanwhile, the processor can vary the planning or control mode for vehicle control based on the confidence level. Accordingly, the reliability of sensor data-based recognition services can be improved based on the confidence level.

[0037] Meanwhile, the processor can control the execution of normal mode, degradation mode, emergency mode, or manual driving mode based on the confidence level. Accordingly, the stability of sensor data-based recognition services can be improved based on the confidence level.

[0038] Meanwhile, the processor can perform recognition services for autonomous driving, vehicle assisted driving, or driver monitoring based on sensor data. This improves the usability of recognition services.

[0039] Meanwhile, the processor can perform recognition services for active steering, lane keeping, emergency braking, automatic lane change, or traffic jam pilot based on sensor data. This improves the usability of recognition services.

[0040] Meanwhile, the processor can perform localization based on vehicle driving conditions, weather information, and sensor data, execute a recognition service based on the localization, perform planning based on the recognition service, and output a vehicle control signal based on the planning. Accordingly, the usability of the recognition service can be improved in response to various situations.

[0041] Meanwhile, the recognition service can be any one of multiple microservices for application execution. This allows for efficient performance of the recognition service.

[0042] Meanwhile, the recognition service can respond to microservices for object detection, object tracking, lane detection, or lane maintenance. This allows for efficient performance of the recognition service.

[0043] Meanwhile, the processor can control switching from normal mode to degradation mode based on the recognition service, or switching to emergency mode or manual driving mode upon a switching request while performing degradation mode. This can improve the usability of the recognition service.

[0044] Meanwhile, the processor can control the vehicle to stop after executing the emergency mode based on a transition request while performing degradation mode based on the recognition service. This improves the usability of the recognition service.

[0045] Meanwhile, the processor can perform object detection or object tracking based on the recognition service. This can improve the usability of the recognition service.

[0046] Meanwhile, the processor can be controlled to perform fail-safe mode, degradation mode, or fail-operation mode based on the recognition service. This can improve the usability of the recognition service.

[0047] Meanwhile, if a monitoring signal is not received while the autonomous driving mode is in effect, the processor can execute fail-safe mode. If fail-safe mode fails, the processor can execute degradation mode. After degradation mode, the processor can perform alternative operations or control the vehicle to stop. This can improve the usability of the recognition service.

[0048] Meanwhile, the processor can reset components in fail-safe mode, thereby improving the usability of the recognition service.

[0049] Meanwhile, the processor can control vehicle speed reduction or steering adjustment in degradation mode, thereby improving the usability of recognition services.

[0050] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0051] Figure 2 is a diagram illustrating various architectures of a vehicle communication gateway.

[0052] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0053] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0054] Fig. 4 is an example of an internal block diagram of the vehicle display device of Fig. 3b.

[0055] FIGS. 5A to 5D are drawings showing various examples of vehicle display devices.

[0056] FIG. 6 is an example of a block diagram of a vehicle display device according to an embodiment of the present disclosure.

[0057] FIGS. 7A and 7B are drawings for reference in the description of a signal processing device related to the present disclosure.

[0058] FIGS. 8A to 8E are diagrams illustrating various examples of execution of microservices according to embodiments of the present disclosure.

[0059] FIG. 9 is an example of an internal block diagram of a signal processing device according to an embodiment of the present disclosure.

[0060] FIG. 10 is an example of an internal block diagram of a signal processing device according to another embodiment of the present disclosure.

[0061] FIG. 11 is a diagram illustrating the operation of a signal processing device according to an embodiment of the present disclosure.

[0062] Figures 12a to 19b are drawings referenced in the operation description of Figure 11.

[0063] FIG. 20 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0064] Figure 21 is a drawing referenced in the description of Figure 20.

[0065] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0066] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0067] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0068] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0069] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0070] Meanwhile, the vehicle (200) may be equipped with multiple displays (180a, 180b) for displaying images, information, etc. inside.

[0071] In Fig. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) is also possible.

[0072] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named a center information display.

[0073] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0074] Figure 2 is a diagram illustrating various architectures of a vehicle communication gateway.

[0075] First, Fig. 2 is a drawing illustrating the first architecture of a vehicle communication gateway.

[0076] Referring to the drawing, the first architecture (300a) can correspond to a zone-based architecture.

[0077] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the plurality of zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the plurality of zones (Z1 to Z4).

[0078] Meanwhile, the signal processing device (170a) may further include, in addition to the vehicle communication gateway (GWDa), an autonomous driving control module (ACC), a cockpit control module (CPG), etc.

[0079] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be an HPC (High Performance Computing) gateway.

[0080] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) within a plurality of zones (Z1 to Z4).

[0081] Figure 3a is a drawing showing an example of the arrangement of a vehicle display device inside a vehicle.

[0082] Referring to the drawing, the interior of the vehicle may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a room mirror display (not shown), etc.

[0083] Figure 3b is a drawing showing another example of the arrangement of a vehicle display device inside a vehicle.

[0084] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0085] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0086] The signal processing device (170) has a processor (175) therein and can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0087] A second virtual machine (not shown) can operate for the first display (180a), and a third virtual machine (not shown) can operate for the second display (180b).

[0088] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0089] Meanwhile, the first virtual machine (not shown) within the processor (175) shares at least a portion of data with the second virtual machine (not shown) and the third virtual machine (not shown) for data sharing processing. Accordingly, data can be shared and processed among multiple virtual machines for multiple displays within the vehicle.

[0090] Meanwhile, a first virtual machine (not shown) within a processor (175) may receive and process vehicle wheel speed sensor data, and transmit the processed wheel speed sensor data to at least one of a second virtual machine (not shown) or a third virtual machine (not shown). Accordingly, the vehicle wheel speed sensor data may be shared with at least one virtual machine.

[0091] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information, or images.

[0092] The signal processing device (170) can control the RSE display (180c) by executing a fourth virtual machine (not shown) in addition to the first virtual machine to the third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0093] Accordingly, it is possible to control various displays (180a to 180c) using one signal processing device (170).

[0094] Meanwhile, some of the multiple displays (180a~180c) may operate under Linux OS, while others may operate under Web OS.

[0095] The signal processing device (170) according to the embodiment of the present disclosure can control the same information or the same image to be displayed in synchronization on displays (180a to 180c) operating under various operating systems (OS).

[0096] Meanwhile, in FIG. 3b, a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications and a vehicle speed indicator (212b) and a vehicle interior temperature indicator (213b) are displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) are displayed on a third display (180c).

[0097] Fig. 4 is an example of an internal block diagram of the vehicle display device of Fig. 3b.

[0098] Referring to the drawings, a vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa to EMd) for internal communication, a memory (140), a signal processing unit (170), a plurality of displays (180a to 180c), an audio output unit (185), and a power supply unit (190).

[0099] A plurality of communication modules (EMa to EMd) can be arranged, for example, in a plurality of zones (Z1 to Z4) of FIG. 2, respectively.

[0100] Meanwhile, the signal processing device (170) may have a communication switch (736b) for data communication with each communication module (EM1 to EM4) inside.

[0101] Each communication module (EM1 to EM4) can perform data communication with multiple sensor devices (SN) or ECUs (770) or area signal processing devices (170Z).

[0102] Meanwhile, the plurality of sensor devices (SN) may include a camera (195), a lidar (196), a radar (197), or a position sensor (198).

[0103] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0104] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0105] The communication unit (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0106] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0107] The communication unit (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (900). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0108] A plurality of communication modules (EM1 to EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or an area signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0109] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0110] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0111] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0112] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can transmit location information data sensed by a GPS module or location sensor (198) to a signal processing device (170).

[0113] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0114] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0115] For example, the memory (140) may store data regarding a hypervisor, a first virtual machine, a third virtual machine, or the like, for execution within the processor (175).

[0116] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0117] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0118] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100).

[0119] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0120] The processor (175) can execute a first virtual machine to a third virtual machine (not shown) on a hypervisor (not shown) within the processor (175).

[0121] Among the first virtual machine to the third virtual machine (not shown), the first virtual machine (not shown) may be named a server virtual machine (Server Virtual Maschine), and the second virtual machine to the third virtual machine (not shown) may be named a guest virtual machine (Guest Virtual Maschine).

[0122] For example, a first virtual machine (not shown) within a processor (175) may receive, process, or output sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data.

[0123] In this way, by performing most of the data processing in the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0124] As another example, a first virtual machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for a second virtual machine or a third virtual machine (not shown).

[0125] And, the first virtual machine (not shown) can transmit processed data to the second virtual machine or the third virtual machine (not shown).

[0126] Accordingly, among the first virtual machine to the third virtual machine (not shown), only the first virtual machine (not shown) receives sensor data, communication data, or external input data from multiple sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtual machines, enabling 1:N data communication, and enabling synchronization when sharing data.

[0127] Meanwhile, the first virtual machine (not shown) can control the second virtual machine (not shown) and the third virtual machine (not shown) to share the same data by writing data to the shared memory (508).

[0128] For example, a first virtual machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in shared memory (508) and control the same data to be shared with a second virtual machine (not shown) and a third virtual machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0129] Ultimately, by performing most of the data processing on the first virtual machine (not shown), data sharing in a 1:N manner becomes possible.

[0130] Meanwhile, the first virtual machine (not shown) within the processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the second virtual machine (not shown) and the third virtual machine (not shown).

[0131] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0132] Meanwhile, the signal processing device (170) in the display device (100) of FIG. 4 may be the same as the signal processing device (170, 170a1, 170a2) of the vehicle display device of FIG. 5a or lower.

[0133] FIGS. 5A to 5D are drawings showing various examples of vehicle display devices.

[0134] FIG. 5A illustrates an example of a vehicle display device according to an embodiment of the present disclosure.

[0135] Referring to the drawings, a vehicle display device (800a) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2) and a plurality of area signal processing devices (170Z1 to 170Z4).

[0136] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0137] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0138] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0139] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0140] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0141] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0142] For example, sensor data within a vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle exterior radar data, and vehicle exterior lidar data.

[0143] Meanwhile, camera data may include vehicle exterior camera data and vehicle interior camera data.

[0144] Meanwhile, the signal processing device (170a1, 170a2) can execute multiple virtual machines (820, 830, 840) based on safety standards.

[0145] In the drawing, it is illustrated that a processor (175) within a signal processing device (170a) executes a hypervisor (505) and, on the hypervisor (505), executes first to third virtual machines (820 to 840) according to an automotive safety integrity level (Automotive SIL; ASIL).

[0146] The first virtual machine (820) may be a virtual machine corresponding to Quality Management (QM), which is the lowest safety level in the Automotive Safety Integrity Level (ASIL) and is a non-enforceable grade.

[0147] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0148] The second virtual machine (820) may be a virtual machine corresponding to ASIL A or ASIL B, where the sum of severity, exposure, and controllability is 7 or 8 in the automotive safety integrity level (ASIL).

[0149] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0150] The third virtual machine (840) may be a virtual machine corresponding to ASIL C or ASIL D, in which the sum of severity, exposure, and controllability is 9 or 10 in the automotive safety integrity level (ASIL).

[0151] Meanwhile, ASIL D can correspond to the grade that requires the highest safety level.

[0152] The third virtual machine (840) can run a safety operating system (842) and an application (845) on the operating system (842).

[0153] Meanwhile, the third virtual machine (840) may also execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0154] Meanwhile, unlike the drawing, the third virtual machine (840) can also be executed through a separate core rather than the processor (175). This will be described later with reference to FIG. 5b.

[0155] FIG. 5b illustrates another example of a vehicle display device according to an embodiment of the present disclosure.

[0156] Referring to the drawings, a vehicle display device (800b) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2) and a plurality of area signal processing devices (170Z1 to 170Z4).

[0157] The vehicle display device (800b) of FIG. 5b is similar to the vehicle display device (800a) of FIG. 5a, but the signal processing device (170a1) has some differences from the signal processing device (170a1) of FIG. 5a.

[0158] To describe the difference, the signal processing device (170a1) may include a processor (175) and a second processor (177).

[0159] The processor (175) within the signal processing unit (170a1) executes a hypervisor (505), and executes first and second virtual machines (820 to 830) on the hypervisor (505) according to the automotive safety integrity level (Automotive SIL; ASIL).

[0160] The first virtual machine (820) can execute an operating system (822), a container runtime (824) on the operating system (822), and containers (827, 829) on the container runtime (824).

[0161] The second virtual machine (820) can execute an operating system (832), a container runtime (834) on the operating system (832), and containers (837, 839) on the container runtime (834).

[0162] Meanwhile, the second processor (177) within the signal processing device (170a1) can execute a third virtual machine (840).

[0163] The third virtual machine (840) can execute a safety operating system (842), an auto-execution (845) on the operating system (842), and an application (845) on the auto-execution (845). That is, unlike FIG. 5A, an auto-execution (846) on the operating system (842) can be executed.

[0164] Meanwhile, the third virtual machine (840) may, similarly to FIG. 5a, execute a safety operating system (842), a container runtime (844) on the safety operating system (842), and a container (847) on the container runtime (844).

[0165] Meanwhile, the third virtual machine (840) requiring a high level of security is preferably executed on a second processor (177), which is a different core or different processor, unlike the first and second virtual machines (820 to 830).

[0166] Meanwhile, in the signal processing devices (170a1, 170a2) of FIGS. 5a and 5b, when the first signal processing device (170a) malfunctions, the second signal processing device (170a2), which is a backup device, can operate.

[0167] Alternatively, it is also possible for the signal processing devices (170a1, 170a2) to operate simultaneously, with the first signal processing device (170a) operating as the main device and the second signal processing device (170a2) operating as the sub device. This will be described with reference to FIGS. 5c and 5d.

[0168] FIG. 5c illustrates another example of a vehicle display device according to an embodiment of the present disclosure.

[0169] Referring to the drawings, a vehicle display device (800c) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2) and a plurality of area signal processing devices (170Z1 to 170Z4).

[0170] Meanwhile, in the drawing, two signal processing devices (170a1, 170a2) are exemplified, but this is for backup purposes, etc., and one is also possible.

[0171] Meanwhile, the signal processing device (170a1, 170a2) may also be named an HPC (High Performance Computing) signal processing device.

[0172] Multiple area signal processing devices (170Z1 to 170Z4) are arranged in each area (Z1 to Z4) and can transmit sensor data to signal processing devices (170a1, 170a2).

[0173] The signal processing device (170a1, 170a2) receives data via a wire from multiple area signal processing devices (170Z1 to 170Z4) or a communication device (120).

[0174] In the drawing, data is exchanged based on wired communication between a signal processing device (170a1, 170a2) and multiple area signal processing devices (170Z1 to 170Z4), and the signal processing device (170a1, 170a2) and the server (400) exchange data based on wireless communication. However, data may be exchanged based on wireless communication between a communication device (120) and a server (400), and the signal processing device (170a1, 170a2) and the communication device (120) may exchange data based on wired communication.

[0175] Meanwhile, data received by the signal processing device (170a1, 170a2) may include camera data or sensor data.

[0176] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175) in the first signal processing device (170a1) executes a hypervisor (505) and can execute a safety virtualization machine (860) and a non-safety virtualization machine (870) on the hypervisor (505), respectively.

[0177] Meanwhile, among the signal processing devices (170a1, 170a2), the processor (175b) in the second signal processing device (170a2) executes the hypervisor (505b) and can execute only the safety virtualization machine (880) on the hypervisor (505).

[0178] In this way, since the processing for safety is separated between the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0179] Meanwhile, high-speed network communication can be performed between the first signal processing device (170a1) and the second signal processing device (170a2).

[0180] FIG. 5d illustrates another example of a vehicle display device according to an embodiment of the present disclosure.

[0181] Referring to the drawings, a vehicle display device (800d) according to an embodiment of the present disclosure includes a signal processing device (170a1, 170a2) and a plurality of area signal processing devices (170Z1 to 170Z4).

[0182] The vehicle display device (800d) of FIG. 5d is similar to the vehicle display device (800c) of FIG. 5c, but the second signal processing device (170a2) has some differences from the second signal processing device (170a2) of FIG. 5c.

[0183] The processor (175b) in the second signal processing device (170a2) of FIG. 5d executes a hypervisor (505b) and can execute a safety virtualization machine (880) and a non-safety virtualization machine (890) on the hypervisor (505).

[0184] That is, unlike FIG. 5c, the difference is that the processor (175b) within the second signal processing device (170a2) further executes a non-safety virtualization machine (890).

[0185] In this way, since the processing for safety and non-safety is separated into the first signal processing device (170a1) and the second signal processing device (170a2), it is possible to improve stability and processing speed.

[0186] FIG. 6 is an example of a block diagram of a vehicle display device according to an embodiment of the present disclosure.

[0187] Referring to the drawings, a vehicle display device (900) according to an embodiment of the present disclosure includes a signal processing device (170) and at least one display.

[0188] In the drawing, at least one display is illustrated, a cluster display (180a) and an AVN display (180b).

[0189] Meanwhile, the vehicle display device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0190] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having multiple CPUs (175), GPUs (178), NPUs (179), etc., and may be called an HPC (High Performance Computing) signal processing device or a central signal processing device.

[0191] A plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) are connected by wired cables (CB1 to CB4).

[0192] Meanwhile, multiple area signal processing devices (170Z1 to 170Z4) can be connected to each other with wired cables (CBa to CBd).

[0193] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0194] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may be equipped with at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0195] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0196] Meanwhile, sensor data may be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data may be stored in a storage device (925) within the signal processing device (170).

[0197] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0198] In the drawing, it is exemplified that camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2), a third area signal processing device (170Z3), etc.

[0199] Meanwhile, as shown in the drawing, radar data from a radar sensor (197) and an actuator from an actuator (193) can be input to a signal processing device (170) via a first region signal processing device (170Z1), etc.

[0200] Meanwhile, since the data read speed or write speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the plurality of area signal processing devices (170Z1 to 170Z4) to the signal processing device (170), it is preferable that multi-path routing be performed so that a network bottleneck does not occur.

[0201] To this end, the signal processing device (170) according to the embodiment of the present disclosure can perform multi-path routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data from the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, the network configuration can be dynamically changed to transmit data.

[0202] Data communication between a plurality of area signal processing devices (170Z1 to 170Z4) and a signal processing device (170) in a vehicle display device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication for high-bandwidth, low-latency communication.

[0203] FIGS. 7A and 7B are drawings for reference in the description of a signal processing device related to the present disclosure.

[0204] Figure 7a illustrates an application based on camera data, etc., running on a signal processing device.

[0205] Referring to the drawing, a signal processing device (170x) related to the present disclosure can execute a Driver Monitoring Systems (DMS) application (785) based on camera data from an in-vehicle camera (195i), sensor data from a pressure sensor (SNp), and sensor data from a gas sensor (SNc), and can control a warning sound to be output to an audio output unit (185) based on the result data.

[0206] Figure 7b is a drawing referenced in the operation description of Figure 7a.

[0207] Referring to the drawing, a signal processing device (170x) related to the present disclosure includes a processor (175x), and the processor (175x) can execute a hypervisor (505).

[0208] Meanwhile, a processor (175x) related to the present disclosure executes a plurality of virtual machines (520x, 530x, 540x) on a hypervisor (505), and a second virtual machine (530x) among the plurality of virtual machines (520x, 530x, 540x) executes a driver monitoring system (DMS) application (785) based on camera data from an internal vehicle camera (195i), sensor data from a pressure sensor (SNp), and sensor data from a gas sensor (SNc), and executes a lane keeping assist system (LKAS) application (787) based on camera data outside the vehicle.

[0209] Meanwhile, among the multiple virtual machines (520x, 530x, 540x), a third virtual machine (540x) can execute a forward collision warning (FCW) application (789) based on vehicle external camera data.

[0210] Meanwhile, as shown in FIGS. 7a and 7b, when a driver monitoring system (DMS) application (785) and a lane keeping assist system (787) are executed within a second virtual machine (530x), there is a problem that the workload of the second virtual machine (530x) is significant.

[0211] In particular, in order to execute the driver monitoring system (DMS) application (785), camera data from an in-vehicle camera (195i), sensor data from a pressure sensor (SNp), and sensor data from a gas sensor (SNc) must be received and processed, so there is a problem that the workload of the second virtual machine (530x) is considerable.

[0212] Meanwhile, the third virtual machine (540x) runs on a separate virtual machine from the lane keeping assist system (787), which is based on vehicle external camera data when executing the forward collision warning (FCW) application (789), so there is a problem that the workload is performed inefficiently.

[0213] Accordingly, in this disclosure, a method is proposed for sharing intermediate result data of an application, etc., when executing a similar application.

[0214] To this end, the signal processing device (170) according to the embodiment of the present disclosure divides the application into a plurality of micro services, and executes other micro services based on the results of the micro services, etc., thereby efficiently distributing the workload.

[0215] For example, a signal processing device (170) according to an embodiment of the present disclosure may control a plurality of micro services, wherein a first micro service is performed in a first virtual machine and a second micro service is performed in a second virtual machine, and the result of the first micro service is shared using a shared memory (508) or the like, so that the second micro service is executed based on the result data of the first virtual machine. Accordingly, data processing can be performed efficiently.

[0216] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute multiple virtual machines by distinguishing them according to the safety level.

[0217] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute applications or microservices by distinguishing them according to the safety level.

[0218] Accordingly, data processing can be performed stably according to the Automotive Safety Integrity Level (ASIL) in relation to advanced driver assistance systems (ADAS) or autonomous driving.

[0219] FIGS. 8A to 8E are diagrams illustrating various examples of execution of microservices according to embodiments of the present disclosure.

[0220] Figure 8a illustrates that multiple microservices corresponding to ASIL B are executed based on camera data from an internal camera (195i).

[0221] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure can execute a driver monitoring system (DMS) application (905) corresponding to ASIL B.

[0222] For example, a signal processing device (170) according to an embodiment of the present disclosure can execute a driver monitoring system (DMS) application (905) by separating it into multiple microservices.

[0223] The drawing illustrates multiple microservices for a driver monitoring system (DMS) application (905), including a face detection microservice (910b), an eye movement microservice (915b), an eye tracking microservice (920b), and an alert microservice (930b).

[0224] That is, the signal processing device (170) according to the embodiment of the present disclosure can execute a face detection microservice (910b), an eye movement microservice (915b), a gaze tracking microservice (920b), and a warning microservice (930b) as a plurality of microservices for a driver monitoring system (DMS) application (905) corresponding to ASIL B.

[0225] Meanwhile, the face detection microservice (910b) is executed based on camera data from the internal camera (195i), and the result data of the face detection microservice (910b) is transmitted to the eye movement microservice (915b).

[0226] Meanwhile, the eye movement microservice (915b) is executed based on the result data of the face detection microservice (910b), and the result data of the eye movement microservice (915b) is transmitted to the gaze tracking microservice (920b).

[0227] Meanwhile, the gaze tracking microservice (920b) is executed based on the result data of the eye movement microservice (915b), and the result data of the gaze tracking microservice (920b) is transmitted to the alert microservice (930b).

[0228] Meanwhile, the warning microservice (930b) is executed based on the result data of the gaze tracking microservice (920b), and the result data is input to the audio output unit (185), so that a warning sound can be output from the audio output unit (185).

[0229] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a second face detection microservice (910c) and a head movement microservice (915c) for a driver monitoring system (DMS) application (905) corresponding to ASIL B.

[0230] The second face detection microservice (910c) is executed based on camera data from the internal camera (195i), and the result data of the second face detection microservice (910c) is transmitted to the head movement microservice (915c).

[0231] Meanwhile, the head movement microservice (915c) is executed based on the result data of the second face detection microservice (910c), and the result data of the head movement microservice (915c) is transmitted to the gaze tracking microservice (920b).

[0232] Meanwhile, the gaze tracking microservice (920b) is executed based on the result data of the head movement microservice (915c) and the result data of the eye movement microservice (915b), and the result data of the gaze tracking microservice (920b) can be transmitted to the warning microservice (930b).

[0233] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a plurality of micro-services that are not related to ASIL B, for example, corresponding to QM (Quality Management).

[0234] In the drawing, a third face detection microservice (910a), a face recognition microservice (915a), and a personal microservice (920a) can each be executed as multiple microservices, not related to ASIL B.

[0235] Figure 8b illustrates an example of execution of multiple microservices corresponding to ASIL B and microservices corresponding to QM based on camera data from an internal camera (195i).

[0236] Referring to the drawing, the signal processing device (170) according to the embodiment of the present disclosure can execute a driver monitoring system (DMS) application (905) corresponding to ASIL B, similar to FIG. 8A.

[0237] For example, a signal processing device (170) according to an embodiment of the present disclosure can execute a face detection microservice (910b), an eye movement microservice (915b), a gaze tracking microservice (920b), and a warning microservice (930b) as a plurality of microservices for a driver monitoring system (DMS) application (905) corresponding to ASIL B.

[0238] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a second face detection microservice (910c) and a head movement microservice (915c) for a driver monitoring system (DMS) application (905) corresponding to ASIL B.

[0239] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute a third face detection microservice (910a), a face recognition microservice (915a), and a personal microservice (920a) as a plurality of microservices, which are not related to ASIL B.

[0240] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute an augmented reality microservice (930c), which is an example of a graphic provision microservice, as a microservice corresponding to QM.

[0241] At this time, the signal processing device (170) according to the embodiment of the present disclosure can transmit the result data of the eye tracking microservice (920b) among the microservices in the application (905) corresponding to ASIL B to the augmented reality microservice (930c) corresponding to QM, which has a lower safety level.

[0242] Accordingly, the augmented reality microservice (930c) corresponding to QM is executed based on the result data of the gaze tracking microservice (920b), and the result data of the augmented reality microservice (930c) can be transmitted to the display (180) and displayed.

[0243] Figure 8c illustrates the execution of multiple microservices corresponding to QM.

[0244] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure can execute a passenger monitoring application (940) corresponding to QM.

[0245] For example, a signal processing device (170) according to an embodiment of the present disclosure can execute a passenger monitoring application (940) by separating it into multiple micro-services.

[0246] The drawing illustrates multiple microservices for a passenger monitoring application (940), including a passenger press detection microservice (950b), a passenger movement microservice (955b), a passenger detection microservice (960b), and a graphics provision microservice (965b).

[0247] That is, the signal processing device (170) according to the embodiment of the present disclosure can execute a plurality of microservices, namely, a passenger seating microservice (950b), a passenger movement microservice (955b), a passenger detection microservice (960b), and a graphics provision microservice (965b), for a passenger monitoring application (940) corresponding to QM.

[0248] Meanwhile, the passenger seating microservice (950b) is executed based on sensor data from the pressure sensor (SNp), and the result data of the passenger seating microservice (950b) is transmitted to the passenger movement microservice (955b).

[0249] Meanwhile, the passenger movement microservice (955b) is executed based on the result data of the passenger seating microservice (950b), and the result data of the passenger movement microservice (955b) is transmitted to the passenger detection microservice (960b).

[0250] Meanwhile, the passenger detection microservice (960b) is executed based on the result data of the passenger movement microservice (955b), and the result data of the passenger detection microservice (960b) is transmitted to the graphics provision microservice (965b).

[0251] Meanwhile, the graphic provision microservice (965b) is executed based on the result data of the passenger detection microservice (960b), and the result data of the graphic provision microservice (965b) can be transmitted to the display (180) and displayed.

[0252] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a gas detection (CO2 detection) microservice (950c) for a passenger monitoring application (940) corresponding to QM.

[0253] The gas detection (CO2 detection) microservice (950c) is executed based on sensor data from the gas sensor (SNc), and the result data of the gas detection (CO2 detection) microservice (950c) is transmitted to the passenger movement microservice (955b).

[0254] Meanwhile, the passenger movement microservice (955b) is executed based on the result data of the gas detection (CO2 detection) microservice (950c), and the result data of the passenger movement microservice (955b) is transmitted to the passenger detection microservice (960b).

[0255] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a plurality of micro-services unrelated to the passenger monitoring application (940).

[0256] In the drawing, the signal processing unit (170) is illustrated as executing a plurality of microservices, each of which is unrelated to the passenger monitoring application (940), including a face detection microservice (910a), a face recognition microservice (915a), and a personal microservice (920a) based on camera data from an in-vehicle camera (195i).

[0257] Figure 8d illustrates the execution of multiple microservices corresponding to ASIL B and microservices corresponding to QM.

[0258] Referring to the drawing, the signal processing device (170) according to the embodiment of the present disclosure can execute a passenger monitoring application (940) corresponding to QM, similar to FIG. 8c.

[0259] For example, a signal processing device (170) according to an embodiment of the present disclosure can execute a plurality of microservices, for a passenger monitoring application (940) corresponding to QM, including a passenger seating microservice (950b), a passenger movement microservice (955b), a passenger detection microservice (960b), and a graphics provision microservice (965b).

[0260] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a gas detection (CO2 detection) microservice (950c) for a passenger monitoring application (940) corresponding to QM.

[0261] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute a plurality of micro-services, including a face detection micro-service (910a), a face recognition micro-service (915a), and a personal micro-service (920a), which are not related to the passenger monitoring application (940).

[0262] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a passenger monitoring application (945) corresponding to ASIL B.

[0263] In the drawing, multiple microservices for an occupant monitoring application (945) corresponding to ASIL B are illustrated, including an occupant seating microservice (950d), an occupant movement microservice (955d), an occupant detection microservice (960d), and an alert microservice (965d).

[0264] Meanwhile, the passenger seating microservice (950d) is executed based on sensor data from the pressure sensor (SNp), and the result data of the passenger seating microservice (950d) is transmitted to the passenger movement microservice (955d).

[0265] Meanwhile, the passenger movement microservice (955d) is executed based on the result data of the passenger seating microservice (950d), and the result data of the passenger movement microservice (955d) is transmitted to the passenger detection microservice (960d).

[0266] Meanwhile, the passenger detection microservice (960d) is executed based on the result data of the passenger movement microservice (955d), and the result data of the passenger detection microservice (960d) is transmitted to the warning microservice (965d).

[0267] Meanwhile, the warning microservice (965d) is executed based on the result data of the passenger detection microservice (960d), and the result data of the warning microservice (965d) can be transmitted to and output by the audio output unit (185).

[0268] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can further execute a gas detection (CO2 detection) microservice (950e) for a passenger monitoring application (945) corresponding to ASIL B.

[0269] Meanwhile, the gas detection (CO2 detection) microservice (950e) is executed based on sensor data from the gas sensor (SNc), and the result data of the gas detection (CO2 detection) microservice (950e) can be transmitted to the passenger movement microservice (955d).

[0270] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may not transmit the result data of the passenger detection microservice (960b) among the microservices in the passenger monitoring application (940) corresponding to QM to the warning microservice (965d) in the passenger monitoring application (945) corresponding to ASIL B.

[0271] That is, since the safety level of the passenger detection microservice (960b) within the passenger monitoring application (940) corresponding to QM is lower than the safety level of the warning microservice (965d) within the passenger monitoring application (945) corresponding to ASIL B, the signal processing device (170) according to the embodiment of the present disclosure cannot transmit the result data of the passenger detection microservice (960b) to the warning microservice (965d) within the passenger monitoring application (945) corresponding to ASIL B. Accordingly, each safety level can be maintained.

[0272] Figure 8e illustrates another example of execution of multiple microservices corresponding to ASIL B and microservices corresponding to QM based on camera data from an internal camera (195i).

[0273] Referring to the drawing, the signal processing device (170) according to the embodiment of the present disclosure can execute a driver monitoring system (DMS) application (985) corresponding to ASIL B, similarly to FIG. 8b.

[0274] The Driver Monitoring System (DMS) application (985) is similar to the Driver Monitoring System (DMS) application (905) of FIG. 8b, but differs in that the second face detection microservice (910c) is not performed.

[0275] For example, a signal processing device (170) according to an embodiment of the present disclosure can execute a plurality of microservices, including a face detection microservice (910b), an eye movement microservice (915b), a head movement microservice (915c), a gaze tracking microservice (920b), and a warning microservice (930b), for a driver monitoring system (DMS) application (985) corresponding to ASIL B.

[0276] Meanwhile, the head movement microservice (915c) is executed based on the result data of the face detection microservice (910b), and the result data of the head movement microservice (915c) is transmitted to the gaze tracking microservice (920b).

[0277] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute a third face detection microservice (910a), a face recognition microservice (915a), and a personal microservice (920a) as a plurality of microservices, which are not related to ASIL B.

[0278] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure can execute an augmented reality microservice (930c) as a microservice corresponding to QM.

[0279] At this time, the signal processing device (170) according to the embodiment of the present disclosure can transmit the result data of the eye tracking microservice (920b) among the microservices in the application (985) corresponding to ASIL B to the augmented reality microservice (930c) corresponding to QM, which has a lower safety level.

[0280] Accordingly, the augmented reality microservice (930c) corresponding to QM is executed based on the result data of the gaze tracking microservice (920b), and the result data of the augmented reality microservice (930c) can be transmitted to the display (180) and displayed.

[0281] FIG. 9 is an example of an internal block diagram of a signal processing device according to an embodiment of the present disclosure.

[0282] Referring to the drawing, a signal processing device (170) in a system (1000) according to one embodiment of the present disclosure includes a processor (175) that executes a hypervisor (505).

[0283] Meanwhile, the processor (175) can correspond to the central processor (CPU) of FIG. 6.

[0284] Meanwhile, the processor (175) may have multiple processor cores.

[0285] The drawing illustrates multiple processor cores each operating based on a safety level of ASIL B, but various variations are possible.

[0286] For example, some of the multiple processor cores may operate based on a safety level of ASIL B, while others may operate based on a safety level of QM.

[0287] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a second processor (177) including an M core or MCU (micom nuit) for executing applications of the highest safety level, ASIL D.

[0288] Meanwhile, the processor (175) executes multiple virtual machines (810 to 830) on the hypervisor (505).

[0289] Meanwhile, among the multiple virtualization machines (810 to 830), the first virtualization machine (830) executes multiple microservices (910b, 915b, 920b, 930b) corresponding to the first safety level such as ASIL B.

[0290] Meanwhile, among the plurality of virtual machines (810 to 830), the first virtual machine (830) transmits the result data of the first microservice (920b) among the plurality of microservices to the second virtual machine (820) among the plurality of virtual machines (810 to 830) corresponding to the second security level lower than the first security level or to the virtual machine within the second signal processing device (170Z). Accordingly, data processing can be performed efficiently using the microservice.

[0291] Meanwhile, the first virtual machine (830) can execute multiple microservices separately to execute the first application corresponding to the first security level.

[0292] That is, among the plurality of virtual machines (810 to 830), the first virtual machine (830) can separately execute a face detection microservice (910b), an eye movement microservice (915b), a gaze tracking microservice (920b), and a warning microservice (930b) as a plurality of microservices for a driver monitoring system (DMS) application (905) corresponding to a first safety level such as ASIL B.

[0293] Meanwhile, the second virtual machine (820) executes a second application corresponding to the second security level, and the second application can be executed based on the result data of the first microservice (920b).

[0294] For example, among multiple virtualization machines (810 to 830), the second virtualization machine (820) can execute an augmented reality microservice (930c) as a microservice corresponding to the second safety level, QM, as shown in FIG. 8b.

[0295] Meanwhile, among the multiple virtualization machines (810 to 830), the second virtualization machine (820) can further execute a facial recognition microservice (915a) as a microservice corresponding to the second security level, QM, as shown in FIG. 8b.

[0296] Meanwhile, among the multiple virtualization machines (810 to 830), the second virtualization machine (820) can execute an augmented reality microservice (930c) as a microservice corresponding to QM.

[0297] Meanwhile, the first virtual machine (830) can transmit the result data of the eye tracking microservice (920b) among the microservices in the application (905) corresponding to ASIL B to the augmented reality microservice (930c) corresponding to QM, which has a lower safety level.

[0298] That is, the first virtual machine (830) can transmit the result data of the eye tracking microservice (920b) among the microservices within the application (905) corresponding to ASIL B to the augmented reality microservice (930c) within the second virtual machine (820), which has a lower safety level, as shown in FIG. 8b.

[0299] Specifically, the first virtual machine (830) may execute a face detection microservice (910b), an eye movement microservice (915b), a gaze tracking microservice (920b), and a warning microservice (930b) based on the received camera data, and transmit the result data of the gaze tracking microservice (920b) to the second virtual machine (820) corresponding to the second safety level.

[0300] Meanwhile, the second virtual machine (820) can be controlled to execute the augmented reality microservice (930c) based on the result data of the eye tracking microservice (920b), as shown in FIG. 8b, and display the result data of the augmented reality microservice (930c) on the display (180).

[0301] In this way, since there is no need to run a separate eye tracking microservice (920b) within the second virtual machine (820), data processing can be performed efficiently.

[0302] Meanwhile, the first virtual machine (830) can transmit the result data of the first microservice (920b) to at least one virtual machine corresponding to a second safety level lower than the first safety level using the shared memory (508).

[0303] Meanwhile, the first virtual machine (830) can transmit the result data of the first micro service (920b) to the second virtual machine (820) corresponding to the second safety level lower than the first safety level using the shared memory (508).

[0304] Meanwhile, the first virtual machine (830) can transmit the result data of the eye tracking microservice (920b) to the augmented reality microservice (930c) within the second virtual machine (820) using the shared memory within the hypervisor (505).

[0305] In this way, when using shared memory (508) to transmit result data, 1:n result data transmission becomes possible.

[0306] Meanwhile, the second virtual machine (820) may not transmit the result data of the microservice being executed or the result data of the application to the first virtual machine (830).

[0307] That is, the second virtual machine (820) may not transmit data to the first virtual machine (830) with a higher security level.

[0308] For example, the result data of the facial recognition microservice (915a) within the second virtual machine (820) is not transmitted to the first virtual machine (830). Accordingly, the security level of each virtual machine can be maintained.

[0309] Meanwhile, among the plurality of virtual machines (810 to 830), the first virtual machine (830) may not transmit the result data of the first micro service (920b) among the plurality of micro services to a virtual machine of a third safety level higher than the first safety level.

[0310] For example, if the third virtual machine (810) among the plurality of virtual machines (810 to 830) has a third safety level higher than the first safety level, such as ASIL D, the first virtual machine (830) may not transmit the result data of the first microservice (920b) among the plurality of microservices to the third virtual machine (810) or the fourth virtual machine (840) at the third safety level higher than the first safety level. Accordingly, the safety level of each virtual machine can be maintained.

[0311] Meanwhile, some cores of the processor (175) may execute a first virtual machine (830), and other cores of the processor (175) may execute a second virtual machine (820).

[0312] In the drawing, some cores of the processor (175) execute a first virtual machine (830) corresponding to the first safety level, ASIL B, and other cores of the processor (175) execute a second virtual machine (820) corresponding to the second safety level, QM. Accordingly, data processing can be performed efficiently using microservices.

[0313] Meanwhile, the second processor (177) can execute an application or virtual machine of ASIL D, the highest safety level.

[0314] Meanwhile, the security level of the application or virtual machine running on the second processor (177) may be higher than the first security level. Accordingly, the stability of the recognition service can be improved.

[0315] Meanwhile, the second virtual machine (820) can execute a face recognition microservice (915a) based on the received camera data, as shown in FIG. 8d, and execute a personal microservice (920a), which is an additional microservice, based on the result data of the face recognition microservice.

[0316] Meanwhile, the second virtual machine (820) may execute the passenger seating microservice (950b), the passenger movement microservice (955b), the passenger detection microservice (960b), and the graphics provision microservice (965b) for the passenger monitoring application (940) corresponding to QM, as shown in FIG. 8d, based on the received sensor data or camera data, and may not transmit the service result data of the passenger detection microservice (960b) to the first virtual machine (830).

[0317] Meanwhile, if the service result data of the passenger detection microservice (960b) is not received from the second virtual machine (820), the first virtual machine (830) can execute the passenger seating microservice (950d), the passenger movement microservice (955d), the passenger detection microservice (960d), and the warning microservice (965d) for the passenger monitoring application (945) corresponding to ASIL B, based on the received sensor data or camera data, as shown in FIG. 8d. Accordingly, data processing can be performed efficiently using the microservices.

[0318] Meanwhile, the first virtual machine (830) within the signal processing device (170) according to another embodiment of the present disclosure executes the first application and transmits the result data or intermediate result data of the first application to the second virtual machine (820) or the virtual machine within the second signal processing device (170Z) corresponding to the second security level. Accordingly, data processing can be efficiently performed using microservices.

[0319] Meanwhile, the first virtual machine (830) may execute a first application including a plurality of microservices and transmit result data of at least some of the plurality of microservices to the second virtual machine (820) or the second signal processing device (170Z).

[0320] FIG. 10 is an example of an internal block diagram of a signal processing device according to another embodiment of the present disclosure.

[0321] Referring to the drawing, a signal processing device (170) in a system (1000b) according to another embodiment of the present disclosure can transmit data to a second signal processing device (170) or receive data from the second signal processing device (170).

[0322] Describing the difference from Fig. 9, the second signal processing device (170z) may be an area signal processing device.

[0323] The second signal processing device (170z) has a processor (175z) that executes a hypervisor (505z).

[0324] Meanwhile, the processor (175z) in the second signal processing device (170z) may have multiple processor cores.

[0325] Meanwhile, the second signal processing device (170z) may further include a separate processor (177z) including an M core or MCU (micom nuit) for executing applications of the highest safety level, ASIL D.

[0326] Meanwhile, the processor (175z) can execute at least one virtualization machine (830z) on the hypervisor (505).

[0327] Meanwhile, a separate processor (177z) can run a virtual machine (840z) corresponding to ASIL D, the highest safety level on the M core.

[0328] Meanwhile, camera data from the internal camera (195i) can be transmitted to the signal processing device (170) or the second signal processing device (170z).

[0329] In the drawing, a virtual machine (830z) within a processor (175z) is illustrated executing a camera data-based video stream application (9993).

[0330] Meanwhile, the processor (175) in the signal processing device (170) can receive sensor data or camera data from the second signal processing device (170Z).

[0331] Meanwhile, the first virtual machine (830) within the signal processing device (170) may execute a plurality of micro-services corresponding to the first safety level based on sensor data or camera data, and, as shown in FIG. 8b, may transmit the result data of the first micro-service (920b) among the plurality of micro-services to the second virtual machine (820) corresponding to the second safety level lower than the first safety level.

[0332] Meanwhile, the first virtual machine (830) within the signal processing device (170) may execute a plurality of microservices corresponding to the first safety level based on sensor data or camera data, and transmit the result data of the first microservice (920b) among the plurality of microservices to the virtual machine within the second signal processing device (170Z) corresponding to the second safety level lower than the first safety level.

[0333] For example, a first virtual machine (830) within a signal processing device (170) can transmit the result data of the gaze tracking microservice (920b) to a virtual machine (830z) within a second signal processing device (170Z) corresponding to the same safety level, ASIL B.

[0334] Accordingly, the virtual machine (830z) within the second signal processing device (170Z) can perform data processing efficiently without having to separately execute the gaze tracking microservice (920b).

[0335] FIG. 11 is a diagram illustrating the operation of a signal processing device according to an embodiment of the present disclosure.

[0336] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175) that executes a hypervisor (505).

[0337] Meanwhile, the processor (175) according to the embodiment of the present disclosure executes, on the hypervisor (505), a recognition service (1210) that performs recognition based on sensor data, and a safety monitor service (1120) that monitors the recognition service (1210), and, when the recognition service (1210) is executed, performs a recovery mode using a redundant recognition service (1210b).

[0338] Accordingly, the stability of sensor data-based recognition services can be improved. Furthermore, the usability of sensor data-based recognition services (1210) can be improved.

[0339] For example, the processor (175) may run an operating system (1103), such as an RTOS, on a hypervisor (505), and may run a workload orchestrator (1110) on the operating system (1103).

[0340] Meanwhile, the processor (175) can execute at least one application on the orchestrator (1110).

[0341] For example, the processor (175) can execute an autonomous driving service (1140), a vehicle driving assistance service (1150), a driver monitoring service (1160), or a cockpit service (1170) on the orchestrator (1110).

[0342] Meanwhile, the processor (175) may execute a second operating system (1104) having a lower safety level than the operating system (1103) corresponding to ASIL B or ASIL D on the hypervisor (505), execute a workload orchestrator (1110) on the second operating system (1104), execute a container runtime (1181) on the orchestrator (1110), and execute an IVI service (1180) for an IVI display on the container runtime (1181).

[0343] Meanwhile, the processor (175) executes a container runtime (1141 to 1171) for execution of each service (1140 to 1170), and can execute a service-oriented architecture (SOA) middleware (1143 to 1173), a platform library (1144 to 1174), etc. on each container runtime (1141 to 1171).

[0344] Meanwhile, the processor (175) can execute an autonomous driving application (1145), a vehicle driving assistance application (1155), a driver monitoring application (1165), or a cockpit application (1175) on each platform library (1144 to 1174).

[0345] Meanwhile, the processor (175) can execute a configuration service (1184) and an IVI application, etc., on the container runtime (1181).

[0346] Meanwhile, the processor (175) can execute a safety monitor service (1120) within the orchestrator (1110).

[0347] The safety monitor service (1120) at this time may be a perception safety monitor service.

[0348] Meanwhile, the safety monitor service (1120) can monitor the recognition service (1210) running in at least one application (1145 to 1185).

[0349] For example, the safety monitor service (1120) controls the operation of the redundant recognition service (1210b) during the operation of the recognition service (1210), and when a failure occurs in the recognition service (1210), it can transmit a recovery command to the recognition service (1210) and an activity command to the redundant recognition service (1210b). Accordingly, the stability of the sensor data-based recognition service can be improved.

[0350] Meanwhile, the safety monitoring service (1120) can control the operation transition between the recognition service (1210) or the redundant recognition service (1210b) based on monitoring data or analysis data from the recognition service (1210) or the redundant recognition service (1210b). Accordingly, the stability of the sensor data-based recognition service can be improved.

[0351] Meanwhile, the safety monitoring service (1120) can vary the operation switching criteria between the recognition service (1210) or the redundant recognition service (1210b) based on monitoring data or analysis data from the recognition service (1210) or the redundant recognition service (1210b). Accordingly, the stability of the sensor data-based recognition service can be improved.

[0352] Meanwhile, the processor (175) can perform a recognition service (1210) for autonomous driving (1140), vehicle assisted driving (1150), or driver monitoring (1160) based on sensor data. Accordingly, the usability of the recognition service (1210) can be improved.

[0353] Meanwhile, the processor (175) can perform a recognition service (1210) for active steering, lane keeping, emergency braking, automatic lane change, or traffic jam pilot based on sensor data. Accordingly, the usability of the recognition service (1210) can be improved.

[0354] Meanwhile, the recognition service (1210) may be any one of multiple microservices for application execution.

[0355] For example, the recognition service (1210) may correspond to a microservice for executing object detection or object tracking or lane detection or lane maintenance.

[0356] That is, the processor (175) can perform object detection or object tracking based on the recognition service (1210). Accordingly, the usability of the recognition service can be improved.

[0357] Meanwhile, the processor (175) can execute multiple microservices as described above when executing each application (1145 to 1185). Accordingly, the applications can be executed efficiently.

[0358] For example, when executing an autonomous driving application (1145), the processor (175) may execute a microservice for executing object detection or object tracking or lane detection or lane maintenance based on sensor data received from a sensor device (SN).

[0359] As another example, the processor (175) may execute a microservice for object detection, object tracking, or lane detection based on sensor data received from a sensor device (SN) when executing a vehicle driving assistance application (1155).

[0360] As another example, the processor (175) may execute a microservice for object detection or object tracking based on sensor data received from a sensor device (SN) when executing a driver monitoring application (1165).

[0361] Meanwhile, the signal processing device (170) of FIG. 11 may further include, in addition to the processor (175), a second processor (177) including an MCU (micom unit).

[0362] At this time, the safety level of the second processor (177) may be higher than or equal to that of the processor (175).

[0363] For example, the safety level of the second processor (177) may be ASIL D, and the safety level of the processor (175) may be ASIL B or ASIL D.

[0364] Meanwhile, the second processor (177) executes AUTOSAR (1106) and can execute a power and temperature monitor (1191), a workload monitor (1192), a fail-operational service (1194), a take-over service (1195), an arbitrator service (1196), a safety monitor (1197), or an active safety (1198) on AUTOSAR (1106).

[0365] Meanwhile, the second processor (177) can control the degradation or emergency mode to be performed through the fail operational service (1194).

[0366] Meanwhile, unlike the drawing, the processor (175) may execute a fail operational service (1194), or the processor (175) may control the execution of a degradation or emergency mode.

[0367] Figures 12a to 19b are drawings referenced in the operation description of Figure 11.

[0368] FIG. 12a is a diagram illustrating various services within the autonomous driving application of FIG. 11.

[0369] Referring to the drawing, the processor (175) in the signal processing device (170) can execute the Active Steering (1261), Lane Keeping (1263), Emergency Braking (1265), Automatic Lane Change (1267), or Traffic Jam Pilot (1269) service based on sensor data when executing the autonomous driving application (1145).

[0370] For example, Active Steering (1261), Lane Keeping (1263), Emergency Braking (1265), Automatic Lane Change (1267), or Traffic Jam Pilot (1269) services may be microservices for executing an autonomous driving application (1145).

[0371] Meanwhile, the processor (175) can perform a recognition service (1210) for active steering (1261), lane keeping (1263), emergency braking (1265), automatic lane change (1267), or traffic jam pilot (1269) based on sensor data. Accordingly, the usability of the recognition service (1210) can be improved.

[0372] Figure 12b is a drawing referenced in the operation description of the safety monitor service of Figure 11.

[0373] Referring to the drawing, the interface (1205) within the signal processing device (170) can receive sensor data from various sensor devices (SN).

[0374] Meanwhile, the processor (175) can perform a recognition service (1210) for an autonomous driving application (1145), a vehicle assisted driving application (1155), or a driver monitoring application (1165) based on map data stored in a memory (925), etc., or sensor data from an interface (1205).

[0375] Meanwhile, the processor (175) may execute localization (1250) based on vehicle driving status or weather information and sensor data, execute recognition service (1210) based on localization (1250), execute planning (1220) based on recognition service (1210), and output a vehicle control signal based on planning (1220).

[0376] For example, the safety monitor service (1120) may transmit vehicle driving conditions such as tunnel driving or weather information such as light rain for localization (1250) processing.

[0377] Meanwhile, the processor (175) can perform localization (1250) based on vehicle driving status or weather information, and sensor data.

[0378] Meanwhile, the safety monitor service (1120) can transmit information such as confidence level, weather condition, or sensor corruption to the interface (1205).

[0379] Meanwhile, the safety monitor service (1120) can transmit workload information, such as delay, to the recognition service (1210).

[0380] Meanwhile, the processor (175) can execute the recognition service (1210) based on workload information such as the result of localization (1250) or delay.

[0381] For example, the processor (175) can execute dynamic object processing such as detection, tracking, and prediction, and traffic light processing such as detection and classification, for executing the recognition service (1210).

[0382] Meanwhile, the processor (175) can execute planning (1220) based on the result of the recognition service (1210) or the result of localization.

[0383] Specifically, the processor (175) can perform mission processing, scenario processing, etc., to execute planning (1220).

[0384] For example, the processor (175) can perform processing such as scenario selection, lane driving, and parking when processing a scenario.

[0385] Meanwhile, the processor (175) can output a vehicle control signal for vehicle control (1230) based on the result of planning (1220).

[0386] At this time, the vehicle control signal can be transmitted to each device in the vehicle through the vehicle interface (1270).

[0387] For example, the vehicle control signal may be a steering control signal, a lane keeping signal, an emergency braking signal, or a lane change signal.

[0388] Accordingly, the processor (175) can perform vehicle control suitable for vehicle driving conditions or weather information, etc., based on sensor data.

[0389] Meanwhile, the processor (175) can vary the mode for planning (1220) or control (1230) for vehicle control based on the confidence level.

[0390] For example, the processor (175) can control the execution of normal mode, degradation mode, emergency mode, or manual driving mode based on the confidence level. Accordingly, the stability of a sensor data-based recognition service can be improved based on the confidence level.

[0391] Meanwhile, the processor (175) can control the recovery mode to be enabled by utilizing the redundant recognition service, as described above, when executing the recognition service (1210). Accordingly, the stability of the recognition service can be improved, ultimately enabling stable vehicle control.

[0392] Figure 13a is a drawing referenced in the description of the operation of the orchestrator.

[0393] Referring to the drawing, the orchestrator (1110) can receive fail-safe data (1305).

[0394] For example, fail-safe data (1305) may include a main recognition service and a redundant recognition service for the recognition service.

[0395] The orchestrator (1110) can monitor (1310) the main recognition service (1210a) based on the reception of fail-safe data (1305).

[0396] Meanwhile, the processor (175) can control the main recognition service (1210a) to be kept in an active state and the redundant recognition service (1210b) to be in a standby state based on the fail-safe data (1305).

[0397] That is, the safety monitor service (1120) can control the operation of the redundant recognition service (1210b) to be temporarily suspended during the operation of the recognition service (1210).

[0398] Meanwhile, the main recognition service (1210a) within the processor (175) can perform recognition based on sensor data from the interface (1205) and transmit the recognition result to planning (1220).

[0399] Figure 13b illustrates that both the main service and the redundant service are active.

[0400] Referring to the drawing, the processor (175) can execute an orchestrator (1110), a main service, and a redundant service.

[0401] For example, the main service may be a main recognition service (1210a), and the redundant service may be a redundant recognition service (1210b).

[0402] That is, the processor (175) can execute the main recognition service (1210a) or the redundant recognition service (1210b) based on the sensor data.

[0403] Meanwhile, the processor (175) can control the main recognition service (1210a) and the redundant recognition service (1210b) to be performed in different containers.

[0404] In the drawing, the first application (1226) is executed within the main recognition service (1210a) and the second application (1236) is executed within the redundant recognition service (1210b), but various modifications are possible.

[0405] For example, the processor (175) can run the main recognition service (1210a) and the redundant recognition service (1210b) within the same container or the same application.

[0406] Meanwhile, a proxy router (1324) within the main recognition service (1210a) or a proxy router (1334) within the redundant recognition service (1210b) can transmit data to the simulator (1330).

[0407] Meanwhile, the monitor (1322) within the main recognition service (1210a) or the monitor (1324) within the redundant recognition service (1210b) can monitor the proxy router (1324) or the proxy router (1334), respectively.

[0408] For example, a monitor (1322) within the main recognition service (1210a) or a monitor (1324) within the redundant recognition service (1210b) may send a failure notification to the orchestrator (1110) when a data error occurs.

[0409] Meanwhile, the monitor (1322) within the main recognition service (1210a) or the monitor (1324) within the redundant recognition service (1210b) can detect a failure or perform error detection through periodic monitoring.

[0410] Meanwhile, the orchestrator (1110) can control the updating of the monitoring module (1322, 1332) or management module within the recognition service (1210) or the redundant recognition service (1210b). Accordingly, the stability of the sensor data-based recognition service can be improved.

[0411] Figure 13c is a drawing referenced in the operation description of the main recognition service or redundant recognition service of Figure 13b.

[0412] Referring to the drawing, the processor (175) can control the main recognition service (1210a) and the redundant recognition service (1210b) to attempt data transmission in the default state, which is the running state (1392).

[0413] Meanwhile, the processor (175) can control data transmitted by the redundant recognition service (1210b) to be blocked.

[0414] Meanwhile, if the transmission is interrupted (1396) due to a failure of the main recognition service (1210a), the processor (175) can control the orchestrator (1110) to transmit a restart command to the main recognition service (1210a).

[0415] Meanwhile, the processor (175) can control the pause state (1394) to be deactivated when both the main recognition service (1210a) and the redundant recognition service (1210b) are active.

[0416] Figure 13d illustrates that the main service is active and the redundant service is passive.

[0417] Referring to the drawing, the processor (175) can execute an orchestrator (1110), a main recognition service (1210a), and a redundant recognition service (1210b), as shown in FIG. 13b.

[0418] Meanwhile, the processor (175) can control the main recognition service (1210a) to be in an operating state (1392) and the redundant recognition service (1210b) to be in a paused state (1394) based on sensor data.

[0419] Accordingly, the main recognition service (1210a) performs data transmission, and the redundant recognition service (1210b) may temporarily suspend data transmission.

[0420] Meanwhile, if the transmission is interrupted (1396) due to a failure of the main recognition service (1210a), the processor (175) can control the orchestrator (1110) to transmit a recovery command to the redundant recognition service (1210b).

[0421] Accordingly, the redundant recognition service (1210b) begins operation based on the recovery command. This improves the stability of the sensor data-based recognition service.

[0422] Meanwhile, the processor (175) can disable a restart command to the main recognition service (1210a) when transmission is interrupted (1396) due to a failure of the main recognition service (1210a).

[0423] Figure 14 is a drawing referenced in the description of the operation of the safety monitor service within the orchestrator.

[0424] Referring to the drawing, the processor (175) can receive sensor data through the interface (1205), perform a recognition service (1210) based on the sensor data, and execute planning (1220) based on the result of the recognition service (1210).

[0425] Meanwhile, the recognition service (1210) can receive fail-safe data (1305b), as shown in the drawing.

[0426] For example, fail-safe data (1305b) may include a main recognition service and a redundant recognition service for the recognition service.

[0427] Meanwhile, the orchestrator (1110) can execute a safety monitor service (1120), as shown in FIG. 11.

[0428] Meanwhile, the safety monitor service (1120) within the orchestrator (1110) can receive fail-safe data (1305b) from the safety monitor service (1120).

[0429] Meanwhile, the processor (175) can control the main recognition service (1210a) to be in an active state and the redundant recognition service (1210b) to be in a standby state based on the fail-safe data (1305b).

[0430] That is, the processor (175) can control the operation of the redundant recognition service (1210b) to be temporarily suspended during the operation of the main recognition service (1210a).

[0431] In the drawing, the first recognition service (1327) is executed within the main recognition service (1210a), and the second recognition service (1337) is executed within the redundant recognition service (1210b), but various modifications are possible.

[0432] Meanwhile, a proxy router (1324) within the main recognition service (1210a) or a proxy router (1334) within the redundant recognition service (1210b) can transmit recognition result data to planning (1220).

[0433] Meanwhile, the analysis module (1325) within the main recognition service (1210a) or the analysis module (1335) within the redundant recognition service (1210b) can calculate and output the confidence level or confidence score of the main recognition service (1210a) or the redundant recognition service (1210b), respectively.

[0434] Meanwhile, the orchestrator (1110) can further execute the gateway (1505).

[0435] Unlike the drawing, the gateway (1505) within the orchestrator (1110) can receive fail-safe data (1305b) and distribute, monitor, or manage the fail-safe data (1305b) to a recognition service (1210), etc.

[0436] Meanwhile, the gateway (1505) can receive the confidence level or confidence score of the main recognition service (1210a) or the redundant recognition service (1210b) from the analysis module (1325) or the analysis module (1335).

[0437] Figure 15 is a drawing referenced in the description of the operation of the gateway of Figure 14.

[0438] Referring to the drawing, the gateway (1505) can process or calculate the operation result data (1305c) based on the confidence level or confidence score from the analysis module (1325) or the analysis module (1335).

[0439] Meanwhile, the gateway (1505) can control the recovery operation to be performed sequentially based on the confidence level or confidence score of the main recognition service (1210a).

[0440] For example, the gateway (1505) can control the redundant recognition service (1210b) to be prepared when the confidence level or confidence score of the main recognition service (1210a) is the first level (e.g., 70).

[0441] As another example, the gateway (1505) may control the redundant recognition service (1210b) to take action when the confidence level or confidence score of the main recognition service (1210a) is at the second level (e.g., 40).

[0442] That is, the gateway (1505) can control the redundant recognition service (1210b) to be sequentially converted to an operational state in a ready state when the confidence level or confidence score sequentially decreases from the first level to the second level.

[0443] Accordingly, when executing the recognition service (1210), a recovery mode can be performed using the redundant recognition service (1210b). Consequently, the stability of the sensor data-based recognition service can be improved.

[0444] Figure 16 is a drawing referenced in the description of the operation of multiple signal processing devices for autonomous driving.

[0445] Referring to the drawing, for executing an autonomous driving application, a processor (175a) within a first signal processing device (170a) may execute an operating system (1605) and execute middleware (1610) on the operating system (1605).

[0446] Meanwhile, the processor (175a) in the first signal processing device (170a) can execute, on the middleware (1610), a sensor fusion service (1620) that processes sensor data such as a camera, radar, and lidar, a mapping service (1626) such as an HD map, road geometry, and lane marking, a localization service (1630) such as location information and 3D geometry, an analysis service (1640) such as internal monitoring (1643) and external monitoring (1644), a recognition service (1210), a planning service (1220), a control service (1645), etc.

[0447] Meanwhile, the processor (175a) in the first signal processing device (170a) can execute a safety monitor service (1120) in addition to the recognition service (1210).

[0448] Meanwhile, the processor (175a) within the first signal processing device (170a) can execute applications (1670) such as object detection (1672), object tracking (1674), lane detection (1674), and lane centering (1678) for traffic jam assist (1670).

[0449] Meanwhile, the second signal processing device (170b) that performs communication such as Ethernet with the first signal processing device (170a) may be equipped with a processor (175b).

[0450] Meanwhile, the processor (175b) within the second signal processing device (170b) can execute a hypervisor (505b), execute a first operating system (1605b) on the hypervisor (505b), and execute middleware (1610b) on the first operating system (1605b).

[0451] Meanwhile, the processor (175b) in the second signal processing device (170b) can execute driver monitoring (1680), intelligent cockpit (1682), vehicle driving assistance (ADAS) (1684), etc. on the middleware (1610b).

[0452] Meanwhile, the processor (175b) in the second signal processing device (170b) can execute a second operating system (1605c) separate from the first operating system (1605b) on the hypervisor (505b), and can execute a setting service (1683), an IVI application (1685), etc. on the second operating system (1605c).

[0453] Meanwhile, the processor (175b) in the second signal processing device (170b) can execute a recognition service (1210) and a safety monitor service (1120).

[0454] Figure 17a is a diagram illustrating various modes being performed.

[0455] Referring to the drawing, the processor (175) within the signal processing device (170) may perform a normal mode (S1710) and then perform a degradation mode based on the confidence level (S1715).

[0456] For example, the processor (175) can switch from normal mode to degradation mode based on the recognition service (1210).

[0457] Specifically, the processor (175) within the signal processing device (170) can be controlled to perform normal mode when the confidence level is the first level, as shown in FIG. 15, and to stop normal mode and perform degradation mode when the confidence level is the second level lower than the first level.

[0458] Here, the degradation mode may include a recovery mode based on the operation of the redundant recognition service (1210b).

[0459] Meanwhile, the processor (175) can control the vehicle to reduce speed or adjust steering in degradation mode. Accordingly, the usability of the recognition service can be improved.

[0460] Meanwhile, the processor (175) can control the execution of an emergency mode or a manual driving mode according to a switching request while performing the degradation mode.

[0461] For example, the processor (175) can control the manual driving mode to be performed (S1728) when a switching request is received within a predetermined time while performing the degradation mode (S1720).

[0462] As another example, the processor (175) may control the emergency mode to be executed if a transition request is not received within a predetermined time while performing the degradation mode (S1725). In addition, the processor (175) may control the vehicle to stop according to the emergency mode (S1735).

[0463] That is, the processor (175) can control the vehicle to stop after performing the emergency mode based on a transition request while performing the degradation mode based on the recognition service (1210). Accordingly, the usability of the recognition service can be improved.

[0464] Meanwhile, the processor (175) can control the normal mode for autonomous driving to be performed again (S1710) when there is a request for autonomous driving (S1730) while performing manual driving mode.

[0465] Figure 17b is a drawing referenced in the description of Figure 17a.

[0466] Referring to the drawing, the processor (175) can control the normal mode to be performed until the Ta1 point. Accordingly, the vehicle speed can be approximately Ya3.

[0467] Meanwhile, the processor (175) can switch from normal mode to degradation mode based on the recognition service (1210).

[0468] That is, the processor (175) can control the vehicle speed to decrease to approximately Ya2, depending on the degradation mode, from the Ta1 point in time.

[0469] Meanwhile, the processor (175) can control the vehicle speed to be maintained at a constant speed Ya2 for a predetermined period of time after the vehicle speed decreases, depending on the degradation mode.

[0470] Meanwhile, the processor (175) can control the execution of an emergency mode or a manual driving mode according to a switching request while performing the degradation mode.

[0471] For example, the processor (175) can control to switch to manual driving mode when a switching request is received within a predetermined time while performing degradation mode.

[0472] The drawing illustrates that the vehicle speed increases from the Ts3 point by switching to manual driving mode.

[0473] As another example, the processor (175) can control the emergency mode to be performed if a transition request is not received within a predetermined time while performing the degradation mode.

[0474] The drawing illustrates that, depending on the emergency mode execution, the vehicle speed decreases from the Ta4 point.

[0475] That is, the processor (175) can control the vehicle speed to be reduced depending on the emergency mode execution.

[0476] Meanwhile, the processor (175) can control the vehicle to eventually stop after reducing the vehicle speed according to the emergency mode execution. Accordingly, vehicle safety can be ensured.

[0477] In the drawing, the vehicle speed is exemplified as Ya1 at time Ta5, but Ya1 can correspond to the vehicle stopping.

[0478] Figure 18 is a drawing for reference in explaining the operation of a processor for vehicle driving.

[0479] Referring to the drawing, the sensor device (SN) may include a vehicle sensor (1802), an inertial sensor (1804), a lidar (196), a radar (197), a camera (195), a position sensor (1806), a speed sensor (1808), etc.

[0480] Various sensor data from the sensor device (SN) can be input to the processor (175) in the signal processing device (170).

[0481] Meanwhile, the interface (1205) within the processor (175) can receive sensor data from a sensor device (SN) and transmit the sensor data to a recognition service (1210).

[0482] For this purpose, the interface (1205) may run or be equipped with a ROS bridge (1813) or an autoware bridge (1815).

[0483] For example, the ROS bridge (1813) can transmit sensor data from a sensor device (SN) to a recognition service (1210) and transmit control commands from the recognition service (1210) or a safety monitor service (1120) to the sensor device (SN).

[0484] Specifically, the ROS bridge (1813) can transmit sensor data from an inertial sensor (1804) or sensor data from a position sensor (1806) to localization (1822) within the perception service (1210).

[0485] Meanwhile, the ROS bridge (1813) can transmit sensor data from a lidar (196) or radar (197) or camera (195) to object detection tracking (1824) within the recognition service (1210).

[0486] Meanwhile, the autoware bridge (1815) can transmit speed data or vehicle status data from the speed sensor (1808) to the planning (1220).

[0487] Meanwhile, the recognition service (1210) can execute localization (1822), object detection tracking (1824), etc. for autonomous driving or vehicle driving assistance.

[0488] Meanwhile, the safety monitor service (1120) can monitor the recognition service (1210).

[0489] Meanwhile, planning (1220) can execute planning based on the result data of the recognition service (1210), for example, obstacles or poses.

[0490] To this end, planning (1220) can execute global path planning (1832), local path planning (1834), and pure pursuit (1836) related to vehicle paths.

[0491] For example, global path planning (1832) and local path planning (1834) can compute or execute path planning based on obstacle and pose data from the recognition service (1210).

[0492] Meanwhile, path following (1836) can execute path following based on local trajectory data from local path planning (1834) and speed data or vehicle status data from autoware bridge (1815).

[0493] Meanwhile, when performing signal processing based on camera data from a camera (195) within a sensor device (SN), it is difficult to know the exact distance to an object, and there is a disadvantage in that it is significantly dependent on weather conditions.

[0494] Accordingly, the processor (175) can perform object detection, such as a pedestrian or bicycle, which is a close-range object, based on camera data from the camera (195).

[0495] Meanwhile, when performing signal processing based on radar data from a radar (197) within a sensor device (SN), there is a disadvantage in that it is difficult to identify small objects and the type of object cannot be read.

[0496] Accordingly, the processor (175) can perform long-distance object recognition, night-time vehicle recognition, or object recognition in rainy conditions, based on radar data from the radar (197).

[0497] Meanwhile, when performing signal processing based on lidar data from lidar (196) within a sensor device (SN), the detection distance is shorter than that of radar (197) and there is a disadvantage in that it is sensitive to weather conditions.

[0498] Accordingly, the processor (175) can perform mid-range object recognition, night-time vehicle recognition, or small object recognition on the ground, based on the rider data from the rider (196).

[0499] Meanwhile, the processor (175) can perform sensor fusion processing based on multiple sensor data.

[0500] For example, the processor (175) may perform first object detection based on rider data from the rider (196), detect a second object based on camera data from the camera (195), and perform object matching based on the first object detection and the second object detection. Accordingly, more precise object detection or matching may be performed.

[0501] Figure 19a is a drawing referenced in the description of localization of Figure 18.

[0502] Referring to the drawing, an interface (1205) within a processor (175) can receive sensor data from a sensor device (SN) and transmit the sensor data to a recognition service (1210).

[0503] For example, the interface (1205) within the processor (175) can receive rider data from a rider (196) and position data from a position sensor (1806).

[0504] Meanwhile, the rider driver (1902) within the processor (175) can process rider data and output 3D rider data.

[0505] Meanwhile, the location information driver (1912) within the processor (175) can process location data and output sample location data.

[0506] Meanwhile, the reference localization map loader (1914) within the processor (175) can output a Lidar reference map based on the reference localization map (1916) and sample location data.

[0507] Meanwhile, Lidar Odometry within the processor (175) can calculate the position of an object and output the calculated position information.

[0508] Meanwhile, the first corrector (1906) within the processor (175) can execute the first correction, which is a global correction, based on the position information calculated with the lidar reference map.

[0509] Meanwhile, the second corrector (1908) within the processor (175) can perform a second correction based on the result data of the first corrector (1906) and the lidar reference map.

[0510] Meanwhile, the output filter (1909) within the processor (175) can filter and output the result data of the second compensator (1908).

[0511] Accordingly, the localization (1822) within the processor (175) can execute localization based on the rider data or the location data from the location sensor (1806).

[0512] Figure 19b is a diagram illustrating various modes executed on the processor.

[0513] Referring to the drawing, on the other hand, the processor (175) can be controlled to perform a fail-safe mode, a degradation mode, or a fail operation mode based on the recognition service (1210).

[0514] In particular, the processor (175) can control the execution of a fail-safe mode, a degradation mode, or a fail-operation mode based on a confidence level, etc.

[0515] At this time, the fail operation mode can correspond to the above-described normal mode, and the fail safe mode can correspond to the above-described emergency mode or manual driving mode.

[0516] For example, the processor (175) may execute a redundancy service, for example, a redundant recognition service (1210b), depending on the fail operation mode.

[0517] Meanwhile, the processor (175) can control the vehicle to reduce speed or perform steering adjustment in the degradation mode, depending on the degradation mode.

[0518] Meanwhile, the processor (175) can be controlled to operate in a manual driving mode that depends on the driver, depending on the fail-safe mode.

[0519] Meanwhile, in the drawing, for convenience of explanation, it is illustrated that when the confidence level is 0 or 1, the fail-safe mode is performed, when the confidence level is 2 or 3, the degradation mode is performed, and when the confidence level is 4 or 5, the fail-operation mode is performed.

[0520] That is, the processor (175) can be controlled to perform fail-safe mode when the confidence level is the lowest, and can be controlled to perform fail-operation mode when the confidence level is the highest.

[0521] FIG. 20 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0522] Referring to the drawing, the processor (175) within the signal processing device (170) can execute an autonomous driving or vehicle driving assistance application (S2010).

[0523] At this time, the processor (175) can execute a recognition service (1210) that performs recognition based on sensor data and a safety monitor service (1120) that monitors the recognition service (1210).

[0524] Meanwhile, the safety monitor service (1120) within the processor (175) can determine whether a monitoring signal is received (S2015) and, if so, control the dynamic driving task to be performed (S2044).

[0525] Meanwhile, the safety monitor service (1120) within the processor (175) can reset the component if a monitoring signal is not received.

[0526] For example, the processor (175) can reset a component running on the processor (175) in fail-safe mode. Accordingly, the usability of the recognition service can be improved.

[0527] Meanwhile, the safety monitor service (1120) within the processor (175) determines whether a monitoring signal is not received and a switching request is made within a predetermined time (Ts) (S2022), and if so, controls a switching alarm notification to be executed (S2025).

[0528] The transition request at this time may be a transition request to degradation mode.

[0529] And, the safety monitor service (1120) within the processor (175) can control the degradation mode to be performed (S2030).

[0530] For example, the processor (175) can control the speed of the vehicle to be reduced depending on the degradation mode.

[0531] Next, the processor (175) can determine whether there is a switching request within a second predetermined time (Tsb) while performing the degradation mode, and if so, control the manual driving mode to be performed (S2035).

[0532] Meanwhile, the processor (175), when performing degradation mode, if there is no switching request or if the switching request has passed the second predetermined time (Tsb), determines whether the vehicle is in a safe state (S2037), and if not in a safe state, controls the vehicle to stop in the lane as an emergency mode (S2040).

[0533] Meanwhile, the processor (175), in step 2037 (S2037), if in a safe state, can control the lane to be changed to move to the shoulder (S2042).

[0534] Accordingly, the stability of driving services can be improved while performing various modes.

[0535] Finally, according to FIG. 20, if a monitoring signal is not received while the autonomous driving mode is being performed, the processor (175) performs fail-safe mode. If the fail-safe mode fails, the processor performs degradation mode. After the degradation mode, the processor performs an alternative operation or controls the vehicle to stop. Accordingly, the usability of the recognition service can be improved.

[0536] Figure 21 is a drawing referenced in the description of Figure 20.

[0537] Referring to the drawing, the lane control module (2140) within the signal processing device (170) can receive yaw rate (2131), roll rate (2133), pitch rate (2135), acceleration information (2137), etc. from the vehicle sensor devices (2130).

[0538] Meanwhile, the external environment data driver (2105) can provide the collected external environment data to rain detection sensors (2120) or a display (180).

[0539] Meanwhile, the rain control module (2140) within the signal processing device (170) can receive sensor data from an offline sensor (2121) or an online sensor (2123) among the rain detection sensors (2120).

[0540] For example, sensor data from an offline sensor (2121) or an online sensor (2123) may include road curvature, lane markings, lane width, location information, road hazards, local map data, etc.

[0541] Meanwhile, the rain control module (2140) can control the rain control to be performed based on sensor data from the vehicle sensor devices (2130) or the rain control module (2140).

[0542] For example, the lane control module (2140) can output steering information, braking information, torque information, etc. based on sensor data.

[0543] Meanwhile, the steering module (2151) within the control module (2150) can output a steering control signal to the vehicle interface (2160) based on steering information.

[0544] Meanwhile, the steering module (2151) within the control module (2150) can output a steering control signal to the vehicle interface (2160) based on steering information.

[0545] Meanwhile, the braking module (2153) within the control module (2150) can output a braking control signal to the vehicle interface (2160) based on the braking information.

[0546] Meanwhile, the propulsion module (2157) within the control module (2150) can output a propulsion control signal based on propulsion information.

[0547] Meanwhile, the active differential module (2158) within the control module (2150) can output an acceleration control signal to the vehicle interface (2160) based on propulsion information, etc.

[0548] Meanwhile, vehicle status information output from the rain control module (2140) can be transmitted to the display interface (2110).

[0549] Meanwhile, the display interface (2110) can output main control information (2113), sub-control information (2112), cockpit information (2115), driver status monitoring information (2117), etc., and display them on the display (180).

[0550] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. A processor running a hypervisor; The above processor, On the hypervisor, a recognition service that performs recognition based on sensor data and a safety monitor service that monitors the recognition service are executed, A signal processing device that performs recovery mode by using a redundant recognition service when executing the above recognition service.

2. In paragraph 1, The above safety monitoring service is, During the operation of the above recognition service, the redundant recognition service is controlled to operate, A signal processing device that transmits a recovery command to the recognition service and an activity command to the redundant recognition service when the above recognition service failure occurs.

3. In paragraph 1, The above safety monitoring service is, During the operation of the above recognition service, the operation of the redundant recognition service is controlled to be temporarily suspended, A signal processing device that transmits a recovery command to the recognition service and a start command to the redundant recognition service when the above recognition service failure occurs.

4. In paragraph 1, The above safety monitoring service is, A signal processing device that controls the switching of operations between the recognition service or the redundant recognition service based on monitoring data or analysis data from the recognition service or the redundant recognition service.

5. In paragraph 1, The above safety monitoring service is, A signal processing device that varies the operation switching criteria between the recognition service or the redundant recognition service based on monitoring data or analysis data from the recognition service or the redundant recognition service.

6. In paragraph 1, The above safety monitoring service is, A signal processing device that controls updating of a monitoring module or a management module within the above recognition service or the above redundant recognition service.

7. In paragraph 1, The above safety monitoring service is, A signal processing device that varies a mode for planning or controlling vehicle control based on a confidence level.

8. In paragraph 1, The above processor, A signal processing device that performs the recognition service for autonomous driving or vehicle assisted driving or driver monitoring based on the above sensor data.

9. In paragraph 1, The above processor, A signal processing device that performs the recognition service for active steering, lane keeping, emergency braking, automatic lane change or traffic jam pilot based on the above sensor data.

10. In paragraph 1, The above processor, A signal processing device that executes localization based on vehicle driving status or weather information and the sensor data, executes the recognition service based on the localization, executes planning based on the recognition service, and outputs a vehicle control signal based on the planning.

11. In paragraph 1, The above recognition service is, A signal processing unit that is one of multiple microservices for running an application.

12. In paragraph 1, The above recognition service is, A signal processing device corresponding to a microservice for executing object detection or object tracking or lane detection or lane maintenance.

13. In paragraph 1, The above processor, A signal processing device that controls switching from normal mode to degradation mode based on the above recognition service, or performing emergency mode or manual driving mode according to a switching request while performing the degradation mode.

14. In paragraph 1, The above processor, A signal processing device that controls the vehicle to stop after performing an emergency mode based on a transition request while performing a degradation mode based on the above recognition service.

15. In paragraph 1, The above processor, A signal processing device that performs object detection or object tracking based on the above recognition service.

16. In paragraph 1, The above processor, A signal processing device that controls to perform fail-safe mode, degradation mode, or fail operation mode based on the above recognition service.

17. In paragraph 1, The above processor, A signal processing device that performs a fail-safe mode when a monitoring signal is not received while performing an autonomous driving mode, performs a degradation mode when the fail-safe mode fails, and performs an alternative operation or controls the vehicle to stop after the degradation mode.

18. In paragraph 17, The above processor, A signal processing device that resets a component in the above fail-safe mode.

19. In paragraph 17, The above processor, A signal processing device that controls to reduce vehicle speed or perform steering adjustment in the above degradation mode.

20. At least one display; A signal processing device for outputting a video signal to the display; The above signal processing device, A vehicle display device comprising a signal processing device according to any one of claims 1 to 19.

Images