Signal processing apparatus and display apparatus equipped with same for vehicles

The signal processing device efficiently operates neural processors by using a hypervisor and virtual machines to adjust frame rates based on vehicle data, addressing inefficiencies and power consumption issues, and enabling diverse applications with reduced energy use.

WO2024253222A1PCT designated stage expired Publication Date: 2024-12-12LG ELECTRONICS INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2023/007877
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-06-08
Publication Date
2024-12-12

AI Technical Summary

Technical Problem

Existing signal processing devices for vehicles face challenges in efficiently operating neural processors due to fixed frame rates, leading to increased power consumption and inefficiency, especially when processing varying vehicle driving information and sensor data.

Method used

A signal processing device equipped with a central processor executing a hypervisor and multiple virtual machines, allowing the neural processor to operate at variable frame rates based on vehicle driving information, sensor data, and different modes such as energy saving, speed, and direction, utilizing shared memory for efficient data transmission and processing.

Benefits of technology

This solution enables efficient operation of neural processors, reduces power consumption, and allows for the effective utilization of vehicle driving information for various applications, including vehicle assistance and augmented reality, by dynamically adjusting processing rates based on real-time conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2023007877_12122024_PF_FP_ABST
    Figure KR2023007877_12122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a signal processing apparatus, and a display apparatus equipped with same for vehicles. A signal processing apparatus according to an embodiment of the present disclosure comprises: at least one neural processor; and a central processor that executes a hypervisor, wherein the central processor executes a plurality of virtualization machines by means of the hypervisor, and, on the basis of sensor data from a sensor apparatus inside a vehicle, a first virtualization machine, from among the plurality of virtualization machines, controls the neural processor to operate at a variable frame rate or to output resulting data. Accordingly, the neural processor can be operated effectively.
Need to check novelty before this filing date? Find Prior Art

Description

Signal processing device and vehicle display device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle display device having the same, and more particularly, to a signal processing device capable of efficiently operating a neural processor and a vehicle display device having the same.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, a signal processing device is used for real-time processing of various sensor data while the vehicle is driving.

[0004] At this time, as the number of sensor devices increases, the amount of real-time processing for sensor data increases.

[0005] Meanwhile, Korean Patent Publication No. 10-2463175 (hereinafter referred to as “prior document”) relates to an object recognition method and device, which extracts features from an input image in a neural network to generate a feature map, extracts a region of interest corresponding to at least one object of interest from the input image in parallel with the generation of the feature map, determines the number of object candidate regions used for detection of the object of interest based on the size of the region of interest, and recognizes the object of interest from the region of interest based on the number of object candidate regions in the neural network.

[0006] However, according to prior literature, since the input image is processed at a fixed frame rate, there is a disadvantage in that processing increases significantly and power consumption is significant.

[0007] The technical problem of the present disclosure is to provide a signal processing device capable of efficiently operating a neural processor, and a vehicle display device having the same.

[0008] Meanwhile, another technical task of the present disclosure is to provide a signal processing device capable of efficiently operating a neural processor based on vehicle driving information, and a vehicle display device having the same.

[0009] Meanwhile, another technical challenge of the present disclosure is to provide a signal processing device capable of reducing power consumption, and a vehicle display device having the same.

[0010] Meanwhile, another technical task of the present disclosure is to provide a signal processing device capable of utilizing vehicle driving information based on sensor data in various ways, and a vehicle display device having the same.

[0011] A signal processing device according to one embodiment of the present disclosure for solving the above technical problem comprises at least one neural processor and a central processor executing a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine among the plurality of virtual machines controls the neural processor to operate at a variable frame rate or output result data based on sensor data from a sensor device in a vehicle.

[0012] Meanwhile, the neural processor can detect vehicle driving information based on sensor data, and operate at a variable frame rate based on the vehicle driving information or output result data including the vehicle driving information.

[0013] Meanwhile, the neural processor can receive sensor data at a fixed frame rate, detect vehicle driving information based on the sensor data, and operate at a variable frame rate based on the vehicle driving information or output result data including the vehicle driving information.

[0014] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data based on sensor data during a first period, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data based on vehicle driving information based on the sensor data during a second period after the first period.

[0015] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a fixed frame rate or output result data based on sensor data when the energy saving mode is turned off, and may control a neural processor to operate at a variable frame rate or output result data based on vehicle driving information based on the sensor data when the energy saving mode is turned on.

[0016] Meanwhile, among the plurality of virtual machines, the first virtual machine can control the neural processor to operate at a first frame rate or output result data when the vehicle moves forward, and can control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data based on vehicle driving information based on sensor data when the vehicle moves backward.

[0017] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data when the vehicle is driven at a first speed, and may control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is driven at a second speed that is faster than the first speed.

[0018] Meanwhile, among the plurality of virtual machines, the first virtual machine may control the neural processor to operate at a first frame rate or output result data when the vehicle is driving straight, and may control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is driving in a lane change.

[0019] Meanwhile, among the plurality of virtual machines, the first virtual machine may control the neural processor to operate at a first frame rate or output result data when the vehicle is driving straight, and may control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is driving to the left or right.

[0020] Meanwhile, among the plurality of virtual machines, the first virtual machine may control the neural processor to operate at a first frame rate or output result data when the vehicle is driving, and may control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is stopped.

[0021] Meanwhile, a signal processing device according to one embodiment of the present disclosure further includes a shared memory, and a central processor can transmit sensor data to a neural processor using the shared memory, and the neural processor can transmit result data to the central processor using the shared memory.

[0022] Meanwhile, the frame rate of sensor data recorded in shared memory is fixed, and the frame rate of result data recorded in shared memory can be variable.

[0023] Meanwhile, the first virtual machine can receive vehicle driving information detected by the neural processor and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information.

[0024] Meanwhile, a second virtual machine among the plurality of virtual machines can receive vehicle driving information from the first virtual machine and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information.

[0025] Meanwhile, a second virtual machine among the plurality of virtual machines may receive vehicle driving information from the first virtual machine and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information, and a third virtual machine among the plurality of virtual machines may receive vehicle driving information from the first virtual machine and execute an augmented reality engine based on the vehicle driving information.

[0026] Meanwhile, a second virtual machine can operate for the first display, and a third virtual machine can operate for the second display.

[0027] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a first neural processor among at least one neural processor to operate at a variable frame rate or output result data based on sensor data, and a second neural processor among at least one neural processor may control a second neural processor among at least one neural processor to operate at a variable frame rate or output result data based on camera data from a camera device within the vehicle.

[0028] Meanwhile, among the plurality of virtual machines, a first virtual machine can control a first neural processor to operate at a first frame rate or output result data during a first period, and can control a second neural processor to operate at a second frame rate or output result data during the first period.

[0029] According to another embodiment of the present disclosure, a signal processing device comprises at least one neural processor and a central processor that executes a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine among the plurality of virtual machines controls the neural processor to operate at a fixed frame rate or output result data based on sensor data from a sensor device in a vehicle when an energy saving mode is turned off, and controls the neural processor to operate at a variable frame rate or output result data based on vehicle driving information based on the sensor data when the energy saving mode is turned on.

[0030] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data when the vehicle is moving forward while the energy saving mode is on, and may control a neural processor to operate at a second frame rate, different from the first frame rate, or output result data based on vehicle driving information based on sensor data when the vehicle is moving backward while the energy saving mode is on.

[0031] A vehicle display device according to one embodiment of the present disclosure comprises a sensor device and a signal processing device that detects vehicle driving information based on sensor data from the sensor device.

[0032] A signal processing device according to one embodiment of the present disclosure comprises at least one neural processor and a central processor executing a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine among the plurality of virtual machines controls the neural processor to operate at a variable frame rate or output result data based on sensor data from a sensor device within a vehicle. Accordingly, the neural processor can be operated efficiently. In addition, power consumption can be reduced.

[0033] Meanwhile, the neural processor can detect vehicle driving information based on sensor data, operate at a variable frame rate based on the vehicle driving information, or output result data containing the vehicle driving information. This allows the neural processor to operate efficiently based on the vehicle driving information.

[0034] Meanwhile, the neural processor can receive sensor data at a fixed frame rate, detect vehicle driving information based on the sensor data, and operate at a variable frame rate or output result data including the vehicle driving information based on the vehicle driving information. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information.

[0035] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data based on sensor data during a first period, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data based on vehicle driving information based on the sensor data during a second period following the first period. Accordingly, the neural processor can be operated efficiently.

[0036] Meanwhile, among the plurality of virtual machines, a first virtual machine can control a neural processor to operate at a fixed frame rate or output result data based on sensor data when the energy saving mode is turned off, and can control a neural processor to operate at a variable frame rate or output result data based on vehicle driving information based on the sensor data when the energy saving mode is turned on. Accordingly, the neural processor can be efficiently operated based on vehicle driving information. In addition, power consumption can be reduced.

[0037] Meanwhile, among the plurality of virtual machines, the first virtual machine can control the neural processor to operate at a first frame rate or output result data when the vehicle is moving forward, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is moving backward. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0038] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data when the vehicle is traveling at a first speed, and, when the vehicle is traveling at a second speed that is faster than the first speed, control the neural processor to operate at a second frame rate different from the first frame rate or output result data based on vehicle driving information based on sensor data. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0039] Meanwhile, among the plurality of virtual machines, the first virtual machine can control the neural processor to operate at a first frame rate or output result data when the vehicle is driving straight, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is driving in a lane change. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0040] Meanwhile, among the plurality of virtual machines, the first virtual machine may control the neural processor to operate at a first frame rate or output result data when the vehicle is driving straight, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is driving to the left or right. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0041] Meanwhile, among the plurality of virtual machines, the first virtual machine can control the neural processor to operate at a first frame rate or output result data when the vehicle is in motion, and control the neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is stopped. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0042] Meanwhile, a signal processing device according to one embodiment of the present disclosure further includes a shared memory. The central processor can use the shared memory to transmit sensor data to the neural processor, and the neural processor can use the shared memory to transmit result data to the central processor. This enables efficient operation of the neural processor. Furthermore, power consumption can be reduced.

[0043] Meanwhile, the frame rate of sensor data recorded in shared memory is fixed, while the frame rate of result data recorded in shared memory can be variable. This allows for efficient operation of the neural processor. Furthermore, power consumption can be reduced.

[0044] Meanwhile, the first virtual machine can receive vehicle driving information detected by the neural processor and execute a vehicle driver assistance engine or autonomous driving engine based on the vehicle driving information. This allows for a variety of uses of sensor-based vehicle driving information.

[0045] Meanwhile, among the multiple virtual machines, the second virtual machine can receive vehicle driving information from the first virtual machine and execute a vehicle driving assistance engine or autonomous driving engine based on the vehicle driving information. This allows for a variety of uses of sensor-based vehicle driving information.

[0046] Meanwhile, among the plurality of virtual machines, a second virtual machine may receive vehicle driving information from the first virtual machine and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information. A third virtual machine among the plurality of virtual machines may receive vehicle driving information from the first virtual machine and execute an augmented reality engine based on the vehicle driving information. Accordingly, vehicle driving information based on sensor data may be utilized in various ways.

[0047] Meanwhile, a second virtual machine can operate for the first display, and a third virtual machine can operate for the second display. This allows for efficient control of multiple displays through virtual machines.

[0048] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a first neural processor among at least one neural processor to operate at a variable frame rate or output result data based on sensor data, and a second neural processor among at least one neural processor may control a second neural processor among at least one neural processor to operate at a variable frame rate or output result data based on camera data from a camera device within the vehicle. Accordingly, the neural processor can be efficiently operated based on vehicle driving information. In addition, power consumption can be reduced.

[0049] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a first neural processor to operate at a first frame rate or output result data during a first period, and a second neural processor to operate at a second frame rate or output result data during the first period. Accordingly, the neural processor can be efficiently operated based on vehicle driving information. In addition, power consumption can be reduced.

[0050] According to another embodiment of the present disclosure, a signal processing device comprises at least one neural processor and a central processor that executes a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine among the plurality of virtual machines controls the neural processor to operate at a fixed frame rate or output result data based on sensor data from a sensor device in a vehicle when the energy saving mode is off, and controls the neural processor to operate at a variable frame rate or output result data based on vehicle driving information based on the sensor data when the energy saving mode is on. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0051] Meanwhile, among the plurality of virtual machines, a first virtual machine may control a neural processor to operate at a first frame rate or output result data when the vehicle is moving forward while the energy saving mode is on, and may control a neural processor to operate at a second frame rate, different from the first frame rate, or output result data, based on vehicle driving information based on sensor data when the vehicle is moving backward while the energy saving mode is on. Accordingly, the neural processor can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0052] A vehicle display device according to one embodiment of the present disclosure comprises a sensor device and a signal processing device that detects vehicle driving information based on sensor data from the sensor device. Accordingly, a neural processor can be efficiently operated based on the vehicle driving information. Furthermore, power consumption can be reduced.

[0053] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0054] FIG. 2 is a drawing illustrating the appearance of a vehicle display device according to an embodiment of the present disclosure.

[0055] FIG. 3 illustrates an example of an internal block diagram of the vehicle display device of FIG. 2.

[0056] FIG. 4 is a diagram illustrating a system driven by a signal processing device related to the present disclosure.

[0057] FIG. 5 is a diagram illustrating an example of a system driven by a signal processing device of the present disclosure.

[0058] FIG. 6 is a drawing for reference in explaining the operation of a system driven by a signal processing device of the present disclosure.

[0059] Figures 7a to 9d are drawings referred to in the description of Figure 5 or Figure 6.

[0060] FIG. 10a is an example of an internal block diagram of a signal processing device related to the present disclosure.

[0061] Figure 10b is a drawing referenced in the description of Figure 10a.

[0062] FIG. 11A is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0063] FIGS. 11b to 11d are diagrams illustrating various examples of systems driven by a signal processing device according to an embodiment of the present disclosure.

[0064] FIG. 12 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0065] FIGS. 13A and 13B are various examples of flowcharts showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0066] Figures 14a to 19 are drawings referenced in the description of Figures 12 to 13b.

[0067] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0068] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0069] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0070] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0071] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0072] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside, and an image projection device (180h) for projecting images onto a windshield (WS).

[0073] In Fig. 1, a plurality of displays (180a, 180b) are illustrated, including a cluster display (180a), an AVN (Audio Video Navigation) display (180b), and a head-up display (HUD) image projection device (180h).

[0074] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named a center information display.

[0075] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0076] FIG. 2 is a drawing illustrating the appearance of a vehicle display device according to an embodiment of the present disclosure.

[0077] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), an image projection device (180h), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) or the image projection device (180h).

[0078] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0079] The image projection device (180h) may be equipped with an optical device (not shown) for image projection as a head-up display.

[0080] The signal processing device (170) has a shared memory (508) and a central processor (175) inside, and can execute a first virtual machine to a third virtual machine (520 to 540) on a hypervisor (505) within the central processor (175).

[0081] The first virtual machine (520) is a server virtual machine and can control the second virtual machine (530) and the third virtual machine (540), which are guest virtual machines.

[0082] Meanwhile, the second virtual machine can be named the first guest virtual machine (Guest Virtual Maschine), and the third virtual machine can be named the second guest virtual machine.

[0083] The first guest virtualization machine (530) can operate for the first display (180a), and the second guest virtualization machine (540) can operate for the second display (180b).

[0084] Meanwhile, the server virtualization machine (520) within the central processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the first guest virtualization machine (530) and the second guest virtualization machine (540). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0085] Meanwhile, the server virtualization machine (520) within the central processor (175) can receive and process the vehicle's wheel speed sensor data, and transmit speed information based on the wheel speed sensor data to at least one of the first guest virtualization machine (530) or the second guest virtualization machine (540). Accordingly, the vehicle's speed information can be shared with at least one virtual machine, etc.

[0086] Accordingly, it is possible to control various displays (180a to 180b) or image projection devices (180h) using one signal processing device (170).

[0087] Meanwhile, some of the multiple displays (180a~180b) may operate under Linux OS, while others may operate under Web OS.

[0088] The signal processing device (170) according to the embodiment of the present disclosure can control the display (180a to 180b) operating under various operating systems (OS) to display the same information or the same image in synchronization.

[0089] FIG. 3 illustrates an example of an internal block diagram of a vehicle display device according to an embodiment of the present disclosure.

[0090] Referring to the drawings, a vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120), an interface (130), a memory (140), a signal processing device (170), a plurality of displays (180a to 180b), an image projection device (180h), an audio output unit (185), and a power supply unit (190).

[0091] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0092] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0093] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (not shown).

[0094] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0095] The communication device (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (not shown). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0096] The interface (130) can receive sensor data, etc. from the ECU (770) or sensor device (700) and transmit the received information to the signal processing device (170).

[0097] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0098] Such sensor data may be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc. Meanwhile, the position module may include a GPS module for receiving GPS information.

[0099] Meanwhile, the interface (130) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195) or a rider (not shown), and transmit the received information to a signal processing device (170).

[0100] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0101] For example, the memory (140) may store data regarding a hypervisor, a server virtualization machine (520), and multiple guest virtualization machines for execution within the central processor (175).

[0102] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0103] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0104] The image projection device (180h) includes an optical device (not shown) for image projection, and can output an augmented reality-based object under the control of a signal processing device (170).

[0105] For example, the video projection device (180h) can output vehicle speed information, vehicle direction information, front vehicle object, distance indicator from front vehicle, etc.

[0106] As another example, the image projection device (180h) can output an augmented reality rain carpet, an augmented reality route carpet, or an augmented reality dynamic carpet corresponding to the image of the road.

[0107] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100).

[0108] For example, the signal processing device (170) may include a shared memory (508), a central processor (175) that performs signal processing for a vehicle display (180a, 180b), and at least one neural processor (177).

[0109] The central processor (175) executes a hypervisor (505 in FIG. 5), and can execute a server virtualization machine (520) and multiple guest virtualization machines (530 to 540) on the executed hypervisor (505 in FIG. 5).

[0110] At this time, the first guest virtualization machine (530) can operate for the first display (180a), and the second guest virtualization machine (540) can operate for the second display (180b).

[0111] For example, the server virtualization machine (520) within the central processor (175) can receive, process, or process vehicle sensor data, location information data, camera data, audio data, or touch input data, and output them. By distinguishing between data processed only in the legacy virtualization machine and data processed in the server virtualization machine (520), data processing can be performed efficiently. In particular, by performing most of the data processing in the server virtualization machine (520), data sharing in a 1:N manner becomes possible.

[0112] As another example, the server virtualization machine (520) can directly receive and process CAN communication data, audio data, radio data, USB data, and wireless communication data for the first guest virtualization machine or the second guest virtualization machine (530 to 540).

[0113] And, the server virtualization machine (520) can transmit processed data to the first guest virtualization machine or the second guest virtualization machine (530 to 540).

[0114] Accordingly, among the server virtualization machine (520) and multiple guest virtualization machines (530-540), only the server virtualization machine (520) receives communication data and external input data and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling efficient implementation of interfaces for external input data and communication data. Furthermore, 1:N data communication becomes possible, enabling synchronization when sharing data.

[0115] Meanwhile, the server virtualization machine (520) can control the first guest virtualization machine (530) and the second guest virtualization machine (540) to share the same data by writing data to the shared memory (508).

[0116] For example, the server virtualization machine (520) can control the first guest virtualization machine (530) and the second guest virtualization machine (540) to share the same data by recording vehicle sensor data, the location information data, the camera data, or the touch input data in the shared memory (508). Accordingly, sharing of data in a 1:N manner becomes possible.

[0117] Ultimately, by performing most of the data processing on the server virtualization machine (520), data sharing in a 1:N manner becomes possible.

[0118] Meanwhile, the server virtualization machine (520) within the central processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transfer to the first guest virtualization machine (530) and the second guest virtualization machine (540).

[0119] That is, the server virtualization machine (520) within the central processor (175) can synchronize and transmit the same data to the first guest virtualization machine (530) and the second guest virtualization machine (540) using the shared memory (508) based on the hypervisor (505). Accordingly, the same image can be synchronized and displayed on multiple displays (180a to 180b) within the vehicle.

[0120] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0121] FIG. 4 is a diagram illustrating a system driven by a signal processing device related to the present disclosure.

[0122] Referring to the drawings, FIG. 4 is a drawing illustrating that virtual machines are used for each of the cluster display (180a) and the AVN display (180b).

[0123] The system (400) driven by the signal processing device of FIG. 4 exemplifies that a cluster virtualization machine (430) and an AVN virtualization machine (440) are executed on a hypervisor (405) within a central processor (175).

[0124] Meanwhile, the system (400) driven by the signal processing device of FIG. 4 exemplifies that a legacy virtualization machine (410) is also executed on a hypervisor (405) within the central processor (175).

[0125] The legacy virtualization machine (410) has an interface (412) for data communication with memory (140) and an interface (413) for Ethernet communication.

[0126] Meanwhile, the cluster virtualization machine (430) may be equipped with an interface (431) for CAN communication, an interface (432) for communication with an interface (412) of a legacy virtualization machine (410), and an interface (433) for communication with an interface (413) of a legacy virtualization machine (410).

[0127] Meanwhile, the AVN virtualization machine (440) may be equipped with an interface (441) for input / output of audio data, radio data, USB data, and wireless communication data, an interface (442) for communication with an interface (412) of a legacy virtualization machine (410), and an interface (443) for communication with an interface (413) of a legacy virtualization machine (410).

[0128] According to this system (400), CAN communication data is input / output only in the cluster virtualization machine (430), so there is a disadvantage in that CAN communication data cannot be utilized in the AVN virtualization machine (440).

[0129] In addition, according to the system (400) of FIG. 4, audio data, radio data, USB data, and wireless communication data are input / output only in the AVN virtualization machine (440), so there is a disadvantage in that such data cannot be utilized in the cluster virtualization machine (430).

[0130] Meanwhile, there is a disadvantage in that the cluster virtualization machine (430) and the cluster virtualization machine (430) must each have separate interfaces (431, 432, 441, 442) for memory data and Ethernet communication data input / output from the legacy virtualization machine (410).

[0131] Accordingly, the present disclosure proposes a method for improving the system of FIG. 4. That is, unlike FIG. 4, the virtual machine is divided into a server virtual machine (520) and a guest virtual machine, and various memory data, communication data, etc. are input / output from the server virtual machine (520) rather than from the guest virtual machine. This will be described with reference to FIG. 5 and below.

[0132] FIG. 5 is a diagram illustrating an example of a system driven by a signal processing device of the present disclosure.

[0133] Referring to the drawing, the system (500) of FIG. 5 illustrates that a server virtualization machine (520), which is a server virtualization machine (520), a first guest virtualization machine (530), which is a guest virtualization machine, and a second guest virtualization machine (540) are executed on a hypervisor (505) within a central processor (175) within a signal processing device (170).

[0134] The first guest virtualization machine (530) may be a virtualization machine for a cluster display (180a), and the second guest virtualization machine (540) may be a virtualization machine for an AVN display (180b).

[0135] That is, the first guest virtualization machine (530) and the second guest virtualization machine (540) can operate for image rendering of the cluster display (180a) and AVN display (180b), respectively.

[0136] Meanwhile, the system (500) driven by the signal processing device (170) of FIG. 5 exemplifies that a legacy virtualization machine (510) is also executed on a hypervisor (505) within the central processor (175).

[0137] The legacy virtualization machine (510) has an interface (511) for data communication with memory (140) and Ethernet communication.

[0138] Meanwhile, the legacy virtualization machine (510) may further include an interface (virtio-backend interface) (512) for data communication with the first and second guest virtualization machines (530, 540).

[0139] The server virtualization machine (520) may be equipped with an interface (521) for input / output of audio data, radio data, USB data, and wireless communication data, and an input / output server interface (522) for data communication with a guest virtualization machine.

[0140] That is, the server virtualization machine (520), which is a server virtualization machine (520), can provide I / Os that are difficult to virtualize by standard virtualization technology (VirtIO) to a plurality of guest virtualization machines, for example, the first and second guest virtualization machines (530, 540).

[0141] Meanwhile, the server virtualization machine (520), which is a server virtualization machine (520), can control radio data, audio data, etc. at the supervisor level and provide them to a number of guest virtualization machines, for example, the first and second guest virtualization machines (530, 540).

[0142] Meanwhile, the server virtualization machine (520), which is a server virtualization machine (520), processes vehicle data, sensor data, vehicle surrounding information, etc., and can provide the processed data or information, etc. to a plurality of guest virtualization machines, for example, first and second guest virtualization machines (530, 540).

[0143] Meanwhile, the server virtualization machine (520) can provide services (Supervisory Services), such as processing of vehicle data and management of audio routing.

[0144] Next, the first guest virtualization machine (530) may be equipped with an input / output client interface (532) for data communication with the server virtualization machine (520) and APIs (533) for controlling the input / output client interface (532).

[0145] Additionally, the first guest virtualization machine (530) may be equipped with an interface (virtio-backend interface) for data communication with the legacy virtualization machine (510).

[0146] The first guest virtualization machine (530) can receive memory data through communication with memory (140) and Ethernet data through Ethernet communication from the interface (virtio-backend interface) (512) of the legacy virtualization machine (510) through the interface (virtio-backend interface).

[0147] Next, the second guest virtualization machine (540) may be equipped with an input / output client interface (542) for data communication with the server virtualization machine (520) and APIs (543) for controlling the input / output client interface (542).

[0148] Additionally, the second guest virtualization machine (540) may be equipped with an interface (virtio-backend interface) for data communication with the legacy virtualization machine (510).

[0149] The second guest virtualization machine (540) can receive memory data through communication with memory (140) and Ethernet data through Ethernet communication from the interface (virtio-backend interface) (512) of the legacy virtualization machine (510) through the interface (virtio-backend interface).

[0150] Meanwhile, unlike FIG. 5, the legacy virtualization machine (510) may also be installed within the server virtualization machine (520).

[0151] According to the system (500), CAN communication data such as sensing data is input / output only in the server virtualization machine (520), but can be provided to multiple guest virtualization machines, for example, the first and second guest virtualization machines (530, 540), through data processing in the server virtualization machine (520). Therefore, 1:N data communication through processing in the server virtualization machine (520) becomes possible.

[0152] In addition, according to the system (500) of FIG. 5, audio data, radio data, USB data, and wireless communication data are input / output only in the server virtualization machine (520), but can be provided to multiple guest virtualization machines, for example, the first and second guest virtualization machines (530, 540), through data processing in the server virtualization machine (520). Therefore, 1:N data communication through processing in the server virtualization machine (520) becomes possible.

[0153] Meanwhile, in the system (500) of FIG. 5, the first and second guest virtualization machines (530, 540) can operate based on different OSs.

[0154] For example, the first guest virtualization machine (540) may operate under a Linux OS, and the second guest virtualization machine (540) may operate under a web OS.

[0155] The server virtualization machine (520) sets up a shared memory (508) based on a hypervisor (505) for data sharing, even though the first and second guest virtualization machines (530, 540) operate under different operating systems (OS). Accordingly, even though the first and second guest virtualization machines (530, 540) operate under different operating systems (OS), they can synchronize and share the same data or the same image. Consequently, the same data or the same image can be synchronized and displayed on multiple displays (180a, 180b).

[0156] FIG. 6 is a drawing for reference in the description of the operation of a system driven by the signal processing device of the present disclosure, and FIGS. 7a to 9b are drawings for reference in the description of FIG. 5 or FIG. 6.

[0157] First, according to the system (500) of FIG. 6, the central processor (175) in the signal processing device (170) executes a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) on a hypervisor (505) in the central processor (175), and the server virtualization machine (520) in the central processor (175) controls a shared memory (508) based on the hypervisor (505) to be set for data transmission to the first guest virtualization machine (530) and the second guest virtualization machine (540).

[0158] For example, the same image data, for example, can be transmitted from a server virtualization machine (520) to a first guest virtualization machine (530) and a second guest virtualization machine (540). Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0159] Meanwhile, according to the system (500) of FIG. 6, the central processor (175) in the signal processing device (170) executes a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) on a hypervisor (505) in the central processor (175), and the server virtualization machine (520) in the central processor (175) can synchronize and transmit the same data to the first guest virtualization machine (530) and the second guest virtualization machine (540) using a shared memory (508) based on the hypervisor (505).

[0160] For example, the same data may include CAN communication data, audio data, radio data, USB data, wireless communication data, location information data, or touch data. Accordingly, the same data can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0161] Meanwhile, the server virtualization machine (520) within the central processor (175) can receive location information data that changes depending on movement, process it, and provide it to the first guest virtualization machine (530) or the second guest virtualization machine (540). Accordingly, rather than 1:1 data communication between virtualization machines, 1:N data communication is possible using shared memory.

[0162] Meanwhile, the first guest virtualization machine (530) and the second guest virtualization machine (540) can be driven by different operating systems. Accordingly, even if multiple virtualization machines (520, 530, 540) are driven by different operating systems, high-speed data communication can be performed.

[0163] Meanwhile, although not shown in FIG. 6, the legacy virtualization machine (510) can transmit memory data from the memory (140) and Ethernet data via Ethernet communication to the first guest virtualization machine (530) and the second guest virtualization machine (540) in a synchronized manner using the shared memory (508) based on the hypervisor (505). In other words, 1:N data communication for memory data or Ethernet data can be performed. Accordingly, the same data can be transmitted in a synchronized manner.

[0164] Meanwhile, the server virtualization machine (520) within the central processor (175) can execute a system manager, a display manager, etc. as a supervisory service.

[0165] Meanwhile, the server virtualization machine (520) within the central processor (175) can execute vehicle information service, location information service, camera service, auto-sa, Bluetooth communication service, radio service, Wi-Fi service, audio service, touch service, etc. as systemic services.

[0166] FIG. 7a illustrates that three virtual machines (420, 420, 430) are operating in the system (400b) of FIG. 4.

[0167] Referring to the drawing, the server virtualization machine (420) is a Linux-based virtualization machine and may be equipped with an input / output server interface (422) for data transmission, and the first guest virtualization machine (530) and the second guest virtualization machine (540) may include input / output client interfaces (432, 442) for data communication with the input / output server interface (422).

[0168] For example, in order for a server virtualization machine (420) to transmit first data to a first guest virtualization machine (430), a first shared memory (408a) must be set within the hypervisor (405), and in order for the server virtualization machine (420) to transmit the same first data to a second guest virtualization machine (440), a second shared memory (408b) separate from the first shared memory (408a) must be set within the hypervisor (405).

[0169] When transmitting the same first data, as in Fig. 7a, if a separate shared memory is used, there is a disadvantage in that resources are wasted and synchronization is not easy.

[0170] FIG. 7b illustrates that, by the system (400b) of FIG. 7a, a first guest virtualization machine (430) displays image data received through a first shared memory (408a) on a first display (180a), and a second guest virtualization machine (440) displays image data received through a second shared memory (408b) on a second display (180b).

[0171] In FIG. 7b, the synchronization between the image (705a) displayed on the first display (180a) and the image (705b) displayed on the second display (180b) is not correct, and the image (705b) displayed on the second display (180b) corresponds to a previous frame than the image (705a) displayed on the first display (180a).

[0172] In this way, when the server virtualization machine (420) transmits the same image data, as in Fig. 7a, there is a disadvantage in that synchronization at the time of image display is not performed when a separate shared memory is used, as in Fig. 7b.

[0173] To address these issues, this disclosure proposes a method of allocating a single shared memory when transmitting identical data. This enables 1:N data communication and synchronized data transmission.

[0174] FIG. 8 illustrates a system (500) according to an embodiment of the present disclosure, in which a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) are executed on a hypervisor (505) within a central processor (175), and the server virtualization machine (520) within the central processor (175) controls a shared memory (508) based on the hypervisor (505) to be set for the same data transfer to a first guest virtualization machine (530) and a second guest virtualization machine (540).

[0175] Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0176] Meanwhile, high-speed data communication can be performed between multiple virtual machines (520, 530, 540). Furthermore, high-speed data communication can be performed even if multiple virtual machines (520, 530, 540) are driven by different operating systems.

[0177] Meanwhile, when the server virtualization machine (520) within the central processor (175) transmits data processed in the server virtualization machine (520) to another virtualization machine, it can utilize a single shared memory (508) rather than allocating a number of memories corresponding to the number of virtualization machines. Accordingly, rather than a 1:1 data communication between virtualization machines, a 1:N data communication is possible using the shared memory (508).

[0178] Meanwhile, the server virtualization machine (520) within the central processor (175) may include an input / output server interface (522) and a security manager (526).

[0179] Meanwhile, the first guest virtualization machine (530) and the second guest virtualization machine (540) may each include input / output client interfaces (532, 542). Accordingly, high-speed data communication between multiple virtualization machines (520, 530, 540) can be performed using the input / output server interface (522) and the input / output client interfaces (532, 542).

[0180] The input / output server interface (522) within the server virtualization machine (520) can receive the same data transmission request from each input / output client interface (532, 542) within the first guest virtualization machine (530) and the second guest virtualization machine (540), and based on this, can transmit the shared data to the shared memory (508) through the security manager (526).

[0181] Figure 9a illustrates a more detailed diagram of shared data transmission.

[0182] Referring to the drawing, for shared data transmission, the input / output server interface (522) within the server virtualization machine (520) transmits a request for allocation of shared memory (508) to the security manager (526) (S1).

[0183] Next, the security manager (526) can allocate shared memory (508) using the hypervisor (505) (S2) and record shared data in the shared memory (508).

[0184] Meanwhile, the input / output client interface (532, 542) can transmit a connection request to the input / output server interface (522) after allocating shared memory (508) (S3).

[0185] Meanwhile, after allocating shared memory (508), the input / output server interface (522) transmits information about the shared memory (508) containing key data to the input / output client interfaces (532, 542) (S4). The key data at this time may be data for data access.

[0186] Meanwhile, the server virtualization machine (520) within the central processor (175) can transmit information about the shared memory (508) to the first guest virtualization machine (530) and the second guest virtualization machine (540) after setting up the shared memory (508).

[0187] The input / output client interface (532, 542) can access the shared memory (508) based on the received key data (S5) and copy the shared data from the shared memory (508).

[0188] Accordingly, the shared memory (508) becomes accessible to the first guest virtualization machine (530) and the second guest virtualization machine (540), and ultimately, shared data can be shared.

[0189] For example, if the shared data is image data, the image data is shared between the first guest virtualization machine (530) and the second guest virtualization machine (540), and ultimately, the same shared image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0190] FIG. 9b illustrates that, by the system (500b) of FIG. 9a, a first guest virtualization machine (530) displays image data received through shared memory (508) on a first display (180a), and a second guest virtualization machine (540) displays image data received through shared memory (508) on a second display (180b).

[0191] In Fig. 9b, synchronization is performed between an image (905) displayed on a first display (180a) and an image (905) displayed on a second display (180b), thereby displaying the same image.

[0192] That is, the image data processed in the server virtualization machine (520) within the central processor (175) is transmitted to the first guest virtualization machine (530) and the second guest virtualization machine (540) via the shared memory (508), and based on the image data, the first image (905) displayed on the first display (180a) and the second image (905) displayed on the second display (180b) may be identical. Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle. In addition, high-speed data communication can be performed between multiple virtualization machines (520, 530, 540).

[0193] FIG. 10a is an example of an internal block diagram of a signal processing device related to the present disclosure.

[0194] Referring to the drawing, a signal processing device (170x) related to the present disclosure may include a central processor (175), a graphic processor (178), and a plurality of neural processors (177a to 177c).

[0195] A signal processing device (170x) related to the present disclosure can receive data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (not shown), and perform signal processing using at least one of a central processor (175), a graphics processor (178), and a plurality of neural processors (177a to 177c).

[0196] Meanwhile, the sensor device (700) can continuously output sensor data to the signal processing device (170x) during vehicle operation.

[0197] The sensor data at this time may include data from sensor devices (700) of various vehicles.

[0198] Meanwhile, the camera device (195) can continuously output camera data to the signal processing device (170x) during vehicle operation.

[0199] Meanwhile, Lidar (not shown) can continuously output Lidar data to a signal processing device (170x) while the vehicle is in motion.

[0200] Figure 10b is a drawing referenced in the description of Figure 10a.

[0201] Referring to the drawing, the signal processing device (170x) of FIG. 10a continuously receives data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (not shown), and performs various signal processing based on the received data.

[0202] FIG. 10b (a) illustrates that the signal processing device (170x) of FIG. 10a receives camera data from the camera device (185) and, based on the received camera data, performs vehicle detection (front), vehicle detection (back), pedestrian detection, static object detection, traffic signal detection, overhead sign recognition, road marking detection, road construction detection, lane detection, crossroad detection, toll gate recognition, etc.

[0203] Meanwhile, (a) of FIG. 10b illustrates that the signal processing device (170x) of FIG. 10a receives lidar data from a lidar device (not shown) and performs vehicle detection (front), vehicle detection (back), pedestrian detection, curve detection, drivable space localization, road surface recognition, etc. based on the received lidar data.

[0204] Meanwhile, in the drawing, a plurality of neural processors (177a to 177c) within the signal processing device (170x) of FIG. 10a perform various detections or recognitions, etc. described above based on camera data or rider data, and in particular, continuously perform various detections or recognitions, etc. described above regardless of the vehicle driving state.

[0205] That is, in the drawing, a plurality of neural processors (177a to 177c) within the signal processing device (170x) of FIG. 10a perform various detections or recognitions, etc. described above, based on camera data or lidar data, regardless of whether the vehicle is driving at low speed, driving in traffic congestion, stopped, driving on a high-speed road, driving at high speed, or parked.

[0206] Meanwhile, when a plurality of neural processors (177a to 177c) in a signal processing device (170x) of FIG. 10a operate at a fixed frame rate regardless of the vehicle driving status such as the vehicle speed, there is a disadvantage that the processing load (RSxa, RSxb, RSxc) of the plurality of neural processors (177a to 177c) is considerable, as shown in (b) of FIG. 10b. Furthermore, there is a disadvantage that power consumption is considerable due to the processing load (RSxa, RSxb, RSxc) of the plurality of neural processors (177a to 177c).

[0207] In this disclosure, a method for efficiently operating a neural processor is proposed. This is described with reference to FIG. 11a and below.

[0208] FIG. 11A is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0209] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a central processor (175) and at least one neural processor (177a to 177c).

[0210] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may further include a graphics processor (178).

[0211] Meanwhile, a system (1100) driven by a signal processing device (170) according to an embodiment of the present disclosure executes a plurality of virtual machines (520 to 530) on a hypervisor (505).

[0212] Specifically, a central processor (175) in a signal processing device (170) according to an embodiment of the present disclosure executes a hypervisor (505) and executes a plurality of virtual machines (520 to 530) on the hypervisor (505).

[0213] Among the multiple virtualization machines (520 to 540), the first virtualization machine (520) controls the operation of the neural processor (177).

[0214] A signal processing device (170) according to an embodiment of the present disclosure may receive data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (not shown), and perform signal processing using at least one of a central processor (175), a graphics processor (178), and a plurality of neural processors (177a to 177c).

[0215] Meanwhile, the sensor device (700) can continuously output sensor data to the signal processing device (170) during vehicle operation.

[0216] The sensor data at this time is data from sensor devices (700) of various vehicles, and may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle internal temperature data, and vehicle internal humidity data.

[0217] Meanwhile, the camera device (195) can continuously output camera data to the signal processing device (170) during vehicle operation.

[0218] Meanwhile, Lidar (not shown) can continuously output Lidar data to a signal processing device (170) while the vehicle is in motion.

[0219] Among the plurality of virtual machines (520-540), the first virtual machine (520) controls the neural processor (177) to operate at a variable frame rate or output result data based on sensor data from a sensor device (700) within the vehicle. Accordingly, the neural processor (177) can be operated efficiently. In addition, power consumption can be reduced.

[0220] Meanwhile, the neural processor (177) can detect vehicle driving information based on sensor data, operate at a variable frame rate based on the vehicle driving information, or output result data including the vehicle driving information. Accordingly, the neural processor (177) can be efficiently operated based on the vehicle driving information.

[0221] Meanwhile, the neural processor (177) can receive sensor data at a fixed frame rate, detect vehicle driving information based on the sensor data, and operate at a variable frame rate based on the vehicle driving information or output result data including the vehicle driving information. Accordingly, the neural processor (177) can be efficiently operated based on the vehicle driving information.

[0222] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a shared memory (508).

[0223] In the drawing, a hypervisor (505) is executed on a central processor (175), and a model container (509) and a shared memory (508) are executed within the hypervisor (505).

[0224] The model container (509) can perform interface of model parameters related to the operation of the neural processor (177) and version management of learning files.

[0225] Meanwhile, unlike the drawing, a shared memory (508) may be placed within the signal processing device (170).

[0226] The central processor (175) can transmit sensor data to the neural processor (177) using the shared memory (508), and the neural processor (177) can transmit result data to the central processor (175) using the shared memory (508). Accordingly, the neural processor (177) can be operated efficiently. In addition, power consumption can be reduced.

[0227] Meanwhile, the frame rate of sensor data recorded in the shared memory (508) is fixed, while the frame rate of result data recorded in the shared memory (508) may be variable. Accordingly, the neural processor (177) can be operated efficiently. In addition, power consumption can be reduced.

[0228] Meanwhile, the first virtual machine (520) can receive vehicle driving information detected by the neural processor (177) and execute a vehicle driver assistance engine (Nad) or autonomous driving engine based on the vehicle driving information. Accordingly, vehicle driving information based on sensor data can be utilized in various ways.

[0229] Meanwhile, the first virtual machine (520) can execute a neural system service (1110) to control at least one neural processor (177a to 177c).

[0230] The neural system service (1110) may execute or include a neural manager (1113) for managing at least one neural processor (177a to 177c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (177a to 177c), and a neural interface (1118) for interfacing with at least one neural processor (177a to 177c).

[0231] The neural manager (1113) can perform artificial intelligence model management, learning model management, camera data management, sensor data management, or command queue management.

[0232] The neural controller (1115) can determine an optimal inference method of at least one neural processor (177a to 177c), perform queue, partition, caching, or scalable coding, or control at least one neural processor (177a to 177c).

[0233] The neural interface (1118) can execute an application program interface (API) related to the accelerator of at least one neural processor (177a-177c).

[0234] Meanwhile, the interface (522) within the first virtual machine (520) can perform interfacing between the neural system service (1110) and the model container (509) or the neural system service (1110) and the shared memory (508).

[0235] In addition, the interface (522) within the first virtual machine (520) can perform interfacing with the vehicle driving assistance engine (Nad) or autonomous driving engine running within the first virtual machine (520) and the neural system service (1110).

[0236] Meanwhile, the interface (522) within the first virtual machine (520) can perform interfacing to another virtual machine (530).

[0237] For example, an interface (522) within a first virtual machine (520) can be controlled to transmit sensor data to a neural processor (177) using shared memory (508).

[0238] Meanwhile, the interface (522) within the first virtual machine (520) can control the transmission of result data output from the neural processor (177) and recorded in the shared memory (508) to the neural system service (1110).

[0239] Meanwhile, the interface (522) within the first virtual machine (520) can control the transmission of result data output from the neural processor (177) and recorded in the shared memory (508) to a vehicle driving assistance engine (Nad) or an autonomous driving engine running within the first virtual machine (520).

[0240] Meanwhile, the interface (522) within the first virtual machine (520) can be controlled to transmit the result data output from the neural processor (177) and recorded in the shared memory (508) to the second virtual machine (530), etc.

[0241] Meanwhile, the first virtual machine (520) of FIG. 11a may be a server virtualization machine, and the second virtualization machine (530) may be a guest virtualization machine.

[0242] FIGS. 11b to 11d are diagrams illustrating various examples of systems driven by a signal processing device according to an embodiment of the present disclosure.

[0243] FIG. 11b is a diagram illustrating another example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0244] Referring to the drawing, a system (1100b) driven by a signal processing device (170) according to an embodiment of the present disclosure is similar to the system (1100) of FIG. 11a, but differs in that a vehicle driving assistance engine (Nad) or an autonomous driving engine is executed in a second virtual machine (530), not a first virtual machine (520).

[0245] That is, on the other hand, among the plurality of virtual machines (520-540), the second virtual machine (530) can receive result data including vehicle driving information from the first virtual machine (520) via the shared memory (508) and execute a vehicle driving assistance engine (Nad) or an autonomous driving engine based on the vehicle driving information. Accordingly, vehicle driving information based on sensor data can be utilized in various ways.

[0246] FIG. 11C is a diagram illustrating another example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0247] Referring to the drawing, a system (1100c) driven by a signal processing device (170) according to an embodiment of the present disclosure is similar to the system (1100) of FIG. 11A, but differs in that a first virtual machine (520), a second virtual machine (530), and a third virtual machine (540) are executed, a vehicle driving assistance engine (Nad) or an autonomous driving engine is executed within the second virtual machine (530), and an augmented reality engine (Nar) or a rotation monitoring engine (Ndm) is executed within the third virtual machine (540).

[0248] That is, among the plurality of virtual machines (520 to 540), the second virtual machine (530) can receive result data including vehicle driving information from the first virtual machine (520) via the shared memory (508) and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information.

[0249] Meanwhile, among the multiple virtual machines (520-540), the third virtual machine (540) can receive result data including vehicle driving information from the first virtual machine (520) via shared memory (508) and execute an augmented reality engine based on the vehicle driving information. Accordingly, vehicle driving information based on sensor data can be utilized in various ways.

[0250] Meanwhile, as shown in the drawing, the operating systems (OS) of the first virtual machine (520), the second virtual machine (530), and the third virtual machine (540) may be different.

[0251] FIG. 11d is a diagram illustrating another example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0252] Referring to the drawing, a system (1100d) driven by a signal processing device (170) according to an embodiment of the present disclosure executes a first virtualization machine (520), a second virtualization machine (530), and a third virtualization machine (540) similarly to the system (1100c) of FIG. 11c.

[0253] The second virtual machine (530) can execute a vehicle driving assistance engine (Nad) or an autonomous driving engine based on vehicle driving information, and the third virtual machine (540) can execute an augmented reality engine (Nar) or a rotation monitoring engine (Ndm) based on vehicle driving information.

[0254] Meanwhile, the second virtualization machine (530) can operate for the first display (180a), and the third virtualization machine (540) can operate for the second display (180b). Accordingly, multiple displays can be efficiently controlled through the virtualization machines.

[0255] Meanwhile, the third virtual machine (540) can operate not only for the second display (180b), but also for the image projection device (180h). Accordingly, the image projection device can be efficiently controlled through the virtual machine.

[0256] FIG. 12 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0257] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure receives sensor data from a sensor device (700) (S1210).

[0258] For example, a central processor (175) in a signal processing device (170) according to an embodiment of the present disclosure may control sensor data to be written to a shared memory (508).

[0259] Meanwhile, the central processor (175) in the signal processing device (170) according to the embodiment of the present disclosure can control at least some of the neural processors (177sa to 177c) to operate for processing sensor data (S1220).

[0260] For example, a central processor (175) in a signal processing device (170) can control a first neural processor (177c) among a plurality of neural processors (177a to 177c) to operate to process sensor data.

[0261] Next, the neural processor (177) can detect vehicle driving information based on the received sensor data (S1230).

[0262] For example, the first neural processor (177c) can detect vehicle driving information including speed information or stop information from wheel speed sensor data recorded in the shared memory (508).

[0263] As another example, the first neural processor (177c) can detect vehicle driving information including vehicle forward information or vehicle backward information from vehicle forward data or vehicle backward data recorded in the shared memory (508).

[0264] As another example, the first neural processor (177c) can detect vehicle driving information, such as lane change information, vehicle right turn information, or vehicle left turn information, from vehicle direction data, vehicle angle data, or vehicle inclination data recorded in the shared memory (508).

[0265] Meanwhile, vehicle driving information detected by the neural processor (177) can be transmitted to the first virtual machine (520) within the central processor (175) via the shared memory (508).

[0266] Meanwhile, the first virtual machine (520) within the central processor (175) can control the neural processor (177) to operate at a variable frame rate or output result data at a variable frame rate based on vehicle driving information (S1240).

[0267] Accordingly, the neural processor (177) can be efficiently operated based on vehicle driving information. Furthermore, the power consumption of the neural processor (177) can be reduced.

[0268] For example, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a first frame rate when the vehicle (200) moves forward, and can control the neural processor (177) to operate at a second frame rate lower than the first frame rate when the vehicle (200) moves backward. Accordingly, when the vehicle (200) moves backward, the neural processor (177) can be efficiently operated based on vehicle driving information. Furthermore, the power consumption of the neural processor (177) can be reduced.

[0269] As another example, among the plurality of virtual machines (520 to 540), the first virtual machine (520) may control the neural processor (177) to operate at a first frame rate when the vehicle (200) is driven at a first speed, and may control the neural processor (177) to operate at a third frame rate that is greater than the first frame rate when the vehicle (200) is driven at a second speed that is faster than the first speed. Accordingly, the neural processor (177) can be efficiently operated based on the speed of the vehicle (200).

[0270] As another example, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a first frame rate when the vehicle (200) is driving straight, and control the neural processor (177) to operate at a third frame rate that is greater than the first frame rate when the vehicle (200) changes lanes or turns left or right. Accordingly, when the vehicle (200) changes lanes, the neural processor (177) can be efficiently operated based on vehicle driving information.

[0271] As another example, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a first frame rate when the vehicle (200) is in motion, and can control the neural processor (177) to operate at a second frame rate lower than the first frame rate when the vehicle (200) is stopped. Accordingly, when the vehicle (200) is stopped, the neural processor (177) can be efficiently operated based on vehicle driving information. Furthermore, the power consumption of the neural processor (177) can be reduced.

[0272] FIGS. 13A and 13B are various examples of flowcharts showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0273] FIG. 13a is another example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0274] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure receives sensor data from a sensor device (700) (S1210). Step 1210 (S1210) may correspond to step 1210 (S1210) of FIG. 12.

[0275] Next, the neural processor (177) can detect vehicle driving information based on the received sensor data (S1230). Step 1230 (S1230) may correspond to step 1230 (S1230) of FIG. 12.

[0276] Next, the neural processor (177) can operate at a first frame rate for a first period or output result data (S1242).

[0277] Next, the neural processor (177) can operate at a second frame rate or output result data during a second period after the first period (S1245).

[0278] For example, among the plurality of virtual machines (520 to 540), the first virtual machine (520) may control the neural processor (177) to operate at a first frame rate or output result data based on sensor data during a first period, and may control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data based on vehicle driving information based on the sensor data during a second period after the first period. Accordingly, the neural processor (177) may be operated efficiently.

[0279] FIG. 13b is another example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0280] Referring to the drawing, a signal processing device (170) according to an embodiment of the present disclosure receives sensor data from a sensor device (700) (S1210). Step 1210 (S1210) may correspond to step 1210 (S1210) of FIG. 12.

[0281] Next, the central processor (175) within the signal processing device (170) determines whether it is in energy saving mode (S1235), and if the energy saving mode is turned off, the neural processor (177) can be controlled to operate at a fixed frame rate or output result data based on the sensor data (S1250).

[0282] Specifically, among the plurality of virtual machines (520 to 540), when the energy saving mode is turned off, the first virtual machine (520) can control the neural processor (177) to operate at a fixed frame rate or output result data based on sensor data.

[0283] Meanwhile, in step 1235 (S1235), when the energy saving mode is turned on, the neural processor (177) can be controlled to operate at a variable frame rate or output result data based on vehicle driving information based on sensor data (S1255).

[0284] Specifically, among the plurality of virtual machines (520-540), when the energy saving mode is turned on, the first virtual machine (520) can control the neural processor (177) to operate at a variable frame rate or output result data based on vehicle driving information based on sensor data. Accordingly, the neural processor (177) can be efficiently operated based on the vehicle driving information. In addition, power consumption can be reduced.

[0285] Figures 14a to 19 are drawings referenced in the description of Figures 12 to 13b.

[0286] FIG. 14a illustrates that sensor data is received by a central processor (175) within a signal processing device (170), and the central processor (175) transmits the sensor data (Sda) to a first neural processor (177a).

[0287] Accordingly, the first neural processor (177a) can receive sensor data at a fixed frame rate and detect vehicle driving information based on the sensor data (Sda).

[0288] At this time, the first neural processor (177a) can vary the frame rate based on vehicle driving information.

[0289] In the drawing, the first neural processor (177a) is illustrated as operating at a variable frame rate FRm.

[0290] In addition, the first neural processor (177a) can transmit result data (Srt) including vehicle driving information to the central processor (175) based on the variable frame rate FRm. Accordingly, the first neural processor (177a) can be efficiently operated based on the vehicle driving information.

[0291] FIG. 14b illustrates that sensor data is received by a central processor (175) within a signal processing device (170), and the central processor (175) transmits the sensor data (Sda) to a first neural processor (177a) via a shared memory (508).

[0292] Accordingly, the shared memory (508) and the first neural processor (177a) can each receive sensor data (Sda) at a fixed frame rate.

[0293] And, the first neural processor (177a) can detect vehicle driving information based on sensor data (Sda).

[0294] At this time, the first neural processor (177a) can vary the frame rate based on vehicle driving information.

[0295] In the drawing, the first neural processor (177a) is illustrated as operating at a variable frame rate FRm.

[0296] In addition, the first neural processor (177a) can transmit result data (Srt) including vehicle driving information to the central processor (175) via the shared memory (508) based on the variable frame rate FRm. Accordingly, the first neural processor (177a) can be efficiently operated based on the vehicle driving information.

[0297] FIG. 14c illustrates sensor data being received by the first neural processor (177a) within the signal processing device (170).

[0298] Accordingly, the first neural processor (177a) can receive sensor data at a fixed frame rate and detect vehicle driving information based on the sensor data (Sda).

[0299] At this time, the first neural processor (177a) can vary the frame rate based on vehicle driving information.

[0300] In the drawing, the first neural processor (177a) is illustrated as operating at a variable frame rate FRm.

[0301] In addition, the first neural processor (177a) can transmit result data (Srt) including vehicle driving information to the central processor (175) based on the variable frame rate FRm. Accordingly, the first neural processor (177a) can be efficiently operated based on the vehicle driving information.

[0302] FIG. 14d illustrates sensor data being received by the first neural processor (177a) within the signal processing device (170).

[0303] Accordingly, the shared memory (508) can receive sensor data (Sda) at a fixed frame rate.

[0304] And, the first neural processor (177a) can detect vehicle driving information based on sensor data (Sda).

[0305] At this time, the first neural processor (177a) can vary the frame rate based on vehicle driving information.

[0306] In the drawing, the first neural processor (177a) is illustrated as operating at a variable frame rate FRm.

[0307] In addition, the first neural processor (177a) can transmit result data (Srt) including vehicle driving information to the central processor (175) via the shared memory (508) based on the variable frame rate FRm. Accordingly, the first neural processor (177a) can be efficiently operated based on the vehicle driving information.

[0308] FIG. 15a illustrates an example of the operation of a first neural processor (177a) among a plurality of neural processors in a signal processing device (170).

[0309] Referring to the drawing, the first neural processor (177a) can operate at a fixed frame rate of Pf during the periods Pa, Pb, and Pc.

[0310] FIG. 15b illustrates another example of the operation of a first neural processor (177a) among a plurality of neural processors in a signal processing device (170).

[0311] Referring to the drawing, the first neural processor (177a) can operate at a fixed frame rate Pf during a first period Pa, a variable frame rate Pa during a second period Pb, and a different variable frame rate Pb1 during a third period Pc.

[0312] FIG. 15c illustrates another example of the operation of a first neural processor (177a) among a plurality of neural processors in a signal processing device (170).

[0313] Referring to the drawing, the first neural processor (177a) can operate at a fixed frame rate Pf during a first period Pa, then operate at a variable frame rate Pa, and then operate at another variable frame rate Pb1.

[0314] At this time, unlike in Fig. 15b, Pf, Pa, and Pb1 can operate continuously without a rest period. After that, the PC can have a rest period.

[0315] Figure 16a illustrates a case where the vehicle moves forward and a case where the vehicle moves backward.

[0316] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at the first frame rate or output result data when the vehicle (200) moves forward (DRa), as shown in (a) of FIG. 16a.

[0317] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data, when the vehicle (200) moves backward, as shown in (b) of FIG. 16a.

[0318] At this time, the second frame rate may be lower than the first frame rate. Accordingly, when the vehicle (200) is moving backward (DRb), the neural processor (177) can be efficiently operated based on vehicle driving information, thereby reducing power consumption.

[0319] Figure 16b illustrates a case where the vehicle speed is variable.

[0320] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a first frame rate or output result data when the vehicle (200) is driven at a first speed (V1), as shown in (a) of FIG. 16b.

[0321] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, which is different from the first frame rate, or to output result data, when the vehicle (200) is driven at a second speed (V2) that is faster than the first speed (V1), as shown in (b) of FIG. 16b.

[0322] At this time, the second frame rate may be higher than the first frame rate. Accordingly, when the vehicle (200) is driven at a second, faster speed, the neural processor (177) can be efficiently operated based on vehicle driving information.

[0323] Figure 16c illustrates a case where the vehicle's driving lane is variable.

[0324] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at the first frame rate or output result data when the vehicle (200) is driving straight, as shown in (a) of FIG. 16c.

[0325] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data, when the vehicle (200) changes lanes, as shown in (b) of FIG. 16c.

[0326] At this time, the second frame rate may be greater than the first frame rate. Accordingly, when the vehicle (200) changes lanes, the neural processor (177) can be efficiently operated based on vehicle driving information.

[0327] Figure 16d illustrates a case where a vehicle turns right.

[0328] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at the first frame rate or output result data when the vehicle (200) is driving straight, as shown in (a) of FIG. 16d.

[0329] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data, when the vehicle (200) is driving in a right turn, as shown in (b) of FIG. 16d.

[0330] At this time, the second frame rate may be greater than the first frame rate. Accordingly, when the vehicle (200) is driving in a right turn, the neural processor (177) can be efficiently operated based on vehicle driving information.

[0331] Figure 16e illustrates a case where a vehicle turns left.

[0332] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at the first frame rate or output result data when the vehicle (200) is driving straight, as shown in (a) of FIG. 16e.

[0333] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data, when the vehicle (200) is driving in a left turn, as shown in (b) of FIG. 16e.

[0334] At this time, the second frame rate may be greater than the first frame rate. Accordingly, when the vehicle (200) is driving in a left turn, the neural processor (177) can be efficiently operated based on vehicle driving information.

[0335] Figure 16f illustrates a case where a vehicle stops while driving.

[0336] Referring to the drawing, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at the first frame rate or output result data when the vehicle (200) is driving, as shown in (a) of FIG. 16f.

[0337] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the neural processor (177) to operate at a second frame rate, different from the first frame rate, or output result data when the vehicle (200) stops, as shown in (b) of FIG. 16f.

[0338] At this time, the second frame rate may be lower than the first frame rate. Accordingly, when the vehicle (200) is driving in a left turn, the neural processor (177) can be efficiently operated based on vehicle driving information, while reducing power consumption.

[0339] FIG. 17 is a drawing for reference in explaining the operation of the first and second neural processors (177a, 177b) among a plurality of neural processors (177a to 177c) of a signal processing device (170) according to an embodiment of the present disclosure.

[0340] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure of FIGS. 11A to 11D continuously receives data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (not shown), and performs various signal processing based on the received data.

[0341] FIG. 17 (a) illustrates that among a plurality of neural processors (177a to 177c) in a signal processing device (170) according to an embodiment of the present disclosure, the first and second neural processors (177a, 177b) receive camera data from a camera device (185) and, based on the received camera data, perform vehicle detection (front), vehicle detection (back), pedestrian detection, static object detection, traffic signal detection, overhead sign recognition, road marking detection, road construction detection, lane detection, crossroad detection, toll gate recognition, etc.

[0342] Meanwhile, (a) of FIG. 17 illustrates that among a plurality of neural processors (177a to 177c) in a signal processing device (170) according to an embodiment of the present disclosure, the first and second neural processors (177a, 177b) receive lidar data from a lidar device (not shown) and perform vehicle detection (front), vehicle detection (back), pedestrian detection, curve detection, drivable space localization, road surface recognition, etc. based on the received lidar data.

[0343] Meanwhile, in the drawing, among the plurality of neural processors (177a to 177c) in the signal processing device (170) according to the embodiment of the present disclosure, the first and second neural processors (177a, 177b) perform various detections or recognitions, etc. described above, based on camera data or rider data, and in particular, can operate at a variable frame rate or output result data based on vehicle driving information.

[0344] In the drawing, it is exemplified that during low-speed driving of a vehicle (200), the frame rate for toll gate recognition is lowered based on camera data.

[0345] Meanwhile, the drawing illustrates that the frame rate of road marking detection, road construction detection, lane detection, crossroad detection, toll gate recognition, etc. is lowered based on camera data during traffic congestion or stoppage of a vehicle (200).

[0346] Meanwhile, the drawing illustrates that the frame rate of pedestrian detection is lowered based on camera data while the vehicle (200) is driving on a high-speed road or while driving at high speed.

[0347] Meanwhile, in the drawing, when a vehicle (200) is parked, the frame rate for traffic signal detection, overhead sign recognition, road marking detection, road construction detection, lane detection, crossroad detection, toll gate recognition, etc. is exemplified as being the lowest based on camera data. Accordingly, the neural processor (177) can be efficiently operated based on vehicle driving information based on camera data.

[0348] Meanwhile, the drawing illustrates that the frame rate for vehicle detection - front, vehicle detection - back, road surface recognition, etc. is lowered based on lidar data during low-speed driving or traffic congestion of a vehicle (200).

[0349] Meanwhile, the drawing illustrates that the frame rate of pedestrian detection is lowered based on lidar data during high-speed road driving or high-speed driving of a vehicle (200).

[0350] Meanwhile, the drawing illustrates that when a vehicle (200) is parked, the frame rate for vehicle detection (vehicle detection - front), vehicle detection (vehicle detection - back), etc. is lowered based on lidar data. Accordingly, the neural processor (177) can be efficiently operated based on vehicle driving information based on lidar data.

[0351] Meanwhile, in (a) of Fig. 17, since only the first and second neural processors (177a, 177b) among the plurality of neural processors (177a to 177c) operate, as in (b) of Fig. 17, the processing loads (RSa, RSb) of the first and second neural processors (177a, 177b) among the plurality of neural processors (177a to 177c) are considerable, but there is an advantage in that the processing load (RSc) of the third neural processor (177c) is significantly reduced. Accordingly, the power consumption of the third neural processor (177c) can be significantly reduced.

[0352] Figure 18 is a diagram illustrating that the frame rate of a neural processor is variable.

[0353] Referring to the drawing, as in (a) of FIG. 18, the frame rate of the central processor (175) and the first to second neural processors (177a, 177b) in the signal processing device (170) may be 30 or 40 fps.

[0354] Meanwhile, as shown in (b) of FIG. 18, the frame rate of the central processor (175) and the first neural processor (177a) in the signal processing device (170) may be 10 or 20 fps (frames per second).

[0355] Compared to (a) of Fig. 18, according to (b) of Fig. 18, the second neural processor (177b) may not operate, thereby reducing power consumption. Furthermore, by operating the neural processor and the like at a variable frame rate, the neural processor can be operated efficiently.

[0356] Meanwhile, (a) of FIG. 18 may be the operation of the central processor (175) and the first and second neural processors (177a, 177b) when the vehicle (200) moves forward, as in (a) of FIG. 16a, and (b) of FIG. 18 may be the operation of the central processor (175) and the first neural processor (177a) when the vehicle (200) moves backward, as in (b) of FIG. 16a.

[0357] Alternatively, (a) of FIG. 18 may be the operation of the central processor (175) and the first and second neural processors (177a, 177b) when the vehicle (200) is driven at a second speed (V2), as in (b) of FIG. 16b, and (b) of FIG. 18 may be the operation of the central processor (175) and the first neural processor (177a) when the vehicle (200) is driven at a first speed (V1), as in (a) of FIG. 16b.

[0358] Alternatively, (a) of FIG. 18 may be the operation of the central processor (175) and the first and second neural processors (177a, 177b) when the vehicle (200) is driving straight, as in (a) of FIG. 16c, and (b) of FIG. 18 may be the operation of the central processor (175) and the first neural processor (177a) when the vehicle (200) is driving by changing lanes, as in (b) of FIG. 16c.

[0359] Alternatively, (a) of FIG. 18 may be the operation of the central processor (175) and the first and second neural processors (177a, 177b) when the vehicle (200) is driving straight, as in (a) of FIG. 16d or (a) of FIG. 16e, and (b) of FIG. 18 may be the operation of the central processor (175) and the first neural processor (177a) when the vehicle (200) is driving in a right or left turn, as in (b) of FIG. 16d or (b) of FIG. 16e.

[0360] Alternatively, (a) of FIG. 18 may be the operation of the central processor (175) and the first and second neural processors (177a, 177b) when the vehicle (200) is stopped, as in (b) of FIG. 16f, and (b) of FIG. 18 may be the operation of the central processor (175) and the first neural processor (177a) when the vehicle (200) is moving, as in (a) of FIG. 16f.

[0361] Figure 19 is a diagram comparing fixed frame rate driving and variable frame rate driving of a neural processor (177).

[0362] According to (a) of Fig. 19, the neural processor (177) operates at a fixed frame rate of 30 fps when performing crossroad detection, toll gate recognition, and pedestrian detection operations based on camera data. Accordingly, the processing load of the neural processor (177) is considerable, and since it operates at a fixed frame rate regardless of vehicle driving information, there is a disadvantage in that power consumption is considerable.

[0363] According to (b) of FIG. 19, the neural processor (177) can operate at a variable frame rate according to vehicle driving information when performing crossroad detection, toll gate recognition, and pedestrian detection based on camera data.

[0364] For example, in the event of a traffic jam, the neural processor (177) can turn off toll gate recognition. This allows the neural processor (177) to operate efficiently. Additionally, power consumption can be reduced.

[0365] As another example, when driving on a highway, the neural processor (177) can turn off crossroad detection. This allows the neural processor (177) to operate efficiently. Furthermore, power consumption can be reduced.

[0366] As another example, when driving on a highway, the neural processor (177) can turn off pedestrian detection or operate at the lowest frame rate of 5 fps. This allows the neural processor (177) to operate efficiently. Furthermore, power consumption can be reduced.

[0367] Meanwhile, separately from what was described in FIGS. 11a to 19, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the first neural processor (177a) among at least one neural processor (177) to operate at a variable frame rate or output result data based on sensor data, and can control the second neural processor (177) among at least one neural processor (177) to operate at a variable frame rate or output result data based on camera data from a camera device (195) in the vehicle. Accordingly, the neural processor (177) can be efficiently operated based on vehicle driving information. In addition, power consumption can be reduced.

[0368] Meanwhile, among the plurality of virtual machines (520 to 540), the first virtual machine (520) can control the first neural processor (177a) to operate at a first frame rate or output result data during a first period, and can control the second neural processor (177b) to operate at a second frame rate or output result data during the first period. Accordingly, the neural processor (177) can be efficiently operated based on vehicle driving information. In addition, power consumption can be reduced.

[0369] Meanwhile, although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. At least one neural processor; A central processor that runs a hypervisor; The central processor, Running multiple virtual machines on the hypervisor, Among the above multiple virtual machines, the first virtual machine is: A signal processing device that controls the neural processor to operate at a variable frame rate or output result data based on sensor data from a sensor device within a vehicle.

2. In paragraph 1, The above neural processor, A signal processing device that detects vehicle driving information based on the sensor data, and operates at a variable frame rate based on the vehicle driving information or outputs the result data including the vehicle driving information.

3. In paragraph 1, The above neural processor, A signal processing device that receives the sensor data at a fixed frame rate, detects vehicle driving information based on the sensor data, and operates at a variable frame rate based on the vehicle driving information or outputs the result data including the vehicle driving information.

4. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: During the first period, based on the sensor data, the neural processor is controlled to operate at a first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate different from the first frame rate or to output the result data based on vehicle driving information based on the sensor data during a second period after the first period.

5. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the energy saving mode is turned off, the neural processor is controlled to operate at a fixed frame rate or output the result data based on the sensor data, A signal processing device that controls the neural processor to operate at a variable frame rate or output the result data based on vehicle driving information based on the sensor data when the energy saving mode is turned on.

6. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the vehicle moves forward, the neural processor is controlled to operate at a first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate, different from the first frame rate, or to output the result data based on vehicle driving information based on the sensor data when the vehicle is moving backward.

7. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the vehicle is driving at a first speed, the neural processor is controlled to operate at a first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate different from the first frame rate or to output the result data based on vehicle driving information based on the sensor data when the vehicle is driving at a second speed that is faster than the first speed.

8. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the vehicle is driving straight, the neural processor is controlled to operate at the first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate different from the first frame rate or to output the result data based on vehicle driving information based on the sensor data when the vehicle changes lanes.

9. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the vehicle is driving straight, the neural processor is controlled to operate at the first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate different from the first frame rate or to output the result data based on vehicle driving information based on the sensor data when the vehicle is driving in a left or right turn.

10. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: When the vehicle is driving, the neural processor is controlled to operate at a first frame rate or output the result data, A signal processing device that controls the neural processor to operate at a second frame rate different from the first frame rate or to output the result data based on vehicle driving information based on the sensor data when the vehicle stops.

11. In paragraph 1, including shared memory; The central processor transmits the sensor data to the neural processor using the shared memory, The neural processor is a signal processing device that transmits the result data to the central processor using the shared memory.

12. In paragraph 11, The frame rate of the sensor data recorded in the shared memory is fixed, A signal processing device wherein the frame rate of the result data recorded in the shared memory is variable.

13. In paragraph 2, The above first virtual machine, A signal processing device that receives the vehicle driving information detected by the neural processor and executes a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information.

14. In paragraph 2, Among the above multiple virtual machines, the second virtual machine is: Receive the vehicle driving information from the first virtual machine, and execute a vehicle driving assistance engine or an autonomous driving engine based on the vehicle driving information, Among the above multiple virtual machines, the third virtual machine is: A signal processing device that receives the vehicle driving information from the first virtual machine and executes an augmented reality engine based on the vehicle driving information.

15. In paragraph 14, The above second virtual machine operates for the first display, The third virtual machine is a signal processing device that operates for a second display.

16. In paragraph 1, Among the above multiple virtual machines, the first virtual machine is: Controlling the first neural processor among the at least one neural processor to operate at a variable frame rate or output result data based on the sensor data, A signal processing device that controls a second neural processor among the at least one neural processor to operate at a variable frame rate or output result data based on camera data from a camera device in a vehicle.

17. In paragraph 16, Among the above multiple virtual machines, the first virtual machine is: During the first period, the first neural processor is controlled to operate at a first frame rate or output result data, A signal processing device that controls the second neural processor to operate at a second frame rate or output result data during the first period.

18. At least one neural processor; A central processor that runs a hypervisor; The central processor, Running multiple virtual machines on the hypervisor, Among the above multiple virtual machines, the first virtual machine is: When the energy saving mode is turned off, the neural processor is controlled to operate at a fixed frame rate or output result data based on sensor data from a sensor device in the vehicle. A signal processing device that controls the neural processor to operate at a variable frame rate or output the result data based on vehicle driving information based on the sensor data when the energy saving mode is turned on.

19. In paragraph 18, Among the above multiple virtual machines, the first virtual machine is: When the energy saving mode is turned on and the vehicle moves forward, the neural processor is controlled to operate at a first frame rate or output the result data. A signal processing device that controls the neural processor to operate at a second frame rate, different from the first frame rate, or to output the result data, based on vehicle driving information based on the sensor data, when the vehicle is moving backwards while the energy saving mode is turned on.

20. Sensor device; A signal processing device for detecting vehicle driving information based on sensor data from the above sensor device; The above signal processing device, A vehicle display device comprising a signal processing device according to any one of claims 1 to 19.

Citation Information

Patent Citations

  • Method and apparatus of recognizing object

    KR102463175B1

  • Camera for vehicle

    KR102284128B1

  • Smart Outlets capable of preventing fire and a fire prevention system including the same

    KR102807786B1

  • Coating Solution Composition for Anti-Reflection and Pollution Prevention of Solar Module, and Method for Manufacturing the Same

    KR102966434B1

  • Training multiple neural networks of a vehicle perception component based on sensor settings

    US20190176841A1