Signal processing device and display device for vehicle including same

Abstract

The present disclosure relates to a signal processing device and a display device for a vehicle, wherein the display device includes the signal processing device. The signal processing device according to an embodiment of the present disclosure comprises at least one neural processor, and a central processor for running a hypervisor. The central processor runs a plurality of virtualization machines on the hypervisor. In a state in which a request for a first operation, a request for a second operation, and a request for a third operation have been sequentially received from a plurality of applications running in at least one of the plurality of virtualization machines, a first virtualization machine controls a first neural processor to process the first operation and the third operation in parallel when the first operation and the third operation can be processed in parallel, and process the second operation after the first operation and the third operation are completed. Accordingly, the neural processor can be operated efficiently.

Description

Signal processing device and vehicle display device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle display device having the same, and more particularly, to a signal processing device capable of efficiently operating a neural processor and a vehicle display device having the same.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, a signal processing device is used for real-time processing of various camera data while the vehicle is driving.

[0004] At this time, as the number of camera devices increases, the amount of real-time processing for camera data increases.

[0005] Meanwhile, when multiple operation requests are received simultaneously from multiple applications, they are generally processed one by one in the order in which they were requested (FIFO), resulting in a delay problem in that subsequent operation requests must wait until the preceding operation is completed.

[0006] Meanwhile, in line with research on ADAS or autonomous driving technology, it is necessary to detect objects such as surrounding vehicles, pedestrians, and lane shapes in real time based on camera data from a front camera or surround camera.

[0007] Furthermore, detection of the driver's gaze, etc., becomes necessary based on camera data from the vehicle's internal camera.

[0008] Accordingly, methods for efficiently processing data in signal processing devices within vehicles are being studied.

[0009] The technical problem of the present disclosure is to provide a signal processing device capable of efficiently operating a neural processor, and a vehicle display device having the same.

[0010] Meanwhile, another technical task of the present disclosure is to provide a signal processing device capable of reducing power consumption, and a vehicle display device having the same.

[0011] In order to solve the above technical problem, a signal processing device according to one embodiment of the present disclosure, and a vehicle display device including the same, comprises at least one neural processor and a central processor that executes a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications executed in at least one of the plurality of virtual machines, and when the first operation and the third operation are capable of parallel processing, the first neural processor controls the first operation and the third operation to be processed in parallel, and controls the second operation to be processed after the first operation and the third operation are completed.

[0012] Meanwhile, the first virtual machine can control the first neural processor to sequentially process the second operation and the fourth operation after the completion of the first and third operations, if the sharing of the operation layers during the second and fourth operations is possible while the third operation has received a request for the fourth operation.

[0013] Meanwhile, the first virtual machine can control the arrangement of data for multiple operations within the internal memory of the first neural processor to be varied when multiple operations are requested from multiple applications.

[0014] Meanwhile, the first virtual machine executes a neural system service for controlling the neural processor, and the neural system service can control the arrangement of data for multiple operations within the internal memory of the first neural processor to vary when multiple operations are requested from multiple applications.

[0015] Meanwhile, the first virtual machine controls the initialization of the learning model in response to a learning model initialization request from the application, controls the initialization of the learning model, receives a file related to the learning model stored in the shared memory, and stores the received file related to the learning model in the internal memory of the first neural processor, controls the parallel processing of data corresponding to the first operation and data corresponding to the third operation among sensor data or camera data stored in the shared memory in response to an inference request from the application, and controls the processing of data corresponding to the second operation among the sensor data or camera data stored in the shared memory after the completion of the first operation and the third operation.

[0016] Meanwhile, the first virtual machine can control the processing result data of the first to third operations to be stored in shared memory within the hypervisor.

[0017] Meanwhile, the first virtual machine can control the discretization level of the second operation to be lowered from the first level to the second level when in energy saving mode or when no object is detected for a predetermined period of time based on the result of the first operation.

[0018] Meanwhile, the first virtual machine can control the first neural processor to perform the first to third operations when the central processor cannot perform the first to third operations.

[0019] Meanwhile, the first virtual machine can perform the first to third operations on the central processor, and if the first and third operations can be processed in parallel, the central processor can be controlled to process the first and third operations in parallel, and after the first and third operations are completed, the second operation can be processed.

[0020] Meanwhile, the first virtual machine can control to store the result data of the second operation in an internal memory within the first neural processor, if the second operation includes an action recognition operation based on audio data from a microphone and the fourth operation includes an eye recognition operation based on camera data, and to process the fourth operation using at least a portion of the result data stored in the internal memory.

[0021] Meanwhile, the first neural processor may include an internal memory that stores weight data or input data, and a calculator that performs a calculation based on data from the internal memory.

[0022] Meanwhile, the calculator within the first neural processor can perform either the first operation or the third operation in the first region and perform the other operation in the second region surrounding the first region.

[0023] Meanwhile, the first neural processor may perform at least one of the first operation to the third operation based on input data of the first length according to the first quantization mode, and may perform at least one of the first operation to the third operation based on input data of the second length longer than the first length according to the second quantization mode.

[0024] Meanwhile, the first neural processor may perform at least one of the first operation to the third operation based on input data of the first length according to the first quantization mode, and may perform a concatenation operation based on input data of the first length and differential data corresponding to the input data of the first length according to the second quantization mode, and may generate input data of a second length longer than the first length according to the concatenation operation, and may perform at least one of the first operation to the third operation based on the input data of the second length.

[0025] Meanwhile, the first neural processor may perform at least one of the first to third operations according to the second quantization mode, and then, if the number of feature points for the detected object is less than or equal to a predetermined value according to result data of at least one of the first to third operations, switch to the first quantization mode.

[0026] Meanwhile, the first neural processor may perform at least one of the first to third operations depending on the second quantization mode or the daytime mode when the vehicle is driven during the day, and may perform at least one of the first to third operations depending on the first quantization mode or the nighttime mode when the vehicle is driven at night.

[0027] Meanwhile, the first virtual machine can control the first neural processor to process the first operation and the third operation in parallel if parallel processing is possible, and can control the second neural processor to process the second operation.

[0028] Meanwhile, the first virtual machine can control the second neural processor to process the fourth operation when a request for the fourth operation is received from multiple applications.

[0029] Meanwhile, the first virtual machine controls the second neural processor to perform a fourth operation based on input data of a first length according to the first quantization mode, and the first neural processor can control the first neural processor to perform at least one of the first to third operations based on input data of a second length longer than the first length according to the second quantization mode.

[0030] A signal processing device according to another embodiment of the present disclosure, and a vehicle display device including the same, includes a central processor executing a hypervisor and a neural processor, and exchanges data with an external device including a second neural processor, either wired or wirelessly, wherein the central processor executes a plurality of virtual machines on the hypervisor, and the first virtual machine transmits data to the second neural processor in the external device when the external device is connected.

[0031] Meanwhile, the central processor can control, when an external device is connected wirelessly, to perform wireless pairing through a communication device and, after wireless pairing is completed, to transmit data to a second neural processor within the external device.

[0032] Meanwhile, the first virtual machine can perform load balancing for load distribution between the neural processor and the second neural processor.

[0033] Meanwhile, the first virtual machine can control the download, installation, and execution of an application for controlling an external device from the server when an external device is connected.

[0034] Meanwhile, the first virtual machine can receive result data from an external device and control the transmission of the result data to the neural processor.

[0035] Meanwhile, the first virtual machine can receive result data and battery information from an external device and control the result data to be stored in shared memory within the hypervisor.

[0036] Meanwhile, the external device may include a second signal processing device or a mobile terminal or server.

[0037] A signal processing device according to one embodiment of the present disclosure, and a vehicle display device including the same, comprises at least one neural processor and a central processor that executes a hypervisor, wherein the central processor executes a plurality of virtual machines on the hypervisor, and a first virtual machine sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications executing in at least one of the plurality of virtual machines, and when the first operation and the third operation can be processed in parallel, the first neural processor controls the first operation and the third operation to be processed in parallel, and controls the second operation to be processed after the first operation and the third operation are completed. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0038] Meanwhile, if the first virtual machine receives a request for a fourth operation after the third operation has been requested and the second and fourth operations can share the computational layers, the first neural processor can control the second and fourth operations to be processed sequentially after the first and third operations are completed. This enables efficient operation of the neural processor.

[0039] Meanwhile, the first virtual machine can control the arrangement of data for multiple operations within the internal memory of the first neural processor to vary when multiple operations are requested from multiple applications. This allows the neural processor to operate efficiently.

[0040] Meanwhile, the first virtual machine executes a neural system service for controlling the neural processor. The neural system service can control the arrangement of data for multiple operations within the internal memory of the first neural processor to vary when multiple operations are requested from multiple applications. This enables efficient operation of the neural processor.

[0041] Meanwhile, the first virtual machine controls the initialization of the learning model in response to a learning model initialization request from an application, controls the initialization of the learning model, receives a file related to the learning model stored in shared memory, and stores the received file related to the learning model in an internal memory within the first neural processor, controls the parallel processing of data corresponding to the first operation and data corresponding to the third operation among sensor data or camera data stored in the shared memory in response to an inference request from the application, and controls the processing of data corresponding to the second operation among sensor data or camera data stored in the shared memory after the completion of the first operation and the third operation. Accordingly, the neural processor can be operated efficiently.

[0042] Meanwhile, the first virtual machine can control the processing results of the first to third operations to be stored in shared memory within the hypervisor. This enables the sharing of the resulting data. Ultimately, this enables efficient operation of the neural processor.

[0043] Meanwhile, the first virtual machine can control the discretization level of the second operation to be lowered from the first level to the second level when in energy-saving mode or when no object is detected for a predetermined period of time based on the results of the first operation. This allows the neural processor to operate efficiently. Furthermore, power consumption can be reduced.

[0044] Meanwhile, the first virtual machine can control the first neural processor to perform the first to third operations when the central processor is unable to perform the first to third operations. This allows the neural processor to operate efficiently.

[0045] Meanwhile, the first virtual machine can perform the first to third operations on the central processor, and if the first and third operations can be processed in parallel, the central processor can control the first and third operations to be processed in parallel, and control the second operation to be processed after the first and third operations are completed. Accordingly, the neural processor can be operated efficiently.

[0046] Meanwhile, if the second operation includes a behavior recognition operation based on audio data from a microphone and the fourth operation includes a gaze recognition operation based on camera data, the first virtual machine can control the first neural processor to store the result data of the second operation in internal memory, and control the fourth operation to be processed using at least a portion of the result data stored in the internal memory. Accordingly, the neural processor can be operated efficiently.

[0047] Meanwhile, the first neural processor may include an internal memory for storing weight data or input data, and a calculator for performing calculations based on data from the internal memory. This allows the neural processor to operate efficiently.

[0048] Meanwhile, the calculator within the first neural processor can perform either the first or third operation in the first region and the other operation in the second region surrounding the first region. This allows for efficient operation of the neural processor.

[0049] Meanwhile, the first neural processor may perform at least one of the first to third operations based on input data of a first length according to the first quantization mode, and may perform at least one of the first to third operations based on input data of a second length that is longer than the first length according to the second quantization mode. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0050] Meanwhile, the first neural processor, according to the first quantization mode, performs at least one of the first operation to the third operation based on input data of the first length, and according to the second quantization mode, performs a concatenation operation based on the input data of the first length and differential data corresponding to the input data of the first length, and generates input data of a second length longer than the first length according to the concatenation operation, and performs at least one of the first operation to the third operation based on the input data of the second length. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0051] Meanwhile, the first neural processor may perform at least one of the first to third operations according to the second quantization mode, and then switch to the first quantization mode if the number of feature points for the detected object is below a predetermined value based on the result data of at least one of the first to third operations. This allows the neural processor to operate efficiently. Furthermore, power consumption can be reduced.

[0052] Meanwhile, the first neural processor may perform at least one of the first to third operations, depending on the second quantization mode or the daytime mode, when the vehicle is driven during the daytime, and may perform at least one of the first to third operations, depending on the first quantization mode or the nighttime mode, when the vehicle is driven at night. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0053] Meanwhile, the first virtual machine can control the first neural processor to process the first and third operations in parallel, if parallel processing is possible, and the second neural processor to process the second operation. This allows for efficient operation of the neural processor. Furthermore, power consumption can be reduced.

[0054] Meanwhile, the first virtual machine can control the second neural processor to process the fourth operation when a request for the fourth operation is received from multiple applications. This allows the neural processor to operate efficiently. Furthermore, power consumption can be reduced.

[0055] Meanwhile, the first virtual machine controls the second neural processor to perform a fourth operation based on input data of a first length according to the first quantization mode, and can control the first neural processor to perform at least one of the first to third operations based on input data of a second length that is longer than the first length according to the second quantization mode. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0056] A signal processing device according to another embodiment of the present disclosure, and a vehicle display device including the same, includes a central processor that executes a hypervisor and a neural processor, and exchanges data with an external device including a second neural processor, either wired or wirelessly, wherein the central processor executes a plurality of virtual machines on the hypervisor, and the first virtual machine transmits data to the second neural processor within the external device when the external device is connected. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0057] Meanwhile, the central processor can control wireless pairing via a communication device when an external device is connected wirelessly, and transmit data to a second neural processor within the external device after wireless pairing is complete. This enables efficient operation of the neural processor.

[0058] Meanwhile, the first virtual machine can perform load balancing to distribute the load between the neural processor and the second neural processor. This allows the neural processor to operate efficiently.

[0059] Meanwhile, the first virtual machine can control the download, installation, and execution of an application for controlling an external device from the server when the external device is connected. This enables stable communication with the external device.

[0060] Meanwhile, the first virtual machine can receive result data from an external device and control the transmission of the result data to the neural processor. This allows the neural processor to operate efficiently.

[0061] Meanwhile, the first virtual machine can receive result data and battery information from an external device and control the storage of the result data in shared memory within the hypervisor. This enables the sharing of result data. Ultimately, this enables efficient operation of the neural processor.

[0062] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0063] FIG. 2 is a drawing illustrating the appearance of a vehicle display device according to an embodiment of the present disclosure.

[0064] FIG. 3 illustrates an example of an internal block diagram of the vehicle display device of FIG. 2.

[0065] FIG. 4 is a diagram illustrating a system driven by a signal processing device related to the present disclosure.

[0066] FIG. 5 is a diagram illustrating an example of a system driven by a signal processing device of the present disclosure.

[0067] FIG. 6 is a drawing for reference in explaining the operation of a system driven by a signal processing device of the present disclosure.

[0068] Figures 7a to 9d are drawings referred to in the description of Figure 5 or Figure 6.

[0069] FIG. 10A is a diagram illustrating an example of a system driven by a signal processing device related to the present disclosure.

[0070] Figures 10b to 10c are drawings referenced in the description of Figure 10a.

[0071] FIG. 11 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0072] FIG. 12 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0073] Figures 13a to 25b are drawings referenced in the description of Figure 11 or Figure 12.

[0074] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0075] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0076] Figure 1 is a drawing showing an example of the exterior and interior of a vehicle.

[0077] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL, etc.) that rotate by a power source and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0078] Meanwhile, the vehicle (200) may further be equipped with a camera (195) for capturing images of the front of the vehicle.

[0079] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside, and an image projection device (180h) for projecting images onto a windshield (WS).

[0080] In Fig. 1, a plurality of displays (180a, 180b) are illustrated, including a cluster display (180a), an AVN (Audio Video Navigation) display (180b), and a head-up display (HUD) image projection device (180h).

[0081] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named a center information display.

[0082] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, and an electric vehicle equipped with an electric motor as a power source.

[0083] FIG. 2 is a drawing illustrating the appearance of a vehicle display device according to an embodiment of the present disclosure.

[0084] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b), an image projection device (180h), and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) or the image projection device (180h).

[0085] Among the plurality of displays (180a to 180b), the first display (180a) may be a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle driving information, a navigation map, various entertainment information, or images.

[0086] The image projection device (180h) may be equipped with an optical device (not shown) for image projection as a head-up display.

[0087] The signal processing device (170) has a shared memory (508) and a central processor (175) inside, and can execute a first virtual machine to a third virtual machine (520 to 540) on a hypervisor (505) within the central processor (175).

[0088] The first virtual machine (520) is a server virtual machine and can control the second virtual machine (530) and the third virtual machine (540), which are guest virtual machines.

[0089] Meanwhile, the second virtual machine can be named the first guest virtual machine (Guest Virtual Maschine), and the third virtual machine can be named the second guest virtual machine.

[0090] The first guest virtualization machine (530) can operate for the first display (180a), and the second guest virtualization machine (540) can operate for the second display (180b).

[0091] Meanwhile, the server virtualization machine (520) within the central processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transmission to the first guest virtualization machine (530) and the second guest virtualization machine (540). Accordingly, the same information or the same image can be displayed in synchronization on the first display (180a) and the second display (180b) within the vehicle.

[0092] Meanwhile, the server virtualization machine (520) within the central processor (175) can receive and process camera data from the camera device and transmit an object based on the camera data to at least one of the first guest virtualization machine (530) or the second guest virtualization machine (540). Accordingly, the detected object can be shared with at least one virtualization machine, etc.

[0093] Meanwhile, the server virtualization machine (520) within the central processor (175) can receive and process the vehicle's wheel speed sensor data, and transmit speed information based on the wheel speed sensor data to at least one of the first guest virtualization machine (530) or the second guest virtualization machine (540). Accordingly, the vehicle's speed information can be shared with at least one virtual machine, etc.

[0094] Accordingly, it is possible to control various displays (180a to 180b) or image projection devices (180h) using one signal processing device (170).

[0095] Meanwhile, some of the multiple displays (180a~180b) may operate under Linux OS, while others may operate under Web OS.

[0096] The signal processing device (170) according to the embodiment of the present disclosure can control the display (180a to 180b) operating under various operating systems (OS) to display the same information or the same image in synchronization.

[0097] FIG. 3 illustrates an example of an internal block diagram of a vehicle display device according to an embodiment of the present disclosure.

[0098] Referring to the drawings, a vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120), an interface (130), a memory (140), a signal processing device (170), a plurality of displays (180a to 180b), an image projection device (180h), an audio output unit (185), and a power supply unit (190).

[0099] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0100] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0101] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (not shown).

[0102] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0103] The communication device (120) can receive weather information, road traffic information, for example, TPEG (Transport Protocol Expert Group) information, from a mobile terminal (800) or a server (not shown). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0104] The interface (130) can receive sensor data, etc. from the ECU (770) or sensor device (700) and transmit the received information to the signal processing device (170).

[0105] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0106] Such sensor data may be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a wheel sensor, a vehicle speed sensor, a body tilt detection sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc. Meanwhile, the position module may include a GPS module for receiving GPS information.

[0107] Meanwhile, the interface (130) can receive vehicle front image data, vehicle side image data, vehicle rear image data, vehicle surrounding obstacle distance information, etc. from a camera (195) or a rider (not shown), and transmit the received information to a signal processing device (170).

[0108] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0109] For example, the memory (140) may store data regarding a hypervisor, a server virtualization machine (520), and multiple guest virtualization machines for execution within the central processor (175).

[0110] The audio output unit (185) converts an electric signal from the signal processing device (170) into an audio signal and outputs it. For this purpose, a speaker or the like may be provided.

[0111] The power supply unit (190) can supply power required for the operation of each component under the control of the signal processing device (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0112] The image projection device (180h) includes an optical device (not shown) for image projection, and can output an augmented reality-based object under the control of a signal processing device (170).

[0113] For example, the video projection device (180h) can output vehicle speed information, vehicle direction information, front vehicle object, distance indicator from front vehicle, etc.

[0114] As another example, the image projection device (180h) can output an augmented reality rain carpet, an augmented reality route carpet, or an augmented reality dynamic carpet corresponding to the image of the road.

[0115] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100).

[0116] For example, the signal processing device (170) may include a shared memory (508), a central processor (175) that performs signal processing for a vehicle display (180a, 180b), and at least one neural processor (177).

[0117] The central processor (175) executes a hypervisor (505 in FIG. 5), and can execute a server virtualization machine (520) and multiple guest virtualization machines (530 to 540) on the executed hypervisor (505 in FIG. 5).

[0118] At this time, the first guest virtualization machine (530) can operate for the first display (180a), and the second guest virtualization machine (540) can operate for the second display (180b).

[0119] For example, the server virtualization machine (520) within the central processor (175) can receive, process, or process vehicle sensor data, location information data, camera data, audio data, or touch input data, and output them. By distinguishing between data processed only in the legacy virtualization machine and data processed in the server virtualization machine (520), data processing can be performed efficiently. In particular, by performing most of the data processing in the server virtualization machine (520), data sharing in a 1:N manner becomes possible.

[0120] As another example, the server virtualization machine (520) can directly receive and process CAN communication data, audio data, radio data, USB data, and wireless communication data for the first guest virtualization machine or the second guest virtualization machine (530 to 540).

[0121] And, the server virtualization machine (520) can transmit processed data to the first guest virtualization machine or the second guest virtualization machine (530 to 540).

[0122] Accordingly, among the server virtualization machine (520) and multiple guest virtualization machines (530-540), only the server virtualization machine (520) receives communication data and external input data and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling efficient implementation of interfaces for external input data and communication data. Furthermore, 1:N data communication becomes possible, enabling synchronization when sharing data.

[0123] Meanwhile, the server virtualization machine (520) can control the first guest virtualization machine (530) and the second guest virtualization machine (540) to share the same data by writing data to the shared memory (508).

[0124] For example, the server virtualization machine (520) can control the first guest virtualization machine (530) and the second guest virtualization machine (540) to share the same data by recording vehicle sensor data, the location information data, the camera data, or the touch input data in the shared memory (508). Accordingly, sharing of data in a 1:N manner becomes possible.

[0125] Ultimately, by performing most of the data processing on the server virtualization machine (520), data sharing in a 1:N manner becomes possible.

[0126] Meanwhile, the server virtualization machine (520) within the central processor (175) can control the shared memory (508) based on the hypervisor (505) to be set for the same data transfer to the first guest virtualization machine (530) and the second guest virtualization machine (540).

[0127] That is, the server virtualization machine (520) within the central processor (175) can synchronize and transmit the same data to the first guest virtualization machine (530) and the second guest virtualization machine (540) using the shared memory (508) based on the hypervisor (505). Accordingly, the same image can be synchronized and displayed on multiple displays (180a to 180b) within the vehicle.

[0128] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a system on chip (SOC).

[0129] FIG. 4 is a diagram illustrating a system driven by a signal processing device related to the present disclosure.

[0130] Referring to the drawings, FIG. 4 is a drawing illustrating that virtual machines are used for each of the cluster display (180a) and the AVN display (180b).

[0131] The system (400) driven by the signal processing device of FIG. 4 exemplifies that a cluster virtualization machine (430) and an AVN virtualization machine (440) are executed on a hypervisor (405) within a central processor (175).

[0132] Meanwhile, the system (400) driven by the signal processing device of FIG. 4 exemplifies that a legacy virtualization machine (410) is also executed on a hypervisor (405) within the central processor (175).

[0133] The legacy virtualization machine (410) has an interface (412) for data communication with memory (140) and an interface (413) for Ethernet communication.

[0134] Meanwhile, the cluster virtualization machine (430) may be equipped with an interface (431) for CAN communication, an interface (432) for communication with an interface (412) of a legacy virtualization machine (410), and an interface (433) for communication with an interface (413) of a legacy virtualization machine (410).

[0135] Meanwhile, the AVN virtualization machine (440) may be equipped with an interface (441) for input / output of audio data, radio data, USB data, and wireless communication data, an interface (442) for communication with an interface (412) of a legacy virtualization machine (410), and an interface (443) for communication with an interface (413) of a legacy virtualization machine (410).

[0136] According to this system (400), CAN communication data is input / output only in the cluster virtualization machine (430), so there is a disadvantage in that CAN communication data cannot be utilized in the AVN virtualization machine (440).

[0137] In addition, according to the system (400) of FIG. 4, audio data, radio data, USB data, and wireless communication data are input / output only in the AVN virtualization machine (440), so there is a disadvantage in that such data cannot be utilized in the cluster virtualization machine (430).

[0138] Meanwhile, there is a disadvantage in that the cluster virtualization machine (430) and the cluster virtualization machine (430) must each have separate interfaces (431, 432, 441, 442) for memory data and Ethernet communication data input / output from the legacy virtualization machine (410).

[0139] Accordingly, the present disclosure proposes a method for improving the system of FIG. 4. That is, unlike FIG. 4, the virtual machine is divided into a server virtual machine (520) and a guest virtual machine, and various memory data, communication data, etc. are input / output from the server virtual machine (520) rather than from the guest virtual machine. This will be described with reference to FIG. 5 and below.

[0140] FIG. 5 is a diagram illustrating an example of a system driven by a signal processing device of the present disclosure.

[0141] Referring to the drawing, the system (500) of FIG. 5 illustrates that a server virtualization machine (520), which is a server virtualization machine (520), a first guest virtualization machine (530), which is a guest virtualization machine, and a second guest virtualization machine (540) are executed on a hypervisor (505) within a central processor (175) within a signal processing device (170).

[0142] The first guest virtualization machine (530) may be a virtualization machine for a cluster display (180a), and the second guest virtualization machine (540) may be a virtualization machine for an AVN display (180b).

[0143] That is, the first guest virtualization machine (530) and the second guest virtualization machine (540) can operate for image rendering of the cluster display (180a) and AVN display (180b), respectively.

[0144] Meanwhile, the system (500) driven by the signal processing device (170) of FIG. 5 exemplifies that a legacy virtualization machine (510) is also executed on a hypervisor (505) within the central processor (175).

[0145] The legacy virtualization machine (510) has an interface (511) for data communication with memory (140) and Ethernet communication.

[0146] Meanwhile, the legacy virtualization machine (510) may further include an interface (virtio-backend interface) (512) for data communication with the first and second guest virtualization machines (530, 540).

[0147] The server virtualization machine (520) may be equipped with an interface (521) for input / output of audio data, radio data, USB data, and wireless communication data, and an input / output server interface (522) for data communication with a guest virtualization machine.

[0148] That is, the server virtualization machine (520), which is a server virtualization machine (520), can provide I / Os that are difficult to virtualize by standard virtualization technology (VirtIO) to a plurality of guest virtualization machines, for example, the first and second guest virtualization machines (530, 540).

[0149] Meanwhile, the server virtualization machine (520), which is a server virtualization machine (520), can control radio data, audio data, etc. at the supervisor level and provide them to a number of guest virtualization machines, for example, the first and second guest virtualization machines (530, 540).

[0150] Meanwhile, the server virtualization machine (520), which is a server virtualization machine (520), processes vehicle data, sensor data, vehicle surrounding information, etc., and can provide the processed data or information, etc. to a plurality of guest virtualization machines, for example, first and second guest virtualization machines (530, 540).

[0151] Meanwhile, the server virtualization machine (520) can provide services (Supervisory Services), such as processing of vehicle data and management of audio routing.

[0152] Next, the first guest virtualization machine (530) may be equipped with an input / output client interface (532) for data communication with the server virtualization machine (520) and APIs (533) for controlling the input / output client interface (532).

[0153] Additionally, the first guest virtualization machine (530) may be equipped with an interface (virtio-backend interface) for data communication with the legacy virtualization machine (510).

[0154] The first guest virtualization machine (530) can receive memory data through communication with memory (140) and Ethernet data through Ethernet communication from the interface (virtio-backend interface) (512) of the legacy virtualization machine (510) through the interface (virtio-backend interface).

[0155] Next, the second guest virtualization machine (540) may be equipped with an input / output client interface (542) for data communication with the server virtualization machine (520) and APIs (543) for controlling the input / output client interface (542).

[0156] Additionally, the second guest virtualization machine (540) may be equipped with an interface (virtio-backend interface) for data communication with the legacy virtualization machine (510).

[0157] The second guest virtualization machine (540) can receive memory data through communication with memory (140) and Ethernet data through Ethernet communication from the interface (virtio-backend interface) (512) of the legacy virtualization machine (510) through the interface (virtio-backend interface).

[0158] Meanwhile, unlike FIG. 5, the legacy virtualization machine (510) may also be installed within the server virtualization machine (520).

[0159] According to the system (500), CAN communication data such as sensing data is input / output only in the server virtualization machine (520), but can be provided to multiple guest virtualization machines, for example, the first and second guest virtualization machines (530, 540), through data processing in the server virtualization machine (520). Therefore, 1:N data communication through processing in the server virtualization machine (520) becomes possible.

[0160] In addition, according to the system (500) of FIG. 5, audio data, radio data, USB data, and wireless communication data are input / output only in the server virtualization machine (520), but can be provided to multiple guest virtualization machines, for example, the first and second guest virtualization machines (530, 540), through data processing in the server virtualization machine (520). Therefore, 1:N data communication through processing in the server virtualization machine (520) becomes possible.

[0161] Meanwhile, in the system (500) of FIG. 5, the first and second guest virtualization machines (530, 540) can operate based on different OSs.

[0162] For example, the first guest virtualization machine (540) may operate under a Linux OS, and the second guest virtualization machine (540) may operate under a web OS.

[0163] The server virtualization machine (520) sets up a shared memory (508) based on a hypervisor (505) for data sharing, even though the first and second guest virtualization machines (530, 540) operate under different operating systems (OS). Accordingly, even though the first and second guest virtualization machines (530, 540) operate under different operating systems (OS), they can synchronize and share the same data or the same image. Consequently, the same data or the same image can be synchronized and displayed on multiple displays (180a, 180b).

[0164] FIG. 6 is a drawing for reference in the description of the operation of a system driven by the signal processing device of the present disclosure, and FIGS. 7a to 9b are drawings for reference in the description of FIG. 5 or FIG. 6.

[0165] First, according to the system (500) of FIG. 6, the central processor (175) in the signal processing device (170) executes a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) on a hypervisor (505) in the central processor (175), and the server virtualization machine (520) in the central processor (175) controls a shared memory (508) based on the hypervisor (505) to be set for data transmission to the first guest virtualization machine (530) and the second guest virtualization machine (540).

[0166] For example, the same image data, for example, can be transmitted from a server virtualization machine (520) to a first guest virtualization machine (530) and a second guest virtualization machine (540). Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0167] Meanwhile, according to the system (500) of FIG. 6, the central processor (175) in the signal processing device (170) executes a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) on a hypervisor (505) in the central processor (175), and the server virtualization machine (520) in the central processor (175) can synchronize and transmit the same data to the first guest virtualization machine (530) and the second guest virtualization machine (540) using a shared memory (508) based on the hypervisor (505).

[0168] For example, the same data may include CAN communication data, audio data, radio data, USB data, wireless communication data, location information data, or touch data. Accordingly, the same data can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0169] Meanwhile, the server virtualization machine (520) within the central processor (175) can receive location information data that changes depending on movement, process it, and provide it to the first guest virtualization machine (530) or the second guest virtualization machine (540). Accordingly, rather than 1:1 data communication between virtualization machines, 1:N data communication is possible using shared memory.

[0170] Meanwhile, the first guest virtualization machine (530) and the second guest virtualization machine (540) can be driven by different operating systems. Accordingly, even if multiple virtualization machines (520, 530, 540) are driven by different operating systems, high-speed data communication can be performed.

[0171] Meanwhile, although not shown in FIG. 6, the legacy virtualization machine (510) can transmit memory data from the memory (140) and Ethernet data via Ethernet communication to the first guest virtualization machine (530) and the second guest virtualization machine (540) in a synchronized manner using the shared memory (508) based on the hypervisor (505). In other words, 1:N data communication for memory data or Ethernet data can be performed. Accordingly, the same data can be transmitted in a synchronized manner.

[0172] Meanwhile, the server virtualization machine (520) within the central processor (175) can execute a system manager, a display manager, etc. as a supervisory service.

[0173] Meanwhile, the server virtualization machine (520) within the central processor (175) can execute vehicle information service, location information service, camera service, auto-sa, Bluetooth communication service, radio service, Wi-Fi service, audio service, touch service, etc. as systemic services.

[0174] FIG. 7a illustrates that three virtual machines (420, 420, 430) are operating in the system (400b) of FIG. 4.

[0175] Referring to the drawing, the server virtualization machine (420) is a Linux-based virtualization machine and may be equipped with an input / output server interface (422) for data transmission, and the first guest virtualization machine (530) and the second guest virtualization machine (540) may include input / output client interfaces (432, 442) for data communication with the input / output server interface (422).

[0176] For example, in order for a server virtualization machine (420) to transmit first data to a first guest virtualization machine (430), a first shared memory (408a) must be set within the hypervisor (405), and in order for the server virtualization machine (420) to transmit the same first data to a second guest virtualization machine (440), a second shared memory (408b) separate from the first shared memory (408a) must be set within the hypervisor (405).

[0177] When transmitting the same first data, as in Fig. 7a, if a separate shared memory is used, there is a disadvantage in that resources are wasted and synchronization is not easy.

[0178] FIG. 7b illustrates that, by the system (400b) of FIG. 7a, a first guest virtualization machine (430) displays image data received through a first shared memory (408a) on a first display (180a), and a second guest virtualization machine (440) displays image data received through a second shared memory (408b) on a second display (180b).

[0179] In FIG. 7b, the synchronization between the image (705a) displayed on the first display (180a) and the image (705b) displayed on the second display (180b) is not correct, and the image (705b) displayed on the second display (180b) corresponds to a previous frame than the image (705a) displayed on the first display (180a).

[0180] In this way, when the server virtualization machine (420) transmits the same image data, as in Fig. 7a, there is a disadvantage in that synchronization at the time of image display is not performed when a separate shared memory is used, as in Fig. 7b.

[0181] To address these issues, this disclosure proposes a method of allocating a single shared memory when transmitting identical data. This enables 1:N data communication and synchronized data transmission.

[0182] FIG. 8 illustrates a system (500) according to an embodiment of the present disclosure, in which a server virtualization machine (520) and a plurality of guest virtualization machines (530 to 540) are executed on a hypervisor (505) within a central processor (175), and the server virtualization machine (520) within the central processor (175) controls a shared memory (508) based on the hypervisor (505) to be set for the same data transfer to a first guest virtualization machine (530) and a second guest virtualization machine (540).

[0183] Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0184] Meanwhile, high-speed data communication can be performed between multiple virtual machines (520, 530, 540). Furthermore, high-speed data communication can be performed even if multiple virtual machines (520, 530, 540) are driven by different operating systems.

[0185] Meanwhile, when the server virtualization machine (520) within the central processor (175) transmits data processed in the server virtualization machine (520) to another virtualization machine, it can utilize a single shared memory (508) rather than allocating a number of memories corresponding to the number of virtualization machines. Accordingly, rather than a 1:1 data communication between virtualization machines, a 1:N data communication is possible using the shared memory (508).

[0186] Meanwhile, the server virtualization machine (520) within the central processor (175) may include an input / output server interface (522) and a security manager (526).

[0187] Meanwhile, the first guest virtualization machine (530) and the second guest virtualization machine (540) may each include input / output client interfaces (532, 542). Accordingly, high-speed data communication between multiple virtualization machines (520, 530, 540) can be performed using the input / output server interface (522) and the input / output client interfaces (532, 542).

[0188] The input / output server interface (522) within the server virtualization machine (520) can receive the same data transmission request from each input / output client interface (532, 542) within the first guest virtualization machine (530) and the second guest virtualization machine (540), and based on this, can transmit the shared data to the shared memory (508) through the security manager (526).

[0189] Figure 9a illustrates a more detailed diagram of shared data transmission.

[0190] Referring to the drawing, for shared data transmission, the input / output server interface (522) within the server virtualization machine (520) transmits a request for allocation of shared memory (508) to the security manager (526) (S1).

[0191] Next, the security manager (526) can allocate shared memory (508) using the hypervisor (505) (S2) and record shared data in the shared memory (508).

[0192] Meanwhile, the input / output client interface (532, 542) can transmit a connection request to the input / output server interface (522) after allocating shared memory (508) (S3).

[0193] Meanwhile, after allocating shared memory (508), the input / output server interface (522) transmits information about the shared memory (508) containing key data to the input / output client interfaces (532, 542) (S4). The key data at this time may be data for data access.

[0194] Meanwhile, the server virtualization machine (520) within the central processor (175) can transmit information about the shared memory (508) to the first guest virtualization machine (530) and the second guest virtualization machine (540) after setting up the shared memory (508).

[0195] The input / output client interface (532, 542) can access the shared memory (508) based on the received key data (S5) and copy the shared data from the shared memory (508).

[0196] Accordingly, the shared memory (508) becomes accessible to the first guest virtualization machine (530) and the second guest virtualization machine (540), and ultimately, shared data can be shared.

[0197] For example, if the shared data is image data, the image data is shared between the first guest virtualization machine (530) and the second guest virtualization machine (540), and ultimately, the same shared image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle.

[0198] FIG. 9b illustrates that, by the system (500b) of FIG. 9a, a first guest virtualization machine (530) displays image data received through shared memory (508) on a first display (180a), and a second guest virtualization machine (540) displays image data received through shared memory (508) on a second display (180b).

[0199] In Fig. 9b, synchronization is performed between an image (905) displayed on a first display (180a) and an image (905) displayed on a second display (180b), thereby displaying the same image.

[0200] That is, the image data processed in the server virtualization machine (520) within the central processor (175) is transmitted to the first guest virtualization machine (530) and the second guest virtualization machine (540) via the shared memory (508), and based on the image data, the first image (905) displayed on the first display (180a) and the second image (905) displayed on the second display (180b) may be identical to each other. Accordingly, the same image can be displayed in synchronization on multiple displays (180a to 180b) within the vehicle. In addition, high-speed data communication can be performed between multiple virtualization machines (520, 530, 540).

[0201] FIG. 10A is a diagram illustrating an example of a system driven by a signal processing device related to the present disclosure.

[0202] Referring to the drawing, a signal processing device (170) related to the present disclosure may include a central processor (175), a plurality of neural processors (177a to 177c), and a graphic processor (178).

[0203] A system (1000) driven by a signal processing device (170) related to the present disclosure can execute a hypervisor (505) on a central processor (175) and execute multiple virtual machines (520 to 540) on the hypervisor (505).

[0204] Among the plurality of virtual machines (520 to 540), the first virtual machine (520) executes a neural interface (1118) for interfacing with the neural processors (177a to 177c), and the second virtual machine (530) and the third virtual machine (540) can execute applications (539, 549), respectively.

[0205] Meanwhile, when the first application (539) is executed in the second virtual machine (530), the first virtual machine (520) performs a first inference request corresponding to the first application (539) with the neural processors (177a to 177c), receives a first inference result from the neural processors (177a to 177c), and then performs a second inference request corresponding to the second application (549) and receives a second inference result from the neural processors (177a to 177c).

[0206] According to this method, when multiple operation requests are received simultaneously from multiple applications (539, 549), they are generally processed one by one in the order in which they were requested (FIFO), resulting in a delay problem in that subsequent operation requests must wait until the preceding operation is completed.

[0207] FIG. 10b illustrates that the signal processing device (170) of FIG. 10a receives camera data from the camera device (185) and, based on the received camera data, performs vehicle detection (front), vehicle detection (back), pedestrian detection, static object detection, traffic signal detection, overhead sign recognition, road marking detection, road construction detection, lane detection, crossroad detection, toll gate recognition, etc.

[0208] Meanwhile, FIG. 10b illustrates that the signal processing device (170) of FIG. 10a receives lidar data from a lidar device (not shown) and performs vehicle detection (front), vehicle detection (back), pedestrian detection, curve detection, drivable space localization, road surface recognition, etc. based on the received lidar data.

[0209] Meanwhile, in the drawing, a plurality of neural processors (177a to 177c) in the signal processing device (170) of FIG. 10a perform various detections or recognitions, etc. described above based on camera data or rider data, and in particular, continuously perform various detections or recognitions, etc. described above regardless of the vehicle driving state.

[0210] That is, in the drawing, a plurality of neural processors (177a to 177c) in the signal processing device (170) of FIG. 10a perform various detections or recognitions, etc. described above, based on camera data or lidar data, regardless of whether the vehicle is driving at low speed, in traffic congestion, stopped, driving on a high-speed road, driving at high speed, or parked.

[0211] Meanwhile, when the plurality of neural processors (177a to 177c) within the signal processing device (170) of FIG. 10a operate at a fixed frame rate regardless of the vehicle driving status, such as the vehicle speed, there is a disadvantage in that the processing load of the plurality of neural processors (177a to 177c) is considerable. Furthermore, there is a disadvantage in that power consumption is considerable due to the processing load of the plurality of neural processors (177a to 177c).

[0212] Fig. 10c illustrates an example of an operation method of the signal processing device of Fig. 10a.

[0213] Referring to the drawing, the first virtual machine (520) can initialize the learning model according to the execution of the first application (539) in the second virtual machine (530) (S1010).

[0214] For example, the first virtual machine (520) can control the initialization of a learning model loaded on at least one of the neural processors (177a to 177c) according to the execution of the first application (539) on the second virtual machine (530).

[0215] Next, the first virtual machine (520) can load the saved confirmation model (S1012).

[0216] For example, the first virtual machine (520) may receive a file related to a learning model stored in shared memory (508) and control the storage of the received file related to the learning model in the internal memory of at least one of the neural processors (177a to 177c).

[0217] Next, the first virtual machine (520) can collect camera data or sensor data for the first application (S1014).

[0218] For example, the first virtual machine (520) can collect camera data from a camera device and sensor data from a sensor device for the first application.

[0219] Next, the first virtual machine (520) can transmit a first inference request for a first application to at least one of the neural processors (177a to 177c) (S1018), transmit camera data or sensor data to at least one of the neural processors (177a to 177c), and receive a first inference result from at least one of the neural processors (177a to 177c) (S1018).

[0220] Next, the first virtual machine (520) can initialize the learning model according to the execution of the second application (549) in the third virtual machine (540) (S1020).

[0221] Next, the first virtual machine (520) can load the saved confirmation model (S1022).

[0222] Next, the first virtual machine (520) can collect camera data or sensor data for the second application (S1024).

[0223] Next, the first virtual machine (520) can transmit a second inference request for a second application to at least one of the neural processors (177a to 177c) (S1028), transmit camera data or sensor data to at least one of the neural processors (177a to 177c), and receive a second inference result from at least one of the neural processors (177a to 177c) (S1018).

[0224] Meanwhile, until the start of step 1020 (S1020), it takes time Px, and the operation for the second application starts after time Px.

[0225] Additionally, there is a disadvantage that Py takes time from step 1018 (S1018) to completion of step 1028 (S1028).

[0226] According to this method, when multiple operation requests are received simultaneously from multiple applications (539, 549), they are generally processed one by one in the order in which they were requested (FIFO), resulting in a delay problem in that subsequent operation requests must wait until the preceding operation is completed.

[0227] Accordingly, this disclosure proposes a method for efficiently operating a neural processor. This is described with reference to FIG. 11 and below.

[0228] FIG. 11 is a diagram illustrating an example of a system driven by a signal processing device according to an embodiment of the present disclosure.

[0229] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a central processor (175) and at least one neural processor (177a to 177c).

[0230] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may further include a graphics processor (178).

[0231] Meanwhile, the central processor (175) according to the embodiment of the present disclosure executes a hypervisor (505).

[0232] Meanwhile, a system (1100) driven by a signal processing device (170) according to an embodiment of the present disclosure executes a plurality of virtual machines (520 to 540) on a hypervisor (505).

[0233] Specifically, a central processor (175) in a signal processing device (170) according to an embodiment of the present disclosure executes a hypervisor (505) and executes a plurality of virtual machines (520 to 540) on the hypervisor (505).

[0234] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a shared memory (508).

[0235] In the drawing, it is illustrated that a hypervisor (505) is executed on a central processor (175) and a shared memory (508) is executed within the hypervisor (505).

[0236] Meanwhile, a signal processing device (170) according to an embodiment of the present disclosure may receive data from a camera device (195), a sensor device (700), a communication device (120), or a lidar device (not shown), and perform signal processing using at least one of a central processor (175), a graphic processor (178), and a plurality of neural processors (177a to 177c).

[0237] Meanwhile, the sensor device (700) can continuously output sensor data to the signal processing device (170) during vehicle operation.

[0238] The sensor data at this time is data from sensor devices (700) of various vehicles, and may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward / backward data, battery data, fuel data, tire data, vehicle lamp data, vehicle internal temperature data, and vehicle internal humidity data.

[0239] Meanwhile, the camera device (195) can continuously output camera data to the signal processing device (170) during vehicle operation.

[0240] Meanwhile, Lidar (not shown) can continuously output Lidar data to a signal processing device (170) while the vehicle is in motion.

[0241] Meanwhile, the neural processor (177) can detect an object based on camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0242] Meanwhile, the neural processor (177) can receive camera data at a fixed frame rate, detect an object based on the camera data, and operate at a variable frame rate based on the object or output result data including the object.

[0243] Meanwhile, among the multiple virtualization machines (520 to 540), the first virtualization machine (520), which is a server virtualization machine, controls the operation of the neural processor (177).

[0244] Meanwhile, among the multiple virtualization machines (520 to 540), the second virtualization machine (530) and the third virtualization machine (540), which are guest virtualization machines, can each execute applications.

[0245] In the drawing, a second virtual machine (530) is illustrated executing an ADAS (Advanced Driver Assistance Systems) application (Nad), and a third virtual machine (540) is illustrated executing a Driver Monitoring System (DMS) application (Ndm) and an Augmented Reality (AR) application (Nar).

[0246] The first virtual machine (520) sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications running on at least one of the plurality of virtual machines (520 to 540), and if the first operation and the third operation can be processed in parallel, the first neural processor (177a) controls the first operation and the third operation to be processed in parallel, and controls the second operation to be processed after the first operation and the third operation are completed. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0247] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to sequentially process the second and fourth operations after the completion of the first and third operations, if the sharing of the operation layers is possible during the second and fourth operations, while the third operation has received a request for the fourth operation after the third operation has been requested. Accordingly, the neural processor can be operated efficiently.

[0248] Meanwhile, the first virtual machine (520) can control the arrangement of data for multiple operations within the internal memory (1620) of the first neural processor (177a) to vary when multiple operations are requested from multiple applications. Accordingly, the neural processor can be operated efficiently.

[0249] Meanwhile, the first virtual machine (520) can execute a neural system service (1110) to control at least one neural processor (177a to 177c).

[0250] Meanwhile, the neural system service (1110) can control the arrangement of data for multiple operations within the internal memory (1620) of the first neural processor (177a) to vary when multiple operations are requested from multiple applications. This enables efficient operation of the neural processor.

[0251] Meanwhile, the neural system service (1110) may execute or be equipped with a neural manager (1113) for managing at least one neural processor (177a to 177c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (177a to 177c), and a neural interface (1118) for interfacing with at least one neural processor (177a to 177c).

[0252] Meanwhile, the neural system service (1110) may further execute or provide a model container (509) that performs interface of model parameters related to the operation of the neural processor (177) and version management of learning files.

[0253] The neural manager (1113) can perform artificial intelligence model management, learning model management, camera data management, sensor data management, or command queue management.

[0254] The neural controller (1115) can determine an optimal inference method of at least one neural processor (177a to 177c), perform queue, partition, caching, or scalable coding, or control at least one neural processor (177a to 177c).

[0255] The neural interface (1118) can execute an application program interface (API) related to the accelerator of at least one neural processor (177a-177c).

[0256] Meanwhile, the interface (522) within the first virtual machine (520) can perform interfacing between the neural system service (1110) and the model container (509) or the neural system service (1110) and the shared memory (508).

[0257] Meanwhile, the interface (522) within the first virtual machine (520) can perform interfacing to the first virtual machine (520).

[0258] Meanwhile, the interface (522) within the first virtual machine (520) can perform interfacing to a vehicle driving assistance application (Nad) running within the second virtual machine (530) or a driver monitoring system application (Ndm) or an augmented reality application (Nar) running within the third virtual machine (540).

[0259] For example, an interface (522) within a first virtual machine (520) can be controlled to transmit camera data, sensor data, or voice data to a neural processor (177) using shared memory (508).

[0260] Meanwhile, the interface (522) within the first virtual machine (520) can control the transmission of result data output from the neural processor (177) and recorded in the shared memory (508) to the neural system service (1110).

[0261] Meanwhile, the interface (522) within the first virtual machine (520) can control the transmission of result data output from the neural processor (177) and recorded in the shared memory (508) to a vehicle driving assistance application (Nad) running within the second virtual machine (530) or a driver monitoring system application (Ndm) or an augmented reality application (Nar) running within the third virtual machine (540).

[0262] Meanwhile, the first virtual machine (520) may be executed based on the first operating system (620), the second virtual machine (530) may be executed based on the second operating system (630) with a high safety level, and the third virtual machine (540) may be executed based on the third operating system (640).

[0263] That is, multiple virtual machines (520 to 540) may be run based on different operating systems, or may be run based on at least two operating systems.

[0264] Meanwhile, the neural manager (1113) can manage the operating requirements of an artificial neural network-based application, control neural network weight data, and process necessary input data.

[0265] Meanwhile, the neural manager (1113) can sequentially process the optimized command queue through a hardware accelerator and transmit the operation result to the application.

[0266] Driving requirements can include the neural network's computational priorities, dependencies, and accuracy. Operational priorities are preset values, such as whether the first operation should always be processed before the second, or, in the case of safety-critical neural networks, whether the first operation should be processed first in the command queue before other candidate neural networks.

[0267] Meanwhile, neural network weight data refers to a file in which the element values ​​of each matrix are structured and stored in the process of inferring the results of a neural network calculated through a series of matrix operations.

[0268] Neural network weight data can be stored in advance as a model container (509) within the neural system service (1110) through an API call of the neural system service (1110) during the application installation process.

[0269] Meanwhile, the basic weight data loaded into the model container (509) can be automatically converted and stored in various discretization levels during the system initialization process. For example, if the basic weight is defined as FP32, it can be sub-discretized into INT8, INT16, and FP16, and a total of four weight files can be stored.

[0270] Required input data refers to input signals required for the current neural network to operate, such as vehicle speed, current location, radar, lidar, camera images, and intermediate or final calculation results of the preceding neural network.

[0271] The input data can be transmitted in real time to the shared memory (508) within the hypervisor (505) through an interface operating through the central processor (175) in the server virtualization machine.

[0272] A command queue is a memory buffer of a sequential FIFO data structure that can define a series of orders for processing artificial neural networks through hardware accelerators.

[0273] A single neural network operation request entering the command queue can be transmitted along with metadata such as the application name, the location of the application virtualization machine, the storage destination of the operation result, hardware accelerator control settings, the memory location of the input data, and the memory location information for each discretization level of the weight data.

[0274] The hardware accelerator control settings may include a unique number of the hardware accelerator in charge of the operation, the current target discretization level of the weight data (INT8, INT16, FP16, FP32, etc.), and a target neural network weight location mapping table for each hardware accelerator internal memory address.

[0275] Meanwhile, the neural controller (1115) can schedule an optimized command queue based on the requested artificial neural network operation commands and the availability of current hardware resources, and control the actual hardware accelerator to match the expected operation of the command queue.

[0276] The neural controller (1115) can receive neural network operation requirements from the neural manager (1113) and optimize the command queue.

[0277] The optimization process, in other words, can be calculated through a simulated scheduling that checks the priority, dependency, and accuracy metadata for each slot in the current command queue, and applies various queue optimization techniques (such as Partition, Caching, and Accuracy Coding) to all candidate commands in the current command queue, and finds a combination that maximizes hardware utilization and minimizes the latency of individual operation requests within a unit of time.

[0278] Based on the optimal slot location obtained in this way, a weight file (learning model) can be requested from the neural manager (1113) and loaded into the hardware internal memory.

[0279] If two different neural networks are managed as a single virtual neural network using Partition among optimization techniques and input as hardware operation requests, the start and end positions of the weights of the first operation corresponding to the address of the hardware internal memory can be recorded in a mapping table, and then the start and end positions of the weights of the second operation can be recorded in the mapping table.

[0280] Through this, the hardware accelerator performs the process of parallel processing of a virtual neural network, but the neural controller (1115) can separate the results of the operation into the results of the first operation and the second operation through a mapping table and transmit them separately to individual applications.

[0281] After completing the above initialization process, the neural controller (1115) can receive a sequential processing request of the command queue from the neural manager (1113).

[0282] At this time, the neural controller (1115) can be controlled to extract input data prepared in advance by the neural manager (1113) from the input data queue, pair the neural network weights with the corresponding input data, and perform computational processing through the hardware accelerator API.

[0283] If, unlike the initial driving requirements, the discretization level of the current neural network is changed according to a specific situation, the neural controller (1115) can perform a bitwise concanate operation that concatenates the weight conversion difference value (Delta) of the hardware internal memory to the basic weight of the current internal memory, thereby converting the discretization level of the basic weight of the internal memory in real time.

[0284] FIG. 12 is an example of a flowchart showing an operation method of a signal processing device according to an embodiment of the present disclosure.

[0285] Referring to the drawing, the first virtual machine (520) in the signal processing device (170) according to the embodiment of the present disclosure can initialize the learning model according to the execution of the first application (Nad) in the second virtual machine (530) (S1110).

[0286] Meanwhile, the first virtual machine (520) can initialize the learning model according to the execution of the second application (Ndm) in the third virtual machine (540) (S1111).

[0287] At this time, unlike FIG. 10c, steps 1110 (S1110) and 1111 (S1110) can be performed in parallel.

[0288] Next, the first virtual machine (520) within the signal processing device (170) can confirm the initialization of the confirmation model (S1112).

[0289] Meanwhile, the first virtual machine (520) can load the storage model for the first application (Nad) after confirming the initialization of the storage model.

[0290] For example, the first virtual machine (520) may receive a file related to a learning model for the first application (Nad) stored in the shared memory (508) and control the storage of the file related to the received learning model in the internal memory of at least one of the neural processors (177a to 177c).

[0291] Next, a first inference request for a first application (Nad) is transmitted to at least one of the neural processors (177a to 177c) (S1118), camera data or sensor data or voice data is transmitted to at least one of the neural processors (177a to 177c), and a first inference result can be received from at least one of the neural processors (177a to 177c) (S1118).

[0292] Meanwhile, the first virtual machine (520) within the signal processing device (170) can confirm the initialization of the confirmation model (S1113).

[0293] Meanwhile, the first virtual machine (520) can load the stored second application (Ndm) after confirming the initialization of the verification model.

[0294] For example, the first virtual machine (520) may receive a file related to a learning model for a second application (Ndm) stored in shared memory (508) and control the storage of the file related to the received learning model in the internal memory of at least one of the neural processors (177a to 177c).

[0295] Next, a second inference request for a second application (Ndm) is transmitted to at least one of the neural processors (177a to 177c) (S1117), camera data or sensor data or voice data is transmitted to at least one of the neural processors (177a to 177c), and a second inference result can be received from at least one of the neural processors (177a to 177c) (S1119).

[0296] Meanwhile, unlike FIG. 10c, steps 1112 (S1112), 1116 (S1116), 1118 (S1118), 1113 (S1113), 1117 (S1117), and 1119 (S1119) can be performed in parallel.

[0297] In this way, by processing the operations for the first application (Nad) and the operations for the second application (Ndm) in parallel and simultaneously, the execution time of multiple applications can be significantly shortened.

[0298] In particular, since multiple applications can be executed in parallel on at least one neural processor (177a to 177c), the neural processors (177a to 177c) can be operated efficiently.

[0299] Figures 13a to 25b are drawings referenced in the description of Figure 11 or Figure 12.

[0300] Figure 13a is a diagram illustrating multiple operations according to execution of multiple applications.

[0301] Referring to the drawing, the first operation (1302) may correspond to a lane recognition operation in an augmented reality application (Nar), the second operation (1304) may correspond to a face recognition operation in a driver monitoring system application (Ndm), and the third operation (1306) may correspond to an object recognition operation in a vehicle driving assistance application (Nad).

[0302] Since the amount of computation of the first operation (1302) is less than that of the second operation (1304), the size (SZa) of the first operation (1302) is illustrated in the drawing as being smaller than the size (SZb) of the second operation (1304).

[0303] Meanwhile, since the amount of calculation of the second operation (1304) is less than that of the third operation (1306), the size (SZb) of the second operation (1304) is illustrated in the drawing as being smaller than the size (SZc) of the third operation (1306).

[0304] Meanwhile, multiple operations resulting from the execution of multiple applications are performed in the first neural processor (177a), and for efficient operation performance, it is necessary to cache them using a technique optimized for the internal memory of the first neural processor (177a).

[0305] Figure 13b is a diagram illustrating various examples of caching for multiple operations.

[0306] Referring to the drawing, (a) of Fig. 13(b) shows the sequential arrangement of the first operation (PRa), the second operation (PRb), and the third operation (PRc) as Inference Queuing.

[0307] (b) of Fig. 13(b) shows the sequential arrangement of the first operation (PRa), the second operation (PRSb), and the first operation (PRaa) as a preemption.

[0308] (c) of Fig. 13(b) shows a parallel arrangement of the second operation (PRb) and the third operation (PRc) after the first operation (PRa) as partitioning.

[0309] (d) of Fig. 13(b) shows a parallel arrangement of BB operations and A1 operations (PRbb), a parallel arrangement of BB operations and A2 operations (PRbb2), and an arrangement of B operations (PRb) as Shared Backbone Caching.

[0310] (e) of Fig. 13(b) shows a parallel arrangement of an 8-bit based A operation (PRaa8) and a difference operation (PRd) after an 8-bit based A operation (PRa8) as Scalable Accuracy Coding.

[0311] Figure 13c is a diagram illustrating a processor in which multiple operations are performed.

[0312] Referring to the drawing, a plurality of operations, such as those in FIG. 13b, may be performed in part by a graphic processor (1362) within a cloud server, in part by a plurality of graphic processors (1364) within a signal processing device (170), and in part by a plurality of neural processors (1365) within the signal processing device (170).

[0313] FIG. 14 is a drawing for reference in explaining the operation when the vehicle driving assistance application (Nad) and the augmented reality application (Nar) of FIG. 11 are executed.

[0314] Referring to the drawing, a first application (Nad) running in a second virtual machine (530) within a central processor (175) can request hardware acceleration operation (SF1) by utilizing an API of a neural system service (1110) within a first virtual machine (520).

[0315] Meanwhile, a second application (Nar) running in a third virtual machine (540) within a central processor (175) can request hardware acceleration operations by utilizing the API of the neural system service (1110) within the first virtual machine (520).

[0316] Meanwhile, the neural manager (1113) within the first virtual machine (520) can receive the necessary weight files from the model container (509) after confirming the operating requirements of the requested first application (Nad) and second application (Nar), and process various candidate discretization processes (SF2).

[0317] Meanwhile, the neural manager (1113) within the first virtual machine (520) can request optimization of the current command queue and the command of the first operation of the first application (Nad) and the command of the second operation of the second application (Nar) through the neural controller (1115).

[0318] Meanwhile, the neural controller (1115) within the first virtual machine (520) can determine the optimal slot arrangement through simulated scheduling of all slots of the existing command queue, the new first operation command, and the second operation command (SF3).

[0319] For example, if the input image data of the first application (Nad) and the second application (Nar) are the same front camera data, and the first operation of the first application (Nad) has a higher priority than the second operation of the second application (Nar) and must be executed first, the first virtual machine (520) may store the intermediate result of the operation of the backbone layer of the first neural network for performing the first operation in the internal memory, and when processing the operation of the second neural network for the second operation, reuse the intermediate result of the operation of the backbone layer of the first neural network, thereby eliminating duplicate operations, and may arrange a command queue.

[0320] As another example, the first virtual machine (520) may control the discretization level of the second operation to be lowered from the first level to the second level when the current driving route is a deserted country road, the electric vehicle is operating in energy saving mode due to insufficient remaining battery, or an object is not detected for a predetermined period of time based on the result of the first operation.

[0321] Specifically, the first virtual machine (520) can control the level of discretization of the second operation to be lowered from the first level, 16-bit integer operation (INT16), to the second level, 8-bit integer operation (INT8). Accordingly, the operation accuracy is lowered, which ultimately reduces power consumption.

[0322] Meanwhile, when the initialization process of the neural manager (1113) is completed, the first virtual machine (520) receives input data (e.g., camera data, sensor data, or voice data) in real time through the interface (522) and loads it into the input queue of the neural processor (177a to 177c), and the neural controller (1115) sequentially reads the input data loaded into the neural processor (177a to 177c), and performs hardware acceleration operations through the neural interface (1118) (SF4).

[0323] Meanwhile, when a neural processor (177a to 177c) completes an operation for one input pair, the neural manager (1113) can read the operation result data from the internal memory in the neural processor (177a to 177c) and transmit it to the first application (Nad) in the second virtual machine (530) and the second application (Nar) in the third virtual machine (540) through the interface (522) (SF5).

[0324] Meanwhile, if it is confirmed through the neural interface (1118) that a hardware error has occurred during the process of initializing the neural processor hardware through the neural controller (1115), the requested neural network operation cannot be performed through the current neural processor (177a to 177c), and therefore, the neural manager (1113) can check the availability of other hardware accelerators supported by the signal processing device (170).

[0325] For example, when an attempt is made to initialize a neural processor (177a to 177c) for an AR operation and the target neural processor (177a to 177c) responds with a no response or initialization failure message, the neural manager (1113) may sequentially attempt to initialize a plurality of other hardware accelerators supported by the signal processing device (170).

[0326] In addition, the neural manager (1113) can change the operating requirements of the current neural network to the use of the corresponding hardware resources if there are available resources among the other multiple hardware accelerators in the signal processing device (170).

[0327] The scope of the changes here may include changing the ID of the target hardware accelerator and setting the default weight file of the neural network to the quantization level supported by the target hardware accelerator.

[0328] Meanwhile, in the event that a situation arises where all hardware accelerators within the signal processing device (170) cannot be used, the neural manager (1113) can proceed with a simulation to determine whether the currently requested neural network operation can be processed on the central processor (175).

[0329] Here, the mock simulation can be calculated based on the time consumed per single operation on the current central processor (175), taking into account both the size of the weights, the total number of operations, and the maximum processing time required.

[0330] Meanwhile, in the case where the total number of weight operations of an artificial neural network with a very deep layer depth cannot achieve the target frames per second (FPS) with only the processing capability of the central processor (175) of the signal processing device (170), the neural manager (1113) can optimize the command queue by giving priority to parallel processing operations through a hardware accelerator such as a neural processor (177a to 177c).

[0331] For example, the neural manager (1113) may not design frequently used menus, frequently set air conditioner settings, etc. with a deep artificial neural network, but may use a simple pattern recognizer with a relatively small amount of computation, such as an SVM (Support Vector Machine), and the neural system service (1110) may proceed with priority setting of the central processor (175) rather than the neural processor (177a to 177c) when optimizing the command queue.

[0332] Meanwhile, if the complexity of the artificial neural network and the set of neural network commands used are in an application that is not effectively supported by the neural processor (177a to 177c), the neural manager (1113) may consider, when optimizing the command queue, giving priority to allocating a module supported by a general-purpose parallel processing function, such as a graphics processor (178), as a candidate hardware accelerator.

[0333] Figure 15 is a flowchart showing the operation of Figure 14.

[0334] Referring to the drawing, according to the execution of the first application (Nad) in the second virtual machine (530), the second virtual machine (530) can request the first virtual machine (520) to initialize the learning model (S1510).

[0335] Next, the neural manager (1113) in the first virtual machine (520) can control the learning model to be initialized in response to a learning model initialization request from the first application (Nad) in the second virtual machine (530) (S1511).

[0336] Meanwhile, the neural manager (1113) in the first virtual machine (520) can transmit a command queue optimization request to the neural controller (1115) (S1512).

[0337] Next, the neural manager (1113) within the first virtual machine (520) can request the confirmation model data through the model container (509) or the neural interface (1118) (S1511).

[0338] In response to this, the model container (509) or the neural interface (1118) can transmit the learning model data stored in the model container (509) or the learning model data stored in the shared memory (505) to the neural manager (1113) (S1515).

[0339] Meanwhile, the neural manager (1113) within the first virtual machine (520) can perform accuracy coding when necessary based on the received stored learning mode data (S1516).

[0340] Next, the neural manager (1113) within the first virtual machine (520) can transmit learning model data to an optimized slot of the command queue (S1518).

[0341] Meanwhile, the neural controller (1115) within the first virtual machine (520) can control the neural processor (177) to be initialized (S1519).

[0342] Next, the neural controller (1115) within the first virtual machine (520) can control the loading of learning model data into the internal memory of the neural processor (177) (S1520). This step may also be referred to as pre-loading.

[0343] Next, the neural manager (1113) within the first virtual machine (520) can initialize the reception of input data required for inference (S1522).

[0344] And, the neural interface (1118) within the first virtual machine (520) may request transmission or subscription of input data, such as camera data, sensor data, or voice data, from a sensor device (700), a camera device (195), or a shared memory (508) (S1524)

[0345] Meanwhile, the second virtual machine (530) can confirm the completion of learning model initialization (S1528), and after confirming the completion of initialization, can request inference (S1530).

[0346] In response to this, the neural manager (1113) within the first virtual machine (520) may receive an initial request and transmit a request for sharing input data to the neural interface (1118) (S1532).

[0347] Next, the neural interface (1118) within the first virtual machine (520) can collect and transmit input data, such as camera data, sensor data, or voice data, from a sensor device (700), a camera device (195), or a shared memory (508) (S1534).

[0348] Next, the neural manager (1113) within the first virtual machine (520) can control the received input data to be transmitted to the input queue (S1536) and arranged in the command queue for processing (S1538).

[0349] Next, the neural controller (1115) within the first virtual machine (520) can control the input queue to be loaded into the internal memory of the neural processor (177) (S1539) and control the command queue to be loaded into the internal memory of the neural processor (177) (S1540).

[0350] At this time, processing of a bitwise scaler for accuracy coding can be performed (S1542).

[0351] Next, the neural controller (1115) within the first virtual machine (520) can transmit a control command to the neural processor (177) (S1544).

[0352] Next, the operation result for inference is received from the neural processor (177), and the inference result data can be controlled to be loaded into the output queue of the internal memory of the neural processor (177) (S1542).

[0353] Next, the neural manager (1113) within the first virtual machine (520) can receive inference result data through an output queue (S1546) and verify learning model requirements (S1548).

[0354] Next, the neural interface (1118) within the first virtual machine (520) can be controlled to store inference result data and verification history data in the shared memory (508) (S1550)

[0355] Next, the first application (Nad) within the second virtual machine (530) can receive inference result data through shared memory (508) (S1552).

[0356] In this way, inference requests from a single application can be processed quickly.

[0357] Furthermore, since multiple inference requests from multiple applications can be performed through parallel operations, it becomes possible to efficiently operate a neural processor (177), etc.

[0358] Figure 16 is a flowchart showing optimization of the command queue.

[0359] Referring to the drawing, the neural manager (1113) within the first virtual machine (520) can check the priority, dependency, and accuracy requirements for each slot or each operation of the current command queue (S1610).

[0360] Next, the neural manager (1113) within the first virtual machine (520) can perform simulation scheduling based on the requirements of the requested learning model (S1612).

[0361] Next, the neural manager (1113) within the first virtual machine (520) can determine the optimal queue slot location or optimal queue operation location (S1616) for a processable slot or operation (S1614) based on simulated scheduling.

[0362] Meanwhile, the neural manager (1113) within the first virtual machine (520) determines whether the model is already cached (S1618), and if not, determines whether it can be loaded into the remaining internal memory of the neural processor (177) (S1620), and if so, controls to load the learning model data in advance (S1622).

[0363] Meanwhile, if not applicable in step 1620 (S1620), loading of learning model data can be controlled to be performed in step 1546 (S1546) of FIG. 15 (S1624). This step may also be referred to as post-loading.

[0364] Figures 17a and 17b illustrate various examples of queuing.

[0365] Figure 17a is a diagram showing sequential queuing.

[0366] Referring to the drawing, the first virtual machine (520) can receive requests for multiple operations (RRa, RRb1, RRbc, RRb2) from multiple applications running on at least one of the multiple virtual machines (520 to 540).

[0367] Meanwhile, as shown in the drawing, the first virtual machine (520) can control a plurality of operations (RRa, RRb1, RRbc, RRb2) to be sequentially arranged in the internal memory (1710) within the first neural processor (177a).

[0368] According to the sequential operation method according to this sequential arrangement, the first neural processor (177a) cannot be operated efficiently, and accordingly, there is a disadvantage in that it takes a considerable amount of time until the completion of multiple operations (RRa, RRb1, RRbc, RRb2).

[0369] Figure 17b is a diagram showing an example of optimized queuing.

[0370] Referring to the drawing, the first virtual machine (520) can receive requests for multiple operations (RRa, RRb1, RRbc, RRb2) from multiple applications running on at least one of the multiple virtual machines (520 to 540).

[0371] Meanwhile, as shown in the drawing, the first virtual machine (520) sequentially receives a request for a first operation (RRa), a request for a second operation (RRb1), and a request for a third operation (RRbc), and, if parallel processing is possible for the first operation (RRa) and the third operation (RRbc), controls the first operation (RRa) and the third operation (RRbc) to be processed in parallel, and controls the second operation (RRb1) to be processed after the completion of the first operation (RRa) and the third operation (RRbc).

[0372] That is, the first virtual machine (520) can place the first operation (RRa) and the third operation (RRbc) in parallel in the internal memory (1710) within the first neural processor (177a), and place the second operation (RRb1) after the first operation (RRa) and the third operation (RRbc). Accordingly, the neural processor can be operated efficiently.

[0373] Meanwhile, the request for the first operation (RRa) may be an operation request from an ADAS application, and the request for the third operation (RRbc) may be an operation request from an AR application.

[0374] For example, if the first operation (RRa) has the highest priority and can complete the entire operation using only 70% of the hardware capacity, and the third operation (RRbc) can complete the entire operation using only 30% of the hardware capacity, the first virtual machine (520) can virtually integrate the first operation (RRa) and the third operation (RRbc) as a single neural network and request the hardware to perform the operation. Accordingly, the neural processor can be operated efficiently.

[0375] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to sequentially process the second operation (RRb1) and the fourth operation (RRb2) after the completion of the first operation (RRa) and the third operation (RRbc), if the sharing of the operation layers during the second operation (RRb1) and the fourth operation (RRb2) is possible while the third operation (RRbc) has received a request for the fourth operation (RRb2).

[0376] That is, the first virtual machine (520) can arrange the first operation (RRa) and the third operation (RRbc) in parallel in the internal memory (1710) within the first neural processor (177a), and arrange the second operation (RRb1) and the fourth operation (RRb2) sequentially after the first operation (RRa) and the third operation (RRbc). Accordingly, the neural processor can be operated efficiently.

[0377] Meanwhile, the request for the second operation (RRb1) may be a request for a voice operation for driver monitoring, and the request for the fourth operation (RRb2) may be a request for a camera data-based action detection operation for driver monitoring.

[0378] For example, if a driver looks at an air conditioning control menu on a vehicle display and utters a voice command such as “lower the temperature,” the first neural processor (177a) maintains the intermediate operation result for the feature point information in the voice signal in the internal memory of the first neural processor (177a) through the second operation (RRb1), and then provides the feature point vector information of the corresponding voice signal through the fourth operation (RRb2), thereby detecting from camera data whether the driver’s gaze is actually focused on the air conditioning control menu.

[0379] Therefore, the detection target for the fourth operation (RRb2) is limited to patterns related to air conditioning control, rather than all driver behavior patterns (Constraint), reducing the computational load. This allows for efficient operation of the neural processor.

[0380] Meanwhile, in the drawing, the above-described optimization queuing technique can secure free space, such as an Arm, in the internal memory (1710) within the first neural processor (177a), and additional operations can be performed through this free space (Arm). This enables efficient operation of the neural processor. Furthermore, power consumption can be reduced.

[0381] Meanwhile, the first virtual machine (520) can control the discretization level of the second operation (RRb1) to be lowered from the first level to the second level when in energy saving mode or when no object is detected for a predetermined period of time based on the results of the first operation (RRa). Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0382] Meanwhile, with respect to FIG. 17b, the first virtual machine (520) can control the first neural processor (177a) to perform the first to third operations (RRa, RRb1, RRbc) when the central processor (175) cannot perform the first to third operations (RRa, RRb1, RRbc). Accordingly, the neural processor can be operated efficiently.

[0383] Meanwhile, the first virtual machine (520) can perform the first to third operations (RRa, RRb1, RRbc) on the central processor (175), and if the first operation (RRa) and the third operation (RRbc) can be processed in parallel, the central processor (175) can be controlled to process the first operation (RRa) and the third operation (RRbc) in parallel, and can be controlled to process the second operation (RRb1) after the first operation (RRa) and the third operation (RRbc) are completed. Accordingly, the neural processor can be operated efficiently.

[0384] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to store the result data of the second operation (RRb1) in the internal memory (1620), if the second operation (RRb1) includes an action recognition operation based on audio data from a microphone, and the fourth operation (RRb2) includes an eye recognition operation based on camera data, and can control the fourth operation (RRb2) to be processed using at least a portion of the result data stored in the internal memory (1620). Accordingly, the neural processor can be operated efficiently.

[0385] Figure 18a is a drawing explaining various arrangements of the first operation and the second operation of Figure 17b.

[0386] Referring to the drawing, (a) of FIG. 18a illustrates an example of an arrangement of the first operation (Qra) and the third operation (Qrb) based on a command queue.

[0387] Meanwhile, the first virtual machine (520) can control the first operation (Qra) and the third operation (Qrb) to be arranged in parallel in a one-dimensional direction.

[0388] Accordingly, the first neural processor (177a) can perform parallel operations in the one-dimensional direction for the first operation (Qra) and the third operation (Qrb).

[0389] Figure 18a (b) illustrates an example of the arrangement of the first operation (Qrc) and the third operation (Qrd) within the internal memory (1720) within the first neural processor (177a).

[0390] Meanwhile, the first virtual machine (520) can control the first operation (Qrc) and the third operation (Qrd) to be arranged in parallel in a two-dimensional direction within the internal memory (1720) of the first neural processor (177a).

[0391] Accordingly, the calculator (1630 in FIG. 18b) within the first neural processor (177a) can perform one of the first operation (Qrc) and the third operation (Qrd) (e.g., the first operation (Qrc)) in the first region, and perform the other one (e.g., the third operation (Qrd)) in the second region surrounding the first region.

[0392] Figure 18b is a diagram illustrating an internal block diagram of a neural processor.

[0393] Referring to the drawing, the neural processor (1117) can be controlled by the neural controller (1115) and the neural interface (1118) within the first virtual machine (520).

[0394] Meanwhile, the neural processor (1117) may include an internal memory (1620) that stores weight data or input data, and a calculator (1630) that performs a calculation based on data from the internal memory (1620).

[0395] Meanwhile, the neural processor (1117) may further include an interface (1610) for exchanging data with the neural interface (1118), a bit operator (1622) for outputting quantization weights, an input buffer (1625), a weight buffer (1627), and an output buffer (1629).

[0396] The internal memory (1620) can store 8-bit integer weight data (INT8 weight), 16-bit difference data (INT16 delta), image data, etc.

[0397] The bit operator (1622) can output weight data of a first length or weight data of a second length based on quantization mode information from the neural controller (1115).

[0398] For example, the bit operator (1622) can output weight data of a first length based on first quantization mode information from the neural controller (1115), or can output weight data of a second length based on second quantization mode information.

[0399] Specifically, the bit operator (1622) can output weight data of an 8-bit integer based on the first quantization mode information, or can output weight data of a 16-bit integer based on the weight data of an 8-bit integer and the difference data for 16 bits based on the second quantization mode information.

[0400] The weight buffer (1627) can perform memory address mapping based on weight data from the bit operator (1622).

[0401] Meanwhile, the operator (1630) may be equipped with multiple parallel operation processing blocks.

[0402] For example, the operator (1630) may perform one of the first operation (Qrc) and the third operation (Qrd) of (b) of FIG. 18a (e.g., the first operation (Qrc)) in an operation processing block corresponding to a first region, and may perform the other one (e.g., the third operation (Qrd)) in an operation processing block corresponding to a second region surrounding the first region.

[0403] For example, if the first operation (Qrc) and the third operation (Qrd) of (b) of FIG. 18a are integrated into a single virtual neural network and can be simultaneously performed as parallel processing operations, the neural controller (1115) can, based on the Partition technique, check the start and end positions of the weights of the first operation (Qrc) and the third operation (Qrd) in the hardware internal memory (1620) through the neural interface (1118).

[0404] Meanwhile, the neural controller (1115) can check the address of the internal memory (1620) where the input data of the first operation (Qrc) and the third operation (Qrd) exist.

[0405] Meanwhile, the neural controller (1115) can input the inputs of the first operation (Qrc) and the third operation (Qrd) separately at the pin array level according to the mapping table to the inputs of the parallel operation processing block.

[0406] Meanwhile, the neural controller (1115) can also separate the results of the first operation (Qrc) and the third operation (Qrd) into the results of the first operation (Qrc) and the third operation (Qrd).

[0407] Meanwhile, the first neural processor (177a) may perform at least one of the first to third operations (RRa, RRb1, RRbc) based on input data of the first length according to the first quantization mode, as shown in FIG. 17b, and may perform at least one of the first to third operations (RRa, RRb1, RRbc) based on input data of the second length that is longer than the first length according to the second quantization mode. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0408] Meanwhile, the first neural processor (177a) performs at least one of the first operation to the third operation (RRa, RRb1, RRbc) based on the input data of the first length according to the first quantization mode, as shown in FIG. 17b, and performs a concatenation operation based on the input data of the first length and the differential data corresponding to the input data of the first length according to the second quantization mode, and generates input data of the second length longer than the first length according to the concatenation operation, and performs at least one of the first operation to the third operation (RRa, RRb1, RRbc) based on the input data of the second length. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0409] At this time, the input data of the first length may be 8-bit integer data, the input data of the second length may be 16-bit integer data, and the concatenation operation may be a concatenate operation.

[0410] Meanwhile, the first neural processor (177a) may perform at least one of the first to third operations (RRa, RRb1, RRbc) according to the second quantization mode, as shown in FIG. 17b, and then, if the number of feature points for the detected object is less than or equal to a predetermined value according to the result data of at least one of the first to third operations (RRa, RRb1, RRbc), the first neural processor may switch to the first quantization mode. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0411] Meanwhile, the first neural processor (177a) may perform at least one of the first to third operations (RRa, RRb1, RRbc) according to the first quantization mode or the daytime mode when the vehicle is driven during the daytime, as shown in FIG. 17b, and may perform at least one of the first to third operations (RRa, RRb1, RRbc) according to the second quantization mode or the nighttime mode when the vehicle is driven at night. Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0412] Figures 19a to 19e are drawings for reference in the description of the quantization mode.

[0413] Figure 19a is a diagram explaining the generation and storage of quantization weights.

[0414] Referring to the drawing, the original weight file (1710) of the neural network can be defined as a 32-bit floating point (FP32) version that has not undergone quantization, as in (a) of Fig. 19a.

[0415] For example, the neural network weights that have completed training can be output as 32-bit floating point (FP32).

[0416] Meanwhile, an application running on a virtual machine can transmit a weight source file (170) to a model container (509) through the API of the neural system service (1110).

[0417] Meanwhile, the neural manager (1113) can quantize the original weight file transmitted to the model container (509) with a precision supported by various hardware accelerators supported by the signal processing device (170), as shown in (b) of FIG. 19a.

[0418] For example, the first neural processor (177a) may support a 16-bit floating point (1716), the second neural processor (177b) may support an 8-bit integer (1718), and the graphics processor (178) may support a 16-bit floating point (1715).

[0419] As another example, the neural processor (177) may support INT8 and INT16, and the graphics processor (178) may support FP16 and FP32.

[0420] Meanwhile, the neural manager (1113) can calculate the difference data between the quantized weight data to obtain the data values ​​required for conversion from the lower stage to the upper stage.

[0421] For example, the difference bits between the upper 16-bit integer (INT8) and the lower 8-bit integer (INT8) pattern can be defined as difference data.

[0422] Here, the differential data (Delta) not only plays a role in varying the Precision, but can also include the learning conditions of the input image, such as the differential data of the model weights learned for daytime images and the differential data of the model weights learned for nighttime images, at the time of learning the input image.

[0423] Meanwhile, after calculating all differential data values ​​for floating point and integer formats, the lower quantization weights that serve as the basis for each numerical format and the differential data required for conversion to the upper format can be stored as a pair in the model container (509).

[0424] For example, the model container (509) can store weight data of an 8-bit integer (INT8) and difference data for 16-bit conversion (INT8→16) as a pair (Ar1), as in (c) of Fig. 19a.

[0425] As another example, the model container (509) can store weight data of 16-bit floating point (FP16) and difference data for 32-bit conversion (FP16→32) as a pair (Ar2), as in (c) of FIG. 19a.

[0426] Figure 19b is a diagram explaining quantization transformation.

[0427] Referring to the drawing, as in (a) of FIG. 19b, when a pair stored in a model container (509) includes, for example, weight data of an 8-bit integer (INT8) and corresponding difference data, converting the precision from an 8-bit integer to a 16-bit integer can generate 16-bit integer data by attaching 8-bit difference data to 8-bit integer data.

[0428] This type of concatenation operation can generally be called a concatenation operation.

[0429] (b) of Fig. 19b illustrates generating 16-bit integer data of '0011101011100000' by concatenation operation of '00111010', which is 8-bit integer (INT8) data, and '11100000', which is 8-bit difference data.

[0430] Figure 19c illustrates a quantization transform related to the present disclosure.

[0431] Referring to the drawing, when 8-bit integer data is stored in the internal memory (1732) within the neural processor (177) and 8-bit weight data (1731) is received, the operator (1734) can perform an operation based on the 8-bit integer data (1734) and output an operation result (Srt) according to an operation request (Srg).

[0432] FIG. 19d illustrates an example of quantization transform according to an embodiment of the present disclosure.

[0433] Referring to the drawing, when 8-bit integer data and 8-bit differential data are stored in the internal memory (1732) within the neural processor (177), 8-bit weight data (1731) and 8-bit differential data are received, and a calculation request (Srg) and second quantization mode information among quantization mode information (Sgm) including the first quantization mode or the second quantization mode are received, the neural processor (177) performs a connection operation through a summator (1746) to output 16-bit integer data.

[0434] And, the calculator (1734) can perform an operation based on 16-bit integer data (1734) and output an operation result (Srt).

[0435] Alternatively, when first quantization mode information is received, the neural processor (177) outputs 8-bit integer data without performing a connection operation.

[0436] And, the operator (1734) can perform an operation based on 8-bit integer data and output an operation result (Srt).

[0437] Accordingly, the neural processor (177) according to the embodiment of the present disclosure is capable of scalable coding and scalable quantization-based operations, enabling efficient operation. Furthermore, power consumption, etc. can be reduced when performing operations based on 8-bit integer data.

[0438] FIG. 19e illustrates another example of quantization transform according to an embodiment of the present disclosure.

[0439] Referring to the drawing, Fig. 19e, unlike Fig. 19d, the server virtualization machine (1115) performs a concatenation operation through an adder (1746) to output 16-bit integer data and stores the 16-bit integer data in the internal memory (1732) within the neural processor (177).

[0440] In addition, the calculator (1734) within the neural processor (177) can perform an operation based on 16-bit integer data (1734) and output an operation result (Srt).

[0441] Meanwhile, Scalable Accuracy Coding such as Fig. 19d or Fig. 19e can convert the output precision of weight quantization using only differential data operations at upper and lower stages.

[0442] Figure 20a illustrates that the first quantization mode or the second quantization mode is performed depending on the vehicle driving situation.

[0443] Referring to the drawing, the first virtual machine (520) can collect the neural network operation results of the previous point in time (S1810).

[0444] Next, the first virtual machine (520) can judge the surrounding situation based on the neural network operation results of the previous point in time (S1815).

[0445] Meanwhile, the first virtual machine (520) can determine whether low-power operation is required based on the surrounding situation (S1820).

[0446] If applicable, the first virtual machine (520) can control the neural processor (177) to operate in the first quantization mode (S1820).

[0447] For example, the first virtual machine (520) can control the processor (177) to operate in the first quantization mode when the number of feature points for surrounding objects (vehicles or pedestrians) is less than a predetermined value as a result of the neural network operation at the previous point in time.

[0448] The first quantization mode at this time may be an 8-bit integer-based quantization mode.

[0449] As another example, the first virtual machine (520) can control the neural processor (177) to operate in the first quantization mode when driving on an intercity or country road with a traffic volume below a predetermined level based on GPS-based location information and map-based location information.

[0450] As another example, the first virtual machine (520) can control the neural processor (177) to operate in the first quantization mode when the current vehicle driving speed is below a predetermined speed.

[0451] As another example, the first virtual machine (520) can control the neural processor (177) to operate in the first quantization mode when the vehicle cannot reach the driving distance to the destination or the distance to a nearby charging station based on the remaining battery of the vehicle.

[0452] Next, the first virtual machine (520) can convert weight quantization based on the first quantization mode (S1827).

[0453] And, the first virtual machine (520) can be controlled to operate in a low power mode based on the first quantization mode (S1829).

[0454] Next, the first virtual machine (520) can control the input of learning model data into an optimized slot of the command queue (S1830). Accordingly, operations based on the first quantization mode can be performed.

[0455] Meanwhile, in step 1820 (S1820), if it is not a low power mode, the first virtual machine (520) can control the neural processor (177) to operate in the second quantization mode (S1826).

[0456] For example, the first virtual machine (520) can control the processor (177) to operate in the second quantization mode when, based on the neural network operation result at a previous point in time, the number of feature points for surrounding objects (vehicles or pedestrians) exceeds a predetermined value or the detection rate of surrounding objects (vehicles or pedestrians) exceeds a set value.

[0457] The second quantization mode at this time may be a 16-bit integer-based quantization mode.

[0458] As another example, the first virtual machine (520) can control the neural processor (177) to operate in the second quantization mode when driving in a city or highway with a traffic volume exceeding a predetermined amount based on GPS-based location information and map-based location information.

[0459] As another example, the first virtual machine (520) can control the neural processor (177) to operate in the second quantization mode when the current vehicle driving speed exceeds a predetermined speed.

[0460] Next, the first virtual machine (520) can convert weight quantization based on the second quantization mode (S1827).

[0461] And, the first virtual machine (520) can be controlled to operate in a high-performance mode based on the second quantization mode (S1828).

[0462] Next, the first virtual machine (520) can control the input of learning model data into an optimized slot of the command queue (S1830). Accordingly, operations based on the second quantization mode can be performed.

[0463] Meanwhile, according to FIG. 20a, the quantization mode can be varied based on vehicle speed or position information or the number of feature points in the image.

[0464] For example, when driving, if the number of feature points for surrounding objects (vehicles or pedestrians) in the camera image is N or less, and the environmental factors that need to be focused on for safe driving are small, the first virtual machine (520) can control the operation mode of the neural processor (177) to be switched from the second quantization mode based on 16-bit integers to the first quantization mode based on 8-bit integers. Accordingly, power consumption can be reduced.

[0465] As another example, if no object is detected, the first virtual machine (520) operates in the first quantization mode, and if the vehicle enters the city based on location information from GPS or location information within a map, the first virtual machine (520) can be controlled to switch from the first quantization mode based on 8-bit integers to the second quantization mode based on 16-bit integers. Accordingly, the accuracy of the operation can be improved.

[0466] As another example, when driving in a city but the vehicle speed is below the reference speed, the first virtual machine (520) can control the operation mode of the neural processor (177) to be switched from a second quantization mode based on 16-bit integers to a first quantization mode based on 8-bit integers. Accordingly, power consumption can be reduced.

[0467] Figure 20b illustrates that day mode or night mode is performed.

[0468] Referring to the drawing, the first virtual machine (520) can collect sensor data and neural network operation results from a previous point in time (S1839).

[0469] For example, the first virtual machine (520) can collect sensor data such as current time data, headline ignition data, indoor / outdoor light sensor data, and neural network operation results from a previous point in time.

[0470] Next, the first virtual machine (520) can judge the surrounding situation based on sensor data and the neural network operation results of the previous point in time (S1842).

[0471] Meanwhile, the first virtual machine (520) can determine whether daytime operation is necessary based on the surrounding situation (S1843).

[0472] If applicable, the first virtual machine (520) can control the neural processor (177) to operate in the second quantization mode or the day mode.

[0473] The second quantization mode at this time may be a 16-bit integer-based quantization mode.

[0474] Next, the first virtual machine (520) can convert weight quantization based on the second quantization mode (S1845).

[0475] And, the first virtual machine (520) can be controlled to operate on weekly learning weights based on the second quantization mode (S1846).

[0476] Next, the first virtual machine (520) can control the input of learning model data into an optimized slot of the command queue (S1850). Accordingly, operations based on the second quantization mode can be performed.

[0477] Meanwhile, in step 1843 (S1843), if it is not daytime operation, the first virtual machine (520) determines that nighttime operation is required and can control the neural processor (177) to operate in the first quantization mode or nighttime mode.

[0478] The first quantization mode at this time may be an 8-bit integer-based quantization mode.

[0479] Next, the first virtual machine (520) can convert weight quantization based on the first quantization mode (S1845).

[0480] And, the first virtual machine (520) can be controlled to operate on the night-time learning weights based on the first quantization mode (S1848).

[0481] Meanwhile, if additional data for the night is required, the first virtual machine (520) transmits a request to receive additional data for the night (S1849) and can receive additional data for the night based on this.

[0482] The additional night-time data at this time may be IR camera data rather than the existing RGB camera data.

[0483] Furthermore, the first virtual machine (520) can control the input of learning model data into an optimized slot of the command queue based on additional night-time data, etc. (S1850). Accordingly, operations based on the first quantization mode can be performed.

[0484] Meanwhile, referring to FIG. 20b, the first virtual machine (520) can control the neural processor (177) to operate in night mode when the current time is a night time zone (e.g., 6 PM) and the vehicle headlights are automatically turned on by a light sensor. The night mode at this time may be the first quantization mode.

[0485] As another example, the first virtual machine (520) can control the neural processor (177) to operate in night mode when the number of feature points for surrounding objects (vehicles or pedestrians) as a result of a neural network operation at a previous point in time is less than a predetermined value.

[0486] As another example, the first virtual machine (520) can control the neural processor (177) to operate in night mode when the indoor lights are turned on based on the current time or an indoor light sensor.

[0487] FIG. 20c illustrates the execution of the operation mode of the first neural processor or the operation mode of the second neural processor.

[0488] Referring to the drawing, the first virtual machine (520) can collect sensor data and neural network operation results from a previous point in time (S1852).

[0489] In particular, the first virtual machine (520) can collect navigation route data, driving speed data, and neural network operation results from a previous point in time.

[0490] Next, the first virtual machine (520) can judge the situation based on sensor data and the neural network operation results of the previous point in time (S1855).

[0491] Meanwhile, the first virtual machine (520) can determine whether the first operation is necessary based on the situation judgment (S1859).

[0492] If applicable, the first virtual machine (520) can control the first neural processor (177a) to operate for the first operation (S1862).

[0493] Meanwhile, in step 1859 (S1859), if a second operation is required in addition to the first operation, the first virtual machine (520) may request additional input data for the additional operation (S1863) and control the second neural processor (177b) to operate for the second operation based on the additional input data (S1865).

[0494] And, the first virtual machine (520) can add the first operation result of the first neural processor (177a) and the operation result of the second neural processor (177b) and output the added result (S1870).

[0495] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to perform a first operation on the front camera and the second neural processor (177b) to perform a second operation on the surrounding camera.

[0496] In particular, the first virtual machine (520) can control the first neural processor (177a) to perform the first operation on the basic front camera without always using multiple vehicle exterior cameras, and then, when high-performance operation is required, to perform the second operation on the front exterior surrounding cameras, so that the second neural processor (177b) can perform the second operation.

[0497] For example, when driving on a highway and using cruise control, if the cruise control at a previous point in time determines that the vehicle is currently maintaining its lane, the first virtual machine (520) can control the vehicle to stop calculating camera data from surrounding cameras, or switch the calculation of camera data from surrounding cameras to the first quantization mode, which is a low-power mode, and perform the calculation of camera data from the front camera to the second quantization mode, which is a high-performance mode.

[0498] Meanwhile, if a condition is created that requires a lane change, such as when the preceding vehicle is delayed compared to surrounding vehicles, the first virtual machine (520) can control the second neural processor (177b) to perform a second operation on camera data from surrounding cameras.

[0499] Meanwhile, the first virtual machine (520) controls the second neural processor (177b) to stop or perform calculations on camera data from surrounding cameras in the first quantization mode when the neural network detecting the shape of the AR carpet determines that the moving is straight up to a predetermined distance, and the first neural processor (177a) can control calculations on camera data for the front camera to be performed in the second quantization mode, which is a high-performance mode.

[0500] Meanwhile, the first virtual machine (520) can control the second neural processor (177b) to perform calculations on camera data from surrounding cameras so that the shape of the intersection can be accurately identified from multiple angles in order to display an accurate steering position in AR when vehicle steering AR guidance, such as a left turn or a right turn, is required within a predetermined distance of the intersection.

[0501] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to perform a first operation for preliminary detection of a starting word among voice data, and the second neural processor (177b) to perform a second operation for subsequent conversation detection.

[0502] For example, the first virtual machine (520) can control the first neural processor (177a) to perform operations for recognizing user voice commands such as “Hi LG” and conducting a conversation with the user.

[0503] At this time, if it is difficult to detect the starting word due to ambient noise such as passenger noise or vehicle road noise, the first virtual machine (520) can control the second neural processor (177b) to perform noise-canceling operations.

[0504] Meanwhile, the first virtual machine (520) can control the second neural processor (177b) to perform an entire voice recognition operation to detect a conversation and request with the user when a activating word is recognized through the first neural processor (177a), and can control the first neural processor (177a) to perform an operation to detect the recognition of activating words of other users.

[0505] Meanwhile, the first virtual machine (520) can control the first neural processor (177a) to perform a first user's ignition word detection operation and the second neural processor (177b) to perform a second user's ignition word detection operation when the vehicle is personalized with multiple user accounts.

[0506] Figure 21 is a diagram showing parallel processing between a signal processing device and an external device.

[0507] Referring to the drawing, a signal processing device (170) according to another embodiment of the present disclosure controls, when an external device is connected, to transmit data to a neural processor within the external device to perform a calculation.

[0508] Specifically, a signal processing device (170) according to another embodiment of the present disclosure includes a central processor (175) and a neural processor (177), and the central processor (175) executes a first virtual machine (520), which is a server virtualization machine, on a hypervisor (505).

[0509] Meanwhile, the first virtual machine (520) transmits data to the second neural processor within the external device when the external device is connected. This enables efficient operation of the neural processor.

[0510] Meanwhile, the first virtual machine (520) can perform load balancing for load distribution between the neural processor and the second neural processor. Accordingly, the neural processor can be operated efficiently.

[0511] Meanwhile, the first virtual machine (520) can control the download, installation, and execution of an application for controlling an external device from the server when an external device is connected. This enables stable communication with the external device.

[0512] Meanwhile, the first virtual machine (520) can receive result data from an external device and control the transmission of the result data to the neural processor (177a). Accordingly, the neural processor can be operated efficiently.

[0513] Meanwhile, the first virtual machine (520) can receive result data and battery information from an external device and control the storage of the result data in shared memory (508) within the hypervisor (505). This enables the sharing of the result data. Ultimately, this enables efficient operation of the neural processor.

[0514] In the drawing, the second signal processing device (170b) is exemplified as an external device, but is not limited thereto, and a mobile terminal or server, etc. may also be used.

[0515] Meanwhile, the first virtual machine (520) can execute a neural system service (1110) to control at least one neural processor (177a to 177c).

[0516] Meanwhile, the neural system service (1110) may execute or be equipped with a neural manager (1113) for managing at least one neural processor (177a to 177c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (177a to 177c), and a neural interface (1118) for interfacing with at least one neural processor (177a to 177c).

[0517] Meanwhile, the neural system service (1110) may further execute or provide a model container (509) that performs interface of model parameters related to the operation of the neural processor (177) and version management of learning files.

[0518] Meanwhile, the second signal processing device (170b) may be equipped with a neural processor (177mb, 177mc) and a neural controller (1115mb, 1115mc) for controlling the neural processor (177mb, 177mc).

[0519] In the drawing, each neural controller (1115mb, 1115mc) is illustrated for two neural processors (177mb, 177mc), but alternatively, it is also possible to have one neural controller.

[0520] Meanwhile, for interfacing with two neural processors (177mb, 177mc) within the second signal processing device (170b), the neural interface (1118) may include a first neural interface (1118b) and a second neural interface (1118c).

[0521] Fig. 22 is a flowchart showing an example of an operation method of the signal processing device and the second signal processing device of Fig. 21.

[0522] Referring to the drawing, the second signal processing device (170b) can transmit unique ID information when connected to the signal processing device (170) by wire (S2005).

[0523] In response to this, the signal processing device (170) can receive unique ID information from the second signal processing device (170b) and detect a wired connection based on this (S2010).

[0524] Next, the signal processing device (170) can transmit a request for specification information for a newly connected or mounted second signal processing device (170b) to the server (900) (S2012).

[0525] In response to this, the server (900) can transmit specification information about the second signal processing device (170b) to the signal processing device (170) (S2014).

[0526] Additionally, the server (900) can transmit update information or an application for controlling the second signal processing device to the signal processing device (170) if there is an application for controlling the second signal processing device (170b) or update information for the second signal processing device (170) (S2016).

[0527] In response to this, the signal processing device (170) may receive update information for the second signal processing device (170b) or an application for controlling the second signal processing device, and install an update for the second signal processing device (170b) or an application for controlling the second signal processing device (S2018).

[0528] Next, the signal processing device (170) can perform abstraction on the internal neural processor (177) (S2020) and monitor the neural processor (177) (S2022).

[0529] Meanwhile, the signal processing device (170) determines whether it is in communication with the second signal processing device (170b) (s2024), and if so, checks the data communication speed with the second signal processing device (170b) and requests optimization of the command queue to the second signal processing device (170b) (S2026).

[0530] In response to this, the second signal processing device (170b) can perform optimization of the command queue for the second neural processor within the second signal processing device (S2030).

[0531] For example, the second signal processing device (170b) can perform parallel operation processing of a command queue slot for a second neural processor within the second signal processing device.

[0532] Next, the second signal processing device (170b) can transmit the result data for the operation processing and the delay time information to the signal processing device (170) (S2032).

[0533] In response to this, the signal processing device (170) can collate the computational processing result data of the internal neural processor (177) and the computational processing result data of the second signal processing device (170b) and transmit the collated result data to the application that requested the computation (S2034).

[0534] Accordingly, the neural processor can be operated efficiently using the second signal processing device (1770b), which is an external device.

[0535] FIG. 23a is a diagram illustrating various examples of queuing using a signal processing device and a second signal processing device.

[0536] Referring to the drawing, (a) of FIG. 23a is a drawing showing sequential queuing.

[0537] Referring to the drawing, the first virtual machine (520) can control a plurality of operations (TRa, TRb, TRb2, TRc) from a plurality of applications running on at least one of the plurality of virtual machines (520 to 540) to be sequentially arranged in the first neural processor (2110) within the signal processing device (170).

[0538] According to the sequential operation method according to this sequential arrangement, the first neural processor (2110) cannot be operated efficiently, and accordingly, there is a disadvantage in that it takes a considerable amount of time until the multiple operations (TRa, TRb, TRb2, TRc) are completed.

[0539] Referring to the drawings, (b) of FIG. 23a is a drawing showing an example of optimized queuing using a signal processing device (170) and a second signal processing device (170b).

[0540] Referring to the drawing, the first virtual machine (520) in the signal processing device (170) sequentially arranges a first operation (TRa) and a second operation (TRb) in the first neural processor (2110) in the signal processing device (170), and sequentially arranges a third operation (TRc) and a fourth operation (TRb2) in the second neural processor (2118) in the second signal processing device (170b), and can control the first operation (TRa) and the third operation (TRc) to be performed in parallel and simultaneously, and the second operation (TRb) and the fourth operation (TRb2) to be performed in parallel and simultaneously.

[0541] Accordingly, a free space, such as Art, can be secured in the first neural processor (2110) within the signal processing device (170), and additional operations can be performed through this free space (Arr). Accordingly, the neural processor can be operated efficiently. Furthermore, power consumption can be reduced.

[0542] Figure 23b is a diagram illustrating a communication delay between a signal processing device and a second signal processing device.

[0543] Referring to the drawing, the first virtual machine (520) within the signal processing device (170) can control the first neural processor (2110) within the signal processing device (170) to sequentially arrange and operate the first operation (TRa) and the second operation (TRb), and can control the second neural processor (2120) within the second signal processing device (170b) to sequentially arrange and operate the third operation (TRc) and the fourth operation (TRb2).

[0544] The second neural processor (2120) within the second signal processing device (170b) represents the sequence and operation of the ideal third operation (TRc) and fourth operation (TRb2).

[0545] At this time, a communication delay (Tdy) may occur between the signal processing device (170) and the second signal processing device (170b), and therefore, the second neural processor (2130) in the second signal processing device (170b) may perform the fourth operation (TRb3) with an actual Tdy delay.

[0546] Accordingly, as in step 2032 (S2032) of FIG. 22, it is preferable that the second signal processing device (170b) transmit the result data for the operation processing and the delay time information (Tdy) to the signal processing device (170).

[0547] Figure 24 is a diagram illustrating a wireless connection between a signal processing device and an external device.

[0548] Referring to the drawing, a signal processing device (170) according to another embodiment of the present disclosure can perform wireless pairing through a communication device (120) when an external device (800) is connected wirelessly, and after completion of wireless pairing, control to transmit data to a second neural processor within the external device.

[0549] In particular, the central processor (175) within the signal processing device (170) executes a server virtualization machine (530), and the server virtualization machine (520) can execute a neural system service (1110) for controlling at least one neural processor (177a to 177c).

[0550] Meanwhile, the neural system service (1110) may execute or be equipped with a neural manager (1113) for managing at least one neural processor (177a to 177c), a neural controller (1115) for determining or controlling an inference method of at least one neural processor (177a to 177c), and a neural interface (1118) for interfacing with at least one neural processor (177a to 177c).

[0551] Meanwhile, the neural system service (1110) may further execute or provide a model container (509) that performs interface of model parameters related to the operation of the neural processor (177) and version management of learning files.

[0552] Meanwhile, the external device (800) may be equipped with a neural processor (877m), a neural controller (815m) for controlling the neural processor (877m), and an interface (818m).

[0553] Meanwhile, the external device (800) may be a mobile terminal or a server.

[0554] Meanwhile, the first virtual machine (520) can transmit data to the second neural processor (877m) within the external device (800) when the external device is connected wirelessly. This enables efficient operation of the neural processor.

[0555] In particular, the central processor (175) within the signal processing device (170) can control, when an external device (800) is connected wirelessly, to perform wireless pairing via the communication device (120) and, after completion of wireless pairing, to transmit data to the second neural processor (877m) within the external device (800). Accordingly, the neural processor can be operated efficiently.

[0556] Meanwhile, if the external device (800) is a device equipped with a battery, it is preferable that the first virtual machine (520) in the signal processing device (170) receive battery information from the external device (800) for computational processing based on battery information.

[0557] Meanwhile, the first virtual machine (520) within the signal processing device (170) can control the backing up of the calculation result data to the shared memory (508) during the calculation processing of the external device (800) based on the battery information received from the external device (800).

[0558] Figure 25a is a diagram illustrating that intermediate backup is not performed during operation processing of an external device.

[0559] Referring to the drawing, the first virtual machine (520) within the signal processing device (170) sequentially arranges the first operation (VRa) and the second operation (VRb) on the first neural processor (2310) within the signal processing device (170), as shown in (a) of FIG. 25a, and sequentially arranges the third operation (VRc) and the fourth operation (VRb2) on the second neural processor (2320) within the external device (800), but can control the first operation (VRa) and the third operation (VRc) to be performed simultaneously and in parallel.

[0560] Meanwhile, an external device (800) that operates based on a battery may terminate its operation during the execution of the fourth operation (VRb2), as shown in (b) of FIG. 25a, when the battery level is below a predetermined level.

[0561] Meanwhile, since the fourth operation (VRb2) performs the operation based on the result of the second operation (VR2), if the operation is terminated during the execution of the fourth operation (VRb2), the fourth operation (VRb2) will not be performed smoothly again even if a new battery is installed or a new power supply is supplied.

[0562] Accordingly, in this disclosure, through a mid-angle backup, the result data of an operation already performed is controlled to be backed up to a shared memory (508), etc.

[0563] Figure 25b is a diagram illustrating that an intermediate backup is performed during operation processing of an external device.

[0564] Referring to the drawing, the first virtual machine (520) within the signal processing device (170) sequentially arranges the first operation (VRa) and the second operation (VRb) on the first neural processor (2310) within the signal processing device (170), as shown in (a) of FIG. 25a, and sequentially arranges the third operation (VRc) and the fourth operation (VRb2) on the second neural processor (2320) within the external device (800), but can control the first operation (VRa) and the third operation (VRc) to be performed simultaneously and in parallel.

[0565] Meanwhile, the first virtual machine (520) can control the result data of the first operation (VRa) and the second operation (VRb) to be stored in the shared memory (508) after the end point of the second operation (VRb) for intermediate backup.

[0566] Meanwhile, an external device (800) that operates based on a battery may terminate its operation during the execution of the fourth operation (VRb2), as shown in (b) of FIG. 25b, when the battery level is below a predetermined level.

[0567] Meanwhile, since the result data of the first operation (VRa) and the second operation (VRb) are stored in the shared memory (508), when a new battery is installed in the external device (800) or a new power supply is supplied, the fourth operation (VRb2) can be performed immediately thereafter. Accordingly, stable and rapid operation performance is enabled.

[0568] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. At least one neural processor; A central processor that runs a hypervisor; The central processor, Running multiple virtual machines on the hypervisor, Among the above multiple virtual machines, the first virtual machine is: A signal processing device that sequentially receives a request for a first operation, a request for a second operation, and a request for a third operation from a plurality of applications running on at least one of the plurality of virtual machines, and, if the first operation and the third operation are capable of parallel processing, controls a first neural processor to process the first operation and the third operation in parallel, and controls the second operation to be processed after the first operation and the third operation are completed.

2. In paragraph 1, The above first virtual machine, A signal processing device in which, when the third operation receives a request for the fourth operation after the request, and when sharing of the operation layers between the second operation and the fourth operation is possible, the first neural processor is controlled to sequentially process the second operation and the fourth operation after the completion of the first operation and the third operation.

3. In paragraph 1, The above first virtual machine, A signal processing device that controls the arrangement of data for the plurality of operations within the internal memory within the first neural processor to vary when a plurality of operations are requested from the plurality of applications.

4. In paragraph 1, The above first virtual machine, Execute a neural system service for controlling the above neural processor, The above neural system service, A signal processing device that controls the arrangement of data for the plurality of operations within the internal memory within the first neural processor to vary when a plurality of operations are requested from the plurality of applications.

5. In paragraph 1, The above first virtual machine, Controls the initialization of the learning model according to a learning model initialization request from the above application, Receiving a file related to a learning model stored in shared memory, and controlling the storage of the file related to the received learning model in an internal memory within the first neural processor, In response to an inference request from the application, control is given to parallel processing in the first neural processor the data corresponding to the first operation and the data corresponding to the third operation among the sensor data or camera data stored in the shared memory, A signal processing device that controls, after completion of the first operation and the third operation, to process data corresponding to the second operation among sensor data or camera data stored in the shared memory.

6. In paragraph 1, The above first virtual machine, A signal processing device that controls the processing result data of the first operation to the third operation to be stored in a shared memory within the hypervisor.

7. In paragraph 1, The above first virtual machine, A signal processing device that controls the discretization level of the second operation to be lowered from the first level to the second level when in energy saving mode or when an object is not detected for a predetermined period of time based on the result of the first operation.

8. In paragraph 1, The above first virtual machine, A signal processing device that controls the first neural processor to perform the first operation to the third operation when the central processor cannot perform the first operation to the third operation.

9. In paragraph 1, The above first virtual machine, A signal processing device capable of performing the first operation to the third operation in the central processor, and, if the first operation and the third operation can be processed in parallel, controlling the central processor to process the first operation and the third operation in parallel, and controlling the central processor to process the second operation after the first operation and the third operation are completed.

10. In paragraph 1, The above first neural processor, Internal memory for storing weight data or input data; A signal processing device comprising: an operator for performing an operation based on data from the internal memory.

11. In paragraph 1, The above first neural processor, According to the first quantization mode, at least one of the first operation to the third operation is performed based on input data of the first length, A signal processing device that performs at least one of the first operation to the third operation based on input data of a second length longer than the first length, according to a second quantization mode.

12. In paragraph 1, The above first neural processor, According to the first quantization mode, at least one of the first operation to the third operation is performed based on input data of the first length, A signal processing device that performs a connection operation based on input data of the first length and differential data corresponding to the input data of the first length according to a second quantization mode, generates input data of a second length longer than the first length according to the connection operation, and performs at least one of the first operation to the third operation based on the input data of the second length.

13. In paragraph 11, The above first neural processor, A signal processing device that performs at least one of the first operation to the third operation according to the second quantization mode, and then switches to the first quantization mode when the number of feature points for the detected object is less than or equal to a predetermined value according to result data of at least one of the first operation to the third operation.

14. In paragraph 11, The above first neural processor, When the vehicle is driven during the daytime, at least one of the first operation to the third operation is performed depending on the second quantization mode or the daytime mode. A signal processing device that performs at least one of the first operation to the third operation, depending on the first quantization mode or the night driving mode, when the vehicle is driven at night.

15. In paragraph 1, The above first virtual machine, When a request for a fourth operation is received from the above multiple applications, the second neural processor is controlled to process the fourth operation, Controlling the second neural processor to perform the fourth operation based on input data of the first length according to the first quantization mode, A signal processing device that controls the first neural processor to perform at least one of the first operation to the third operation based on input data of a second length longer than the first length, according to a second quantization mode.

16. Central processor running the hypervisor; Includes a neural processor; Exchange data with an external device equipped with a second neural processor, either wired or wirelessly; The central processor, Running multiple virtual machines on the hypervisor, Among the above multiple virtual machines, the first virtual machine is: A signal processing device that transmits data to the second neural processor within the external device when the external device is connected.

17. In paragraph 16, The central processor, A signal processing device that performs wireless pairing through a communication device when the external device is connected wirelessly, and controls data to be transmitted to the second neural processor within the external device after the wireless pairing is completed.

18. In paragraph 16, The above first virtual machine, A signal processing device that receives result data from the external device and controls the transmission of the result data to the neural processor.

19. In paragraph 16, The above first virtual machine, A signal processing device that receives result data and battery information from the external device and controls the result data to be stored in shared memory within the hypervisor.

20. A vehicle display device comprising a signal processing device according to any one of claims 1 to 19.

Images