Method for controlling a buffer memory and vehicle using the same

Abstract

This document discloses a method for controlling a buffer memory and a vehicle using the method. The method includes: sorting multiple sets of preliminary data based on the size of multiple sets of preliminary data generated by processing multiple sensor data for detecting the surrounding environment through a first processor; storing the multiple sets of preliminary data in the buffer memory in sorted order based on the capacity of the buffer memory; performing task processing on the preliminary data stored in the buffer memory by a second processor in sorted order to generate task data; storing the task data in system memory if the processing of all the multiple sets of preliminary data is not completed; and generating subsequent data by the first processor based on the task data if the processing of all the multiple sets of preliminary data is completed.

Description

[0001] Cross-references to related applications This application claims priority to Korean Patent Application No. 10-2025-0047939, filed on April 14, 2025, with the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to a method for controlling a buffer memory and a vehicle using the method, and more specifically, to a method and vehicle for managing a buffer memory, which improves resource efficiency and processing speed by using dynamic memory management of the system cache. Background Technology

[0003] The descriptions in this background section are for the purpose of enhancing the understanding of the background technology of this invention and should not be construed as an admission that they correspond to prior art known to those skilled in the art.

[0004] Autonomous driving functions can be configured to estimate paths and control the vehicle by recognizing the surrounding environment. To meet the high level of perception performance required for autonomous driving, the complexity of learning models is increasing.

[0005] To process learning models at high speed, a System-on-a-Chip (SoC) equipped with a neural network processor can be used. Neural network processors are designed for general-purpose computations primarily used in learning models, and therefore may experience bottlenecks when handling the specific computations included in a particular model. To compensate for this, software pipeline techniques can be applied using the various resources within the SoC. However, from a system perspective, software pipelines can lead to performance penalties due to data sharing among multiple resources. Summary of the Invention

[0006] Various examples of the present invention are intended to provide a method and vehicle for managing buffer memory, which improves resource efficiency and processing speed by using dynamic memory management of the system cache.

[0007] The various examples to be achieved in this invention are not limited to the technical problems described above, and those skilled in the art will clearly understand from the following description other technical problems not mentioned.

[0008] According to the present invention, a method performed by a vehicle may include: acquiring sensor data related to the vehicle's surrounding environment via the vehicle's sensors; storing the sensor data in the vehicle's system memory; generating preliminary data based on the sensor data; sorting the portions of the preliminary data based on their sizes; storing the portions of the preliminary data in the order of the sorted portions in a buffer memory, wherein the buffer memory has a higher access speed and a smaller storage capacity than the system memory; performing task processing on the portions stored in the buffer memory in the order of the sorted portions to generate task data; storing the task data in the system memory if processing of all portions is not completed; generating control data based on the task data if processing of all portions is completed; outputting control commands based on the control data; and controlling the vehicle's operation based on the control commands.

[0009] According to the present invention, a vehicle may include: a sensor, a main memory, a local memory, and a processor circuit, wherein the sensor is configured to acquire sensor data indicating the vehicle's surrounding environment; the main memory is configured to store the sensor data; the local memory is configured to provide a higher access speed and a smaller storage capacity than the main memory; and the processor circuit is configured to: determine a path for the vehicle based on the sensor data; select path-related sensor data from the sensor data; perform a first-stage processing on the selected sensor data to generate preliminary data and store the preliminary data in the main memory; sort the portions of the preliminary data based on the size of each portion; store a portion of the sorted portions in the local memory based on the capacity of the local memory; perform a second-stage processing on the portion stored in the local memory; during the second-stage processing, determine whether the remaining portions of the preliminary data remain in the main memory; based on the presence of the remaining portions in the main memory, load the remaining portions from the main memory into the local memory in the order of the sorted portions; perform a third-stage processing on the task data generated by the second-stage processing to generate control data; output signals related to the control data; and control the vehicle's autonomous driving based on the signals.

[0010] The vehicle may include: sorting the portions of the preceding data in descending order of their size. The vehicle may include: during the sorting of the portions of the preceding data, the processor circuitry assigning a first task identifier to a first portion of the preceding data portions, and after completing the second stage processing of the first portion, the processor circuitry deleting the first task identifier assigned to the first portion. The vehicle may include: the processor circuitry and local memory disposed within the on-chip system, and the main memory may include volatile memory.

[0011] According to the present invention, a vehicle may include: a plurality of sensors, a system memory, a buffer memory, and a processor; the plurality of sensors are configured to acquire sensor data related to the vehicle's surrounding environment; the system memory is configured to store the sensor data; the buffer memory is configured to have a higher access speed and a smaller storage capacity than the system memory; wherein the system memory stores at least one instruction, which, when executed by the processor, is configured to: generate preliminary data based on the sensor data; sort the portions of the preliminary data based on the size of each portion; store the portions of the preliminary data in the buffer memory in the order of the sorted portions, based on the capacity of the buffer memory; perform task processing on the portions stored in the buffer memory in the order of the sorted portions to generate task data; store the task data in the system memory if the processing of all portions is not completed; generate control data based on the task data if the processing of all portions is completed; output control instructions based on the control data; and control the vehicle's operation based on the control instructions.

[0012] The vehicle may include: the generation of preliminary data and the generation of control data performed by a neural processing device; and task processing performed by at least one of a central processing unit, a graphics processing unit, or a digital signal processor. The vehicle may include: generating portions of the preliminary data in system memory before sorting and storing them in a buffer. The vehicle may include: determining the order of the sorted portions in descending order of their size. The vehicle may include: during the storage of portions of the preliminary data in the buffer, sequentially storing at least a portion of the portions of the preliminary data in the buffer to match the capacity of the buffer; and during task processing, sequentially storing the remaining portions of the preliminary data in the buffer based on their presence.

[0013] The vehicle may include: assigning task identifiers to each portion of the preceding data during sorting of the portions of the preceding data. The vehicle may include: deleting the task identifiers assigned to a portion after completing task processing of that portion of the preceding data. The vehicle may include instructions, when executed by a processor, configured to: select multiple sensor data corresponding to the estimated path of the vehicle from multiple sensor data based on previous sensor data, and generate preceding data based on the selected multiple sensor data. The vehicle may include instructions, when executed by a processor, configured to: execute at least one layer of a neural network-based learning model based on sensor data to generate preceding data. The vehicle may include: a processor and buffer memory disposed within an on-chip system, and the system memory may include volatile memory.

[0014] The features briefly outlined above are merely examples of the detailed description of the invention below and are not intended to limit the scope of the invention. Attached Figure Description

[0015] The above and other objects, features, and advantages of the invention will become more apparent to those skilled in the art from one or more exemplary examples described in detail with reference to the accompanying drawings, wherein: Figure 1 An example is shown where a vehicle communicates with other devices to send and receive data.

[0016] Figure 2 An example of a module for configuring a vehicle is shown.

[0017] Figure 3 An example of a module for configuring the controller and system memory is shown.

[0018] Figure 4 An example of memory mapping of system memory is shown.

[0019] Figure 5 An example of a module for configuring a server is shown.

[0020] Figure 6A and Figure 6B This is an example flowchart of a method for controlling a buffer memory.

[0021] Figure 7 An example of sensor data obtained from multiple sensors is shown.

[0022] Figure 8 An example of the specification for task processing in a second processor is shown.

[0023] Figure 9 Examples of data processing based on a combination of buffer memory and system memory, as well as using system memory alone, are shown.

[0024] Figure 10 An example computing system is shown. Detailed Implementation

[0025] Hereinafter, examples of the invention will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement the invention. However, the invention can be implemented in various ways and is not limited to the examples described herein.

[0026] In describing examples of the invention, well-known functions or structures will not be described in detail because they may unnecessarily obscure the understanding of the invention. Identical constituent elements in the drawings are indicated by the same reference numerals, and repeated descriptions of identical elements will be omitted.

[0027] In this invention, when an element is simply referred to as "connected to," "joined to," or "attached to" another element, this can mean that the element is "directly connected to," "directly joined to," or "directly attached to" another element, or is connected to, joined to, or attached to another element with other elements interposed therebetween. Additionally, when an element "comprises" or "has" another element, this means that an element may further include another element without excluding another component, unless otherwise specifically stated.

[0028] In this invention, the terms first, second, etc., are used only to distinguish one element from another and do not limit the order or importance of the elements, unless specifically mentioned. Accordingly, without departing from the scope of the invention, a first element in one example may be referred to as a second element in another example, and similarly, a second element in one example may be referred to as a first element in another example.

[0029] In this invention, elements are distinguished from each other for the purpose of clearly describing each feature, but this does not necessarily mean that the elements are separate. That is, multiple elements may be integrated into a single hardware or software unit, or a single element may be distributed and formed in multiple hardware or software units. Therefore, such integrated or distributed examples are included within the scope of this invention, even if not otherwise mentioned.

[0030] In this invention, the elements described in the various examples are not necessarily essential elements, and some of the elements may be optional. Therefore, examples consisting of a subset of the elements described in the examples are also included within the scope of this invention. Additionally, examples including elements other than those described in the various examples are also included within the scope of this invention.

[0031] The advantages and features of the invention, as well as the ways in which they are implemented, will become apparent from the examples described in detail below with reference to the accompanying drawings. However, the examples may be implemented in many different forms and should not be construed as limiting to the exemplary examples set forth herein. Rather, these examples are provided so that the invention will be complete and will fully convey the scope of the invention to those skilled in the art.

[0032] For the purposes of this application and claims, the exemplary phrases “at least one of: A; B; or C” or “at least one of A, B, or C” are used, which means “at least one A, or at least one B, or at least one C, or at least one A, at least one B, and at least one C”. Furthermore, exemplary phrases such as “A, B, or C”, “at least one of A, B, and C”, “at least one of A, B, or C”, etc., as used herein, may refer to each of the listed items or all combinations of the listed items. For example, “at least one of A or B” may mean (1) at least one A; (2) at least one B; or (3) at least one A and at least one B.

[0033] As used in this specification, the terms "module" or "unit" refer to software and / or hardware components, and a "module" or "unit" performs certain operations / functions / roles. However, a "module" or "unit" is not to be construed as limited to software or hardware. A "module" or "unit" may be configured to reside in addressable storage media or to execute on one or more processors. Thus, by way of example, a "module" or "unit" may include at least one of the following components: software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The functionality provided in a component, "module," or "unit" may be combined into a smaller number of components, "modules," or "units," or further divided into additional components, "modules," or "units."

[0034] In this invention, a "module" or "unit" can be implemented as a processor and a memory. "Processor" should be broadly interpreted to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), microcontrollers, state machines, etc. In some contexts, "processor" can refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or field-programmable gate arrays (FPGAs). For example, "processor" can refer to a combination of processing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such combination. Furthermore, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" can refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage devices, and registers. The memory can be in a state of electronic communication with the processor when the processor can read information from the memory and / or record information in the memory. The memory integrated into the processor is in electronic communication with the processor.

[0035] One or more features described herein can be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium can continuously store a computer-executable program or temporarily store a program for execution or download. Furthermore, the medium can be a variety of recording or storage devices in the form of a single hardware device or multiple combined hardware devices, and is not limited to media directly connected to some computer system, but can also be distributed over a network. Examples of such media include magnetic media such as hard disks, floppy disks, or magnetic tapes; optical recording media such as CD-ROMs or DVDs; magneto-optical media such as floppy disks; and ROMs, RAMs, or flash memory configured to store program instructions. Further examples of such media include media or storage media managed by application stores that distribute applications or by various other sites or servers that provide or distribute software.

[0036] In a hardware implementation, the processing unit for performing the technology may be implemented within one or more ASICs, DSPs, digital signal processing devices, programmable logic devices, field-programmable gate arrays, processors, controllers, microcontrollers, microprocessors, electronic devices, or computers or combinations thereof, designed to perform the functions described in this invention.

[0037] In this invention, for ease of explanation, expressions of positional relationships such as "up," "down," "left," and "right" as used in this specification are employed, and the positional relationships described in this specification can be understood in reverse if the accompanying drawings shown in this specification are reversed.

[0038] According to the Society of Automotive Engineers (SAE), the automation levels of autonomous vehicles can be classified as follows: Level 0, corresponding to "No Automation," involves the autonomous driving system intervening only temporarily in emergency situations (e.g., automatic emergency braking) and / or providing only warnings (e.g., blind spot warning, lane departure warning, etc.), requiring driver intervention. Level 1, corresponding to "Driver Assistance," involves the system performing some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.), while the driver operates the vehicle in normal driving conditions, requiring the driver to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. Level 2, corresponding to "Partial Automation," involves the system performing steering, acceleration, and / or braking under driver supervision, requiring the driver to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard can correspond to "conditional automation," where the system drives the vehicle under limited conditions (e.g., performing driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver when the required conditions are not met. The driver is required to determine the system's operating status and / or timing, and to take over control in emergency situations, while not operating the vehicle (e.g., steering, acceleration, and / or braking) under other circumstances. At Level 4 of autonomous driving, the SAE classification standard can correspond to "high automation," where the system performs all driving functions, and the driver is required to take over control of the vehicle only in emergency situations. At Level 5 of autonomous driving, the SAE classification standard can correspond to "full automation," where the system performs full driving functions without any driver assistance (including in emergency situations), and the driver is not required to perform any driving functions other than determining the system's operating status. Although this invention applies the SAE classification standard to autonomous driving classification, other classification methods and / or algorithms can be used for one or more configurations described herein.

[0039] One or more features related to autonomous driving control can be activated based on configured autonomous driving settings (e.g., the selected level of autonomous driving or classification of driving modes). Based on the features of priority sensor data processing in the adaptive buffer assist described herein, the vehicle can more flexibly control various autonomous driving operations (e.g., updating lane keeping control when high-priority front camera data is processed first, adjusting braking control when larger radar feature blocks are processed first, or refining steering timing when additional sensor data is dynamically loaded into the buffer). Vehicle control can include vehicle-related operational controls (e.g., autonomous driving control, sensor control, braking timing control, acceleration adjustment, alarm timing control, or forward collision warning timing control).

[0040] It is also possible to control one or more auxiliary devices (e.g., engine brakes, exhaust brakes, hydraulic retarders, electromagnetic retarders, regenerative brakes, etc.) based on the characteristics of priority sensor data processing for adaptive buffer assistance. For example, when high-priority downhill sensor data is processed first, the vehicle can activate the engine brakes earlier; when radar-based distance characteristics are dynamically loaded into the buffer, the hydraulic retarder can be adjusted to maintain a safe distance; when the characteristics of the front-side camera indicate rapid deceleration ahead, regenerative braking can be increased to aid energy recovery.

[0041] It is also possible to control one or more communication devices (e.g., modems, network adapters, radio transceivers, or antennas supporting Ethernet, Wi-Fi, NFC, Bluetooth, LTE, 5G NR, or V2X) based on the characteristics of adaptive buffer-assisted priority sensor data processing. For example, a V2X transmitter can immediately broadcast a warning when high-priority hazard-related sensor data is processed first; a 5G modem can prioritize uplink telemetry when buffered multi-camera features indicate congestion; and a Wi-Fi or Bluetooth module can adjust the bandwidth allocation of the onboard unit when dynamically loaded radar features indicate a lane change event.

[0042] The Minimum Risk Management (MRM) strategy can also be controlled based on the characteristics of priority sensor data processing in adaptive buffer assistance. For example, when high-priority front-side camera or radar features indicating sudden obstacles are processed first, the vehicle can initiate immediate deceleration MRM; when dynamically loaded sensor data indicates loss of lane boundary information, the vehicle can steer towards a safe shoulder; when prioritized perception data indicates driver inaction, the processor can maintain controlled movement for a set period to bring the vehicle to a minimum risk state. During autonomous driving, when the driver is unable to respond to intervention requests, the minimum risk strategy can be activated, and one or more processors can control vehicle movement to achieve a reduced risk situation.

[0043] Bias driving can also be controlled based on the characteristics of prioritized sensor data processing in adaptive buffer assist. For example, when prioritizing high-priority side camera or radar features, the driving control unit can bias the vehicle toward the lane with fewer obstacles; when dynamically loaded sensor data indicates a large truck in an adjacent lane, the system can increase the lateral offset to gain additional clearance; when prioritized front camera data indicates an uneven road edge, the unit can adjust the target lateral distance of the bias to maintain stability. Bias driving control can intentionally maintain a lateral distance different from the lane center to improve safety in changing situations, such as lane changes, obstacle avoidance, or approaching other vehicles.

[0044] It is also possible to control one or more sensors (e.g., IMU sensors, cameras, LiDAR, radar, blind spot sensors, lane departure sensors, parking sensors, light or rain sensors, traction or stability sensors, tire pressure sensors, or airbag sensors) based on the characteristics of priority sensor data processing with adaptive buffer assistance. For example, when high-priority features from the front-side camera are processed first, the system can increase the frame rate of the front-side camera; when dynamically loaded radar features indicate a rapidly approaching object, the radar can switch to a higher resolution mode; when prioritized IMU data indicates instability, additional sensors such as traction or stability sensors can be activated to refine the perception. This sensor control can support autonomous driving operations, including acceleration, deceleration, steering, braking, lane keeping, collision avoidance, and other driving control functions.

[0045] The system can also control the level of autonomous driving and / or activate or deactivate it based on the characteristics of prioritized sensor data processing with adaptive buffering assistance. For example, when prioritized sensor features indicate complex road geometry or dense traffic, the system can automatically increase the level of autonomous driving to require increased driver attention; when dynamically loaded camera and radar features indicate driver inattention or an unsafe environment, autonomous driving can be temporarily deactivated; when high-priority sensor data confirms stable conditions, the system can maintain or resume a higher level of autonomous driving. Adjustments may include increasing the required frequency of hand-on steering wheel or forward-looking checks, or limiting the display of video content during higher levels of autonomy.

[0046] According to the present invention, vehicles can use a buffer memory system to more efficiently manage large amounts of sensor data, which prioritizes and organizes the data before processing. Modern vehicles equipped with autonomous driving capabilities receive continuous streams of image, radar signals, and other sensor inputs, which can exceed the processing speed of the vehicle's various internal processors. The present invention describes dynamically selecting only sensor data relevant to the vehicle's estimated path, prioritizing the selected data by size, and temporarily storing higher-priority data in a high-speed buffer cache. As each portion of data is processed, the buffer memory provides the next portion in an appropriate order, thereby reducing latency that occurs when the processor repeatedly accesses slower system memory. By controlling the data flow in this way, vehicles can reduce unnecessary processing, improve overall responsiveness, and support more reliable autonomous driving behavior.

[0047] In the following description, examples of the invention will be described with reference to the accompanying drawings.

[0048] In the following text, reference will be made to Figures 1 to 4 Such a vehicle is described, which dynamically manages a buffer memory installed in a controller that performs processes related to the control of the vehicle, and processes the data read from the buffer memory in a processor.

[0049] Figure 1 An example is shown where a vehicle communicates with other devices to send and receive data.

[0050] refer to Figure 1 Vehicle 100 can be powered by either electricity or fossil fuels. In the case of electricity, vehicle 100 can be a purely battery-based vehicle, for example, powered solely by a high-voltage battery, or it can utilize a gas-based fuel cell as its energy source. Furthermore, the fuel cell can utilize various forms of gas capable of generating electricity, and the gas can be, for example, added to vehicle 100 in a liquefied state. Here, the gas can be, for example, hydrogen (e.g., compressed hydrogen, liquid hydrogen, ammonia-based hydrogen carrier, or reformable hydrocarbons, etc.). However, various gases can be used, and this is not a limitation. In the case of fossil fuels, vehicle 100 can be powered by fuels such as gasoline, diesel, or liquefied gas, and can be equipped with an internal combustion engine that drives actuator 114 through fuel combustion. The power source 112 can include an engine to provide the driving rotational force for the wheels. As another example, vehicle 100 can selectively utilize energy from a fossil fuel-based internal combustion engine and a battery to drive actuator 114; vehicle 100 can be a hybrid type vehicle (e.g., HEV, PHEV, or mild hybrid configuration, etc.).

[0051] Vehicle 100 can refer to a mobile device. Vehicle 100 is a ground vehicle that travels on the ground and can be a conventional passenger vehicle, commercial vehicle, or purpose-built vehicle (PBV). Vehicle 100 can be a four-wheeled vehicle, such as a passenger car, SUV, or minivan, or a vehicle with more than four wheels, such as a bus, large truck, container truck, or heavy equipment vehicle (e.g., mining dump truck, airport shuttle, or warehouse logistics transport vehicle). Vehicle 100 can also be a robot in a broad sense, such as a mobile device, and the robot can move using wheels, tracks, or other mobile modules (e.g., articulated tracks, omnidirectional wheels, or robotic leg assemblies).

[0052] Vehicle 100 can be controlled and driven by the user through manual driving or autonomous driving. Autonomous driving can be implemented as semi-autonomous or fully autonomous driving. Fully autonomous driving can be configured such that the controller 118 of vehicle 100 can fully control the operation of vehicle 100 without user intervention, even when driving conditions are uncertain. Semi-autonomous driving can be configured such that driver intervention is required based on specific driving conditions (e.g., low visibility, complex intersections, or emergency avoidance strategies). Semi-autonomous driving can be implemented such that the controller 118 deactivates autonomous driving in such situations, transferring control to the user, thus allowing the user to perform manual driving. According to the SAE (Society of Automotive Engineers) definition of autonomous driving levels, semi-autonomous driving corresponds to levels 1 to 4, and fully autonomous driving corresponds to level 5.

[0053] On the other hand, vehicle 100 can communicate with other devices 200, 300, or other vehicles 400. These other devices may include, for example, a server 200 supporting various controls, status management, and driving functions of vehicle 100; an ITS device 300 receiving information from an intelligent transportation system (ITS); and various types of user devices (e.g., smartphones, tablets, or wearable devices). Server 200, for example, is an external device operated by the vehicle manufacturer or configured to provide driving support functions, and can receive networked data from vehicle 100 or send data required for manual and autonomous driving. In response to requests and data sent from vehicle 100 and user devices, server 200 can send various information and software modules to vehicle 100 for controlling vehicle 100, supporting driving and various services (e.g., map updates, over-the-air diagnostics, or software feature activation).

[0054] The ITS device 300, such as a roadside unit (RSU), can exchange vehicle identification data, driving control and status data, environmental data around the vehicle, map data, etc. (e.g., traffic signal status, road hazard warnings, or congestion data) with vehicle 100 via V2I, thereby assisting the user in driving the vehicle or supporting the autonomous driving of vehicle 100. Vehicle 100 can exchange the data listed above (e.g., cooperative lane merging data, blind spot warnings, or platooning information) with other vehicles 400 via V2V, thereby supporting manual or autonomous driving.

[0055] Vehicle 100 can communicate with other vehicles or other devices based on cellular communication, Wireless Access in Vehicular Environment (WAVE) communication, Dedicated Short Range Communication (DSRC) or near-field communication or other communication schemes (e.g., UWB, Bluetooth or infrared links).

[0056] For example, vehicle 100 can use a cellular communication network such as LTE or 5G, WiFi, or WAVE to communicate with server 200, ITS device 300, or other vehicles 400. As another example, DSRC or similar technologies used in vehicle 100 can be used for inter-vehicle communication (e.g., low-latency collision avoidance message passing, cooperative braking signals, or local hazard broadcasting). The communication methods between vehicle 100, server 200, ITS device 300, other vehicles 400, and user devices are not limited to the examples described above.

[0057] Figure 2 An example of a module for configuring a vehicle according to an example of the present invention is shown.

[0058] The vehicle 100 may include: a sensor unit 102, an operation unit 104, a display 106, a load device 108, and a communication unit 110.

[0059] The sensor unit 102 may include various types of sensors for sensing various states and conditions occurring in the external surrounding environment, internal systems, user operations, and passenger space of the vehicle 100 (e.g., changes in cabin temperature, steering input, or passenger presence).

[0060] Specifically, the sensor unit 102 may include an externally facing camera 102a, radar sensor 102b, etc., to identify dynamic and static objects present around the vehicle 100.

[0061] Camera 102a can identify external objects as images, generate image data, and send the image data to controller 118 during use of vehicle 100. At least one camera 102a can be installed at a specific location on vehicle 100, such as the front, rear, or side of vehicle 100 (e.g., mounted on the grille, roof, or rearview mirror). When multiple cameras 102a are installed at various locations, this can be referred to as multiple cameras in this invention. Image data obtained by multiple cameras 102a can be referred to as multi-view image data. Although not shown, sensor unit 102 may further include cameras pointing into the interior of vehicle 100 (e.g., driver monitoring camera, passenger compartment occupant camera, or gesture recognition camera, etc.).

[0062] To detect the presence of external objects (e.g., approaching vehicles, roadside obstacles, or pedestrians) and their relative distance, speed, and direction, radar sensor 102b can emit radio waves of a specific frequency around vehicle 100 to generate radar data from the radio waves reflected from the external objects. In this invention, radar sensor 102b identifies objects in a different manner than camera 102a, and therefore can be a heterogeneous sensor of a different type from camera 102a.

[0063] In this invention, an example without a lidar sensor (not shown) is shown, but in another example, the lidar sensor may be mounted on vehicle 100. The lidar sensor can generate point cloud data as identification data for external objects and send it to a processor to generate three-dimensional spatial information that at least identifies the shape of the external object (e.g., vehicle outline, roadside structure, or road surface irregularity). The lidar sensor may be a heterogeneous sensor of a different type than camera 102a.

[0064] Sensor unit 102 may include attitude sensor 102c. Attitude sensor 102c can detect, for example, the three-axis state of vehicle 100 (e.g., yaw, pitch, and roll) and output various attitude states of the vehicle based on these factors. The attitude sensor may be configured as, for example, an IMU sensor, a gyroscope sensor, etc. (e.g., a six-axis IMU, a MEMS-based gyroscope, or a combined accelerometer-gyroscope sensor, etc.). Controller 118 can generate attitude data based on the attitude states.

[0065] Additionally, sensor unit 102 may include a positioning sensor 102d for confirming the location of vehicle 100. Positioning sensor 102d is, for example, a GPS sensor or a GNSS sensor (e.g., a GLONASS, Galileo, or BeiDou receiver), but is not limited thereto.

[0066] In addition, although in Figure 1Not shown, but sensor unit 102 may include a brake sensor and wheel sensors. The brake sensor detects the degree of braking of the brake, and the wheel sensors may (e.g., via a Hall effect sensor, magnetoresistive encoder, or optical encoder, etc.) detect, for example, wheel speed, wheel rotation angle, and wheel rotation angular velocity. The present invention primarily describes the sensors of sensor unit 102 mentioned in the example description, but may further include sensors for sensing various conditions not listed herein (e.g., rain sensors, temperature sensors, or ultrasonic parking sensors, etc.).

[0067] The operating unit 104 can be configured as a module controlled by a user for driving. For example, the operating unit 104 can be a steering wheel for manual driving, an automatic or manual transmission actuator, an accelerator pedal, a brake pedal, a gearbox, etc. The operating unit 104 may further include interfaces for the user to request activation or deactivation of the automatic driving mode and to select specific functions, enabling the user to use the automatic driving function. In order to receive various requests related to automatic driving (e.g., lane change requests, parking assistance activation, or driver monitoring confirmation, etc.), the operating unit 104 can be configured with a hardware-type interface, for example, located at a predetermined location inside the vehicle 100, or a software-type interface that can be touched on the display 106.

[0068] Display 106 can be used as a user interface. Display 106 can be controlled by controller 118 to display information such as the operating status of vehicle 100, control status, route / traffic information, remaining energy information, and content requested by the driver (e.g., navigation map, multimedia playback, or charging station information). Additionally, display 106 can be configured with a touchscreen that can detect driver input to receive driver requests instructing controller 118.

[0069] The communication unit 110 can support communication with the server 200, the ITS device 300, surrounding vehicles 400, etc. The communication unit 110 may include modules for processing, for example, cellular communication, WAVE, DSRC communication, etc. In this invention, the transmitting / receiving unit can transmit data generated or stored during driving to the server 200 in signal form, and receive data and software modules sent from the server 200 as specific types of signals (e.g., OTA updates, firmware patches, or remote diagnostic messages, etc.).

[0070] Communication unit 110 can support the use of in Figure 2 Communication between components, involving the sending or receiving of signals related to data, control, instructions, and requests. Components can be... Figure 2The devices or modules shown are referred to as 102-118. Communication unit 110 can use, for example, Controller Area Network (CAN) communication to send signals (i.e., signals related to data, control, commands, and requests) transmitted between components to a communication bus and to specific components. Data, control, commands, and requests sent via CAN communication can be referred to as CAN data (e.g., sensor status frames, actuator control signals, or diagnostic requests, etc.).

[0071] The communication unit 110 can support communication with electronic devices carried by passengers within the vehicle 100 (e.g., smartphone pairing, media streaming, or wireless key access, etc.). In this invention, the vehicle 100 can use the communication unit 110 to send data utilized in the method according to the invention to the outside and receive data utilized in the method according to the invention from the outside.

[0072] Additionally, vehicle 100 may include power source unit 112 and actuator 114.

[0073] The power source unit 112 can generate and supply power and electricity for both the driving electrical system (e.g., actuator 114) and the non-driving electrical system. The non-driving electrical system may be, for example, a sensor unit 102, an operation unit 104, a display 106, a load device 108, a communication unit 110, etc., but is not limited thereto, and may include various components (e.g., an HVAC system, an infotainment unit, or an interior lighting controller, etc.) that implement sensing, interface, communication, and convenience functions, in addition to the components that directly participate in the driving operation.

[0074] When vehicle 100 is driven by electric power, power source unit 112 may be configured with, for example, a battery that is charged from an external source, or a combination of a battery and a fuel cell (e.g., a PEM fuel cell, SOFC, or metal hydride fuel system) for charging the battery. In the case of a battery and fuel cell combination, power source unit 112 may include a tank storing materials used by the fuel cell to generate electricity, such as liquefied hydrogen. When vehicle 100 is driven by fossil fuels, power source unit 112 may be configured as an internal combustion engine. Additionally, when vehicle 100 is a hybrid type (e.g., series hybrid, parallel hybrid, or power split hybrid configuration), power source unit 112 may be configured as a combination of an internal combustion engine and a battery.

[0075] Actuator 114 includes at least one module for implementing driving operations and can perform driving operations based on signals from a user request from operating unit 104 or a request from controller 118. The requested signals may be related to at least one of longitudinal control (e.g., acceleration or deceleration), lateral control (e.g., steering), or gear shifting (e.g., upshifting, downshifting, or mode selection). Here, gear shifting can be handled by the controller 118 in manual driving or by a user request using a gear shifter.

[0076] Actuator 114 may include a wheel drive unit, mechanical components for performing driving operations within the wheel drive unit, and electronic modules to execute driving operations via manual operation by a user or instructions from an autonomous driving controller 118. When vehicle 100 operates on electric power, it may include components for transmitting requested driving operations to the wheel drive unit (e.g., a motor inverter, reduction gears, or an electric drive axle assembly). When vehicle 100 operates on fossil fuel power, actuator 114 may include a transmission or gear module for transmitting power from an internal combustion engine.

[0077] The wheel drive unit may include: multiple wheels, a drive force generation module, a braking module, and a steering module. The drive force generation module generates drive force and applies or transmits it to the wheels; the braking module decelerates the wheels; and the steering module enables lateral control of the wheels (e.g., rack and pinion steering, steer-by-wire, or four-wheel steering). When the vehicle 100 is driven by electric power, the drive force generation module may be configured with a motor assembly that generates drive force based on electricity output from a battery. When the vehicle 100 is driven by fossil fuels, the drive force generation module may be an engine or transmission assembly. The braking module of the electric vehicle 100 may further have regenerative braking functionality (e.g., hybrid braking, coasting regeneration, or one-pedal driving).

[0078] Additionally, vehicle 100 may include memory 116 and controller 118.

[0079] The memory 116 can store applications and various data used to control the vehicle 100, and can load applications or read and write data upon request from the controller 118. The application can be executed by using the controller 118 to perform specific functions (e.g., perception processing, path planning, or system diagnostics).

[0080] In this invention, memory 116 may include volatile memory and non-volatile memory. The volatile memory may be system memory 160, including DRAM, such as... Figure 3 As shown. Figure 3An example of a module configuring the controller and system memory is shown. System memory 160 may include a storage device for temporary data storage and a memory controller for controlling / managing the storage device (e.g., read scheduling, refresh operations, or address mapping). Non-volatile memory may be, for example, ROM, flash memory, etc. System memory 160 may store data processed or completed by the processor 130 of controller 118. System memory 160 may also store application programs used by controller 118. System memory 160 may communicate data with controller 118 using system memory bus 170.

[0081] Non-volatile memory can store application programs (e.g., motion control logic, sensor fusion routines, or low-level hardware drivers) that perform processing of multiple sensor data detected by sensor unit 102, dynamic management of buffer memory 140 storing intermediate data generated during processing, and control of vehicle 100. System memory 160 loads and stores application programs via request messages from controller 118, which can then execute the application programs stored in system memory 160.

[0082] Specifically, the application can be software that dynamically stores intermediate data generated from multiple sensor data input to a specific processor of controller 118 and transmitted to other processors in buffer memory 140. Alternatively, the application can be software that uses the environment around vehicle 100 identified based on sensor data processed by the processor to establish a path (or estimate a path or driving path) for vehicle 100, and controls vehicle 100 based on that path (e.g., lane following, obstacle avoidance, or intersection handling).

[0083] Multiple sensor data may include, for example, multiple image data generated for each of multiple cameras. As another example, multiple sensor data may include image data from camera 102a and heterogeneous data from heterogeneous sensors (e.g., radar sensors, lidar sensors, or ultrasonic sensors). Multiple sensor data is not limited to the examples above and may include sensor data obtained from at least one of different homogeneous and heterogeneous sensors (e.g., multi-radar fusion, camera-IMU fusion, or radar-lidar, etc.).

[0084] The controller 118 can perform overall control of the vehicle 100. The controller 118 can be configured to execute applications and instructions stored in the memory 116.

[0085] Controller 118 can generate control-related signals for components of vehicle 100 in response to control requests in both manual and automated driving. In this invention, controller 118 is exemplified as an automated driving controller, which may be mounted on, for example, an electronic control unit (ECU). Although an automated driving controller is shown in this invention, vehicle 100 may include various controllers. For example, vehicle 100 may include a motor control unit (MCU), a transmission control unit (TCU), a battery management system (BMS), an automated driving controller, an air conditioning controller, an electric drive control unit (EDCU), or a vehicle communication network controller, etc. (e.g., a distributed domain controller, a zone controller, or a centralized onboard computer, etc.).

[0086] The controller 118 can be implemented as, for example, a system-on-a-chip (SoC). The controller 118 includes components (or modules) with various functions on a single chip and may include a processor 130, a buffer memory 140, and an on-chip memory bus 150 for communicating data between the processor 130 and the buffer memory 140, such as... Figure 3 As shown. In addition to the components described above, controller 118 also includes a timing generator, an external interface, converters, etc., which can be connected to the chip memory bus 150. The timing generator includes, for example, a timing generator phase-locked loop (PLL), and the external interface may include at least one of, for example, PCIe or Universal Serial Bus (USB). The converter may include, for example, an analog-to-digital converter or a digital-to-analog converter (e.g., a high-speed analog-to-digital converter, a ΣΔ digital-to-analog converter, or a mixed-signal converter, etc.).

[0087] The components described above are exemplary, and the controller 118 may further include other modules. Furthermore, the components of the controller 118 may be mounted on a single chip. In this invention, for ease of description, the processor 130, buffer memory 140, and chip memory bus 150 are described.

[0088] The controller 118 may include a first processor 132 and a second processor 134 to perform preprocessing, sensing processing and control processing, and to perform the above processing (e.g., sensor decoding, feature extraction or trajectory calculation, etc.).

[0089] The first processor 132 may include, for example, a neural processing device, and the neural processing device may be a neural processing unit (NPU). The first processor 132 may, for example, execute at least some portions of detailed modules of a learning model related to object recognition (e.g., bounding box detection, lane segmentation, or deep inference, etc.). The first processor 132 may include a first processing unit 132a and a first cache memory 132b. The first processing unit 132a may execute detailed modules stored in system memory 160. The first cache memory 132b may receive and temporarily store data from buffer memory 140 for use by the first processing unit 132a, or temporarily store processed data for transmission from the first processing unit 132a to system memory 160.

[0090] The learning model can be configured as a deep learning model that receives data from multiple sensors to infer perceptual data about an object. For example, the deep learning model may include a network model that performs object detection, semantic segmentation, depth estimation, etc. (e.g., a YOLO-type detector, an encoder-decoder architecture, or a transformer-based perceptual model). The learning model can be configured to have multiple detailed modules. Detailed modules may include, for example, a convolution generator for encoder or feature extraction, a matrix operator for generating vector graphics, and various generators and operators used in the decoder (e.g., upsampling blocks, attention blocks, or fusion blocks, etc.). In this invention, detailed modules may be referred to as detailed components or detailed modules.

[0091] The second processor 134 includes, for example, a central processing unit (CPU). The second processor 134 may include a second processing unit 134a and a second cache memory 134b. The second processing unit 134a may execute portions of detailed modules of the learning model stored in system memory 160, preprocessing related to the learning model, intermediate processing, or control processing (e.g., temporal filtering, object tracking, or cost graph generation). The second cache memory 134b may receive and temporarily store data from buffer memory 140 for use by the second processing unit 134a, or may temporarily store processed data for transmission from the second processing unit 134a to system memory 160.

[0092] Preprocessing may include, for example, noise removal, filtering, or clustering of sensor data (e.g., denoising radar points, normalizing image frames, or merging adjacent lidar points, etc.). The control process may estimate the path based on perception data inferred from the learning model and generate control commands based on the estimated path. The control commands include at least one of longitudinal control and lateral control and may be generated as control signals for actuator 114 (e.g., throttle command, braking torque, or steering angle request, etc.). For example, the control processing may be performed using the learning model used for path estimation and control command generation.

[0093] In addition to the central processing unit, the second processor may further include a graphics processing unit or a digital signal processing unit. In this invention, the graphics processing unit and the digital signal processing unit can be described as... Figure 3 The third processor 136 and the fourth processor 138 are used to distinguish them from the central processing unit.

[0094] The third processor 136, corresponding to the graphics processing device, may include a graphics processing unit (GPU) supporting the execution of the learning model. The third processor 136 may include a third processing unit 136a and a third cache memory 136b. The third processing unit 136a may execute a portion of the detailed modules of the learning model stored in system memory 160 (e.g., batch parallel computation, tensor operations, or large matrix multiplication). The third cache memory 136b may receive and temporarily store data from buffer memory 140 for use by the third processing unit 136a, or may temporarily store processed data for transmission from the third processing unit 136a to system memory 160.

[0095] The fourth processor 138 corresponding to the digital signal processing apparatus may include, for example, a digital signal processor (DSP) for post-processing the sensed data (e.g., smoothing, encoding, or formatting the sensed results). The fourth processor 138 may include a fourth processing unit 138a and a fourth cache memory 138b. The fourth processing unit 138a can perform post-processing. The fourth cache memory 138b may receive and temporarily store data from the buffer memory 140 for use by the fourth processing unit 138a, or may temporarily store processed data for transmission to the system memory 160 after processing in the fourth processing unit 138a. Post-processing may process the sensed data generated by the sensed processing into a format suitable for control processing, or process the sensed data into a format suitable for transmission for control processing. Post-processing may include, for example, encoding, compressing, or grouping the sensed data in a predetermined manner.

[0096] Buffer memory 140 can store multiple pieces of prior data processed by the first processor 132 and stored in system memory 160 according to predetermined criteria, and load the prior data into the second processor according to requests from the second processor (e.g., polling loading, priority-based loading, or size-based loading, etc.). Buffer memory 140 can also send prior data (i.e., task data) already processed by the second processor to system memory 160. The second processor is a central processing unit, a graphics processing unit, or a digital signal processing unit, and for ease of description in this invention, the second processor 134 is exemplified as a central processing unit.

[0097] Buffer memory 140 may include cache memory 142 and buffer controller 144. Cache memory 142 may receive multiple pieces of prior data from system memory 160 and temporarily store the prior data. Cache memory 142 (e.g., an SRAM-based cache, a multi-bank temporary storage memory, or a high-speed on-chip buffer) may have a higher bandwidth than system memory 160 configured with DRAM, but a smaller storage capacity. In other words, system memory 160 may have a lower access speed than cache memory 142, but a larger storage capacity.

[0098] like Figure 4 As shown, most of the processor's components, applications, cache memory 142, and related modules can be mapped into the address space of system memory 160. Figure 4 An example of memory mapping in system memory is shown. Software control can be manipulated by operating registers at mapped addresses or by performing read / write operations on data in the address space. For example... Figure 4 As shown, the cache memory 142 is mapped to the address space corresponding to the cache so that the software control logic can read or write relevant data (e.g., intermediate feature maps, object vectors, or task identifiers).

[0099] The buffer controller 144 can dynamically manage the storage of preceding data received from the system memory 160 and the transfer of task data to the system memory 160 (e.g., batch transfer, prioritizing larger data blocks, or reordering tasks based on load).

[0100] Figure 5 An example of a module for configuring a server according to the present invention is shown.

[0101] Server 200 can send response data to vehicle 100 according to the request of vehicle 100, and also send applications embedded in vehicle 100 and information for supporting vehicle driving (e.g., HD map updates, model updates, or driving assistance information). Server 200 may include communication unit 202, memory 204, and processor 206.

[0102] The communication unit 202 can send data to and receive data from external devices, supports mutual communication with the vehicle 100 in this invention, and exchanges data with the vehicle 100 (e.g., driving logs, control requests, or diagnostic information).

[0103] The memory 204 can store programs and various data used to run the server 200, and can load programs or read and write data according to the request of the processor 206. The memory 204 can save and manage programs used to process requests from the vehicle 100, applications embedded in the vehicle 100, and information for driving assistance (e.g., firmware, perception models, or service policies).

[0104] Processor 206 can perform overall control of server 200. Server 200 can be configured to execute programs and instructions stored in memory 204. Processor 206 can execute programs to process and respond to user requests sent from vehicle 100 (e.g., route planning, remote diagnostics, or remote control authorization).

[0105] In this invention, processor 206 is exemplified as a single processing module. In another example, processor 206 may be distributed across multiple processing modules, and the aforementioned processing may be performed by a distributed processing model (e.g., edge servers, cloud nodes, or load-balanced clusters).

[0106] In the following text, reference will be made to Figure 2 , Figure 3 , Figure 6A and Figure 6B A method for controlling a buffer memory in a controller according to another example of the present invention is described in detail. Figure 6A and Figure 6B This is an example flowchart of a method for controlling a buffer memory according to another example of the present invention. The second processor is a central processing unit, a graphics processing unit, or a digital signal processing unit, and for the convenience of the following description, the second processor 134 is illustrated as a central processing unit. Unless there is a technical contradiction, the following can be applied to examples of second processors configured with graphics processing units or digital signal processing units.

[0107] Furthermore, in this invention, the data processed by processor 130 is described as sensor data used to detect the surrounding environment. The data is not limited to sensor data, but can be various types of data processed by multiple processors of controller 118 (e.g., fused BEV features, compressed depth maps, or processed radar point sets, etc.). Unless technically contradictory, the following can be similarly applied to the processing of various types of data by processors and the dynamic management via buffer memory.

[0108] refer to Figure 6A In S105, the controller 118 of the vehicle 100 can receive multiple sensor data from the sensor unit 102 that detects the surrounding environment and estimate the path based on previous sensor data (e.g., previous frames, previous radar scans, or previously calculated trajectories).

[0109] Multiple sensor data can include, for example, multiple image data generated for each of multiple cameras. In another example, multiple sensor data can include image data from camera 102a and heterogeneous data from heterogeneous sensors. Heterogeneous sensors can be, for example, radar sensors or lidar sensors.

[0110] The preceding sensor data can be sensor data acquired by sensor unit 102 prior to the sensor data related to S105. An estimated path can be generated through a series of processes on the preceding sensor data, such as... Figure 3 As described. As another example, it can be based on the output to... Figure 3 The control signals from the previous control processing described herein, the steering request signal received from the operation unit 104, or the turn signal of the direction indicator (e.g., a right turn indicator, lane change request, or steering wheel rotation) received from the operation unit 104 are used to predict and estimate the path.

[0111] refer to Figure 6A In step S110, controller 118 can determine multiple selected sensor data from multiple sensor data based on the estimated path. The determination of the selected sensor data can be performed, for example, by a first processor 132. The multiple selected sensor data can be stored, for example, in system memory 160.

[0112] like Figure 7 As shown, when the estimated path is for a right turn into the right lane followed by a straight drive (see...) Figure 7 When cutting in from the right (as indicated by the arrow in the image), sensor data (or image data) associated with the front camera 182 and the right camera 184 from among the multiple cameras 182 to 188 can be identified as the selected sensor data (e.g., a front view image or a right lane image, etc.). Figure 7An example of sensor data obtained from multiple sensors is shown. In the case of turning right and then going straight, the environment captured by the front camera 182 and the right camera 184 may be more important than the environment captured by the left camera 186 and the rear camera 188.

[0113] refer to Figure 6A In S115, the first processor 132 of the controller 118 can perform preliminary processing on multiple selected sensor data to generate multiple preliminary data (e.g., feature tensors, intermediate representations, or encoded spatial graphs, etc.) and store the multiple preliminary data in the system memory 160.

[0114] The first processor 132 may include, for example, a neural processing device, and the neural processing device may be an NPU. The prior processing of the first processor 132 may include, for example, at least some of the detailed modules of a learning model related to object recognition. As described above, the learning model may be a deep learning model that infers perceptual data about an object. The detailed modules of the learning model may include, for example, a convolution generator for encoder or feature extraction, a matrix operator for generating vector graphics, various generators and operators used in the decoder, and related components.

[0115] The first processing unit 132a can load detailed modules stored in the system memory 160, receive multiple selected sensor data stored in the system memory 160, and output multiple preliminary data (e.g., intermediate data (e.g., feature maps or coded representations) generated by preliminary processing of the selected sensor data before the task processing stage).

[0116] refer to Figure 6A In S120, buffer controller 144 can sort multiple preceding data based on the size of multiple preceding data sent from system memory 160 and assign task identifiers to the preceding data (e.g., first sorting larger feature maps, grouping similar sizes, or prioritizing tasks based on computational cost, etc.).

[0117] The buffer controller 144 can, for example, sort the preceding data in descending order of size. Figure 8 As shown, the buffer controller 144 can assign a task identifier (task ID) to the sorted preceding data. Figure 8 An example of the specification for task processing in a second processor is shown. Figure 8 Instead of showing the selected sensor data in S110, preliminary data based on sensor data from the front camera 182, right camera 184, left camera 186, and rear camera 188 is shown. This is as a result of... Figure 8Another example of the example is that the ordering of the preceding data can be handled such that the larger task identifiers (task IDs) [1] and [4] are placed above the task IDs [2] and [3].

[0118] The buffer controller 144 can store information related to task data sorted by task identifier (e.g., size table, task metadata, or load order information) in the buffer cache memory 142.

[0119] refer to Figure 6B In S125, the buffer controller 144 can store multiple preceding data into the buffer cache memory 142 in sorted order based on the capacity of the buffer cache memory 142 (e.g., first store the largest feature map, fill the remaining capacity with smaller cells, or replace older entries, etc.).

[0120] Specifically, the buffer controller 144 can store at least a portion of multiple preceding data items into the buffer cache memory 142 in a sorted order to match the capacity of the buffer cache memory 142. The sorting order can be determined in descending order of the size of the preceding data. For example... Figure 8 and Figure 9 As shown in the figure above, when the cache memory 142 has a storage capacity of 4M and the sorting order is task identifiers [1], [4], [2], [3], the buffer controller 144 can store multiple preceding data corresponding to task IDs [1] and [4] into the cache memory 142 to match the storage capacity of the cache memory 142 (for example, if IDs [1] and [4] can be combined into 4M, then IDs [1] and [4] are combined, etc.). Figure 9 Examples of data processing based on a combination of buffer memory and system memory, as well as using system memory alone, are shown.

[0121] refer to Figure 6B In S130, the second processor 134 can perform task processing (e.g., compute feature refinement, run decoder blocks, or perform downstream control estimation, etc.) on the prior data stored in the buffer cache memory 142 in sorted order to generate task data.

[0122] The second processor 134 includes a central processing unit and can load some or related intermediate processing software from the detailed modules of the learning model stored in the system memory 160 to perform task processing on the prior data received from the buffer cache memory 142.

[0123] Additionally, the second processor 134 can check during task processing whether subsequent precedent data exists in the system memory 160. In response to (or based on) the presence of subsequent precedent data, the buffer controller 144 can store subsequent precedent data (e.g., the next largest unused item, a smaller block, or a low-priority entry, etc.) in sorted order into the buffer cache memory 142 based on the capacity of the buffer cache memory 142.

[0124] like Figure 9 As shown in the figure above, when the second processor 134 sequentially performs task processing on the prior data of task identifiers [1] and [4] received from the buffer cache memory 142, the buffer controller 144 can determine whether there is subsequent prior data in the system memory 160. In response to (or based on) the presence of prior data of task identifiers [2] and [3] corresponding to the subsequent prior data, the buffer controller 144 can store the prior data of task identifiers [2] and [3] in the buffer cache memory 142 in sorted order based on the capacity of the buffer cache memory 142. Specifically, when the second processor 134 loads and processes the prior data of the task identifier [4] stored in the buffer cache memory 142 in a later order, the prior data of task identifiers [2] and [3] can be stored in the buffer cache memory 142 in sorted order (e.g., filling free space once it becomes available, etc.).

[0125] refer to Figure 6B S135, the second processor 134 can determine whether the task processing of all preceding data has been completed.

[0126] like Figure 9 As shown in the figure above, after completing the task processing of the preceding data for task identifiers [1] and [4], the second processor 134 can determine whether the task processing of all preceding data has been completed by requesting the buffer controller 144 to confirm whether there is any remaining subsequent preceding data in the buffer cache memory 142. When there is subsequent preceding data, it can be determined that the task processing of all preceding data has not been completed. Alternatively, when there is no subsequent preceding data in the buffer cache memory 142, it can be determined that the task processing of all preceding data has been completed.

[0127] refer to Figure 6BIn S140, the buffer controller 144 may store the task data in system memory 160 and delete the task identifier associated with the task data if task processing of all preceding data is not completed. The second processor 134 may perform task processing on subsequent preceding data stored in buffer cache memory 142 in sorted order to generate task data, thereby performing substantially the same task processing as in S130 (e.g., applying the remaining decoder layer, calculating post-processing filters, or generating control-related tensors, etc.).

[0128] refer to Figure 6B In S145, the first processor 132 can perform subsequent processing on multiple task data stored in the system memory 160 based on the completion of all task processing on multiple preceding data, in order to generate multiple subsequent data (e.g., a fused BEV map, an object list, or a spatiotemporal scene representation).

[0129] The first processor 132 may receive a task processing completion message from the second processor 134 and perform subsequent processing. The subsequent processing may, for example, execute at least some of the detailed modules of the learning model related to object recognition, and specifically execute detailed modules different from the prior processing (e.g., decoder level, feature merging block, or time tracking unit, etc.).

[0130] refer to Figure 6B In S150, controller 118 (e.g., first processor 132) can generate an estimated path for vehicle 100 based on multiple subsequent data, and generate control commands (e.g., issuing steering requests, lane keeping control, or collision avoidance commands, etc.) for the operation (e.g., driving) of vehicle 100 based on the estimated path.

[0131] refer to Figure 8 and Figure 9 The processing time of the combination of the buffer memory 140 and the system memory 160 of the controller 118 according to the present invention was compared with the processing time of the system memory used alone. To calculate the total processing time, the size of the preceding data, the processing time of the second processor on the preceding data, and the read / write time of the system memory (e.g., a fixed feature map size, a predetermined CPU execution cycle, or an estimated DRAM latency value, etc.) can be referenced. Figure 8 Examples.

[0132] Figure 9 The figure above illustrates the data processing time of the combination of the buffer memory 140 and the system memory 160 of the controller 118 according to the present invention. Figure 9 The following diagram illustrates the data processing time based on a controller without buffer memory combined with system memory. Figure 9The Direct Memory Access (DMA) indicated in the above figure can mean accessing memory and performing read / write operations without occupying a secondary processor (CPU). In this invention, DMA can be an operation that transfers prior data from system memory 160 to buffer cache memory 142 (e.g., background transfer of large feature blocks, burst read cycles, or queued DMA copy instructions, etc.).

[0133] from Figure 9 It can be seen that, according to Figure 8 For example, compared to processing without using a buffer memory, the total data processing time according to the invention can be reduced by 8 ms. According to the invention, resource efficiency and processing speed can be improved by using dynamic memory management based on the system cache of the buffer memory (e.g., reducing CPU pauses, minimizing DRAM access bottlenecks, or increasing pipelined concurrency, etc.).

[0134] Figure 10 An example computing system (e.g., a computing device for a vehicle or any other device) is illustrated. One or more controllers, processors, etc., described herein (e.g., one or more components of vehicle 100, one or more components of server 200, one or more components of another vehicle 400, and any other components and devices disclosed herein) can be provided by, for example... Figure 10 The computing system shown is implemented or, as in Figure 10 This is implemented in the computing system shown.

[0135] The computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 interconnected via a bus 1200. The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. Each of the memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include read-only memory (ROM) and random access memory (RAM).

[0136] A communication interface, such as network interface 1700 (also referred to as a communication device, communicator, communication module, communication unit, etc.), enables the transmission of software and / or data between the device and one or more external devices, and / or between one or more components of the device. The communication interface may include a receiver, transmitter, transceiver, modem, network interface and / or adapter (e.g., Ethernet adapter), radio transceiver, antenna, communication port, PCMCIA slot and card, etc. Software and data transmitted via the communication interface may be in the form of signals, which may be electronic, electromagnetic, optical, infrared, or other signals that can be received by the communication interface. These signals may be provided to the communication interface via a communication path of the device, which may be implemented using, for example, wires or cables, optical fibers, cellular links, radio frequency (RF) links, and / or other communication channels. The communication interface can communicate using one or more communication protocols, such as Ethernet, Wi-Fi, Near Field Communication (NFC), Infrared Data Association (IrDA), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle to Everything (V2X), Controller Area Network (CAN), or Local Interconnect Network (LIN).

[0137] Therefore, the operation of the methods or algorithms described in conjunction with the exemplary examples disclosed in the specification can be directly implemented using hardware modules, software modules, or a combination of hardware modules and software modules executed by processor 1100. Software modules can reside on storage media (e.g., memory 1300 and / or storage device 1600), such as RAM, flash memory, ROM, erasable and programmable ROM (EPROM), electrically EPROM (EEPROM), registers, hard disk drives, removable disks, or optical disc-ROMs (CD-ROMs).

[0138] The storage medium can be connected to the processor 1100. The processor 1100 can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor 1100. The processor and storage medium can be implemented using an application-specific integrated circuit (ASIC). The ASIC can be located in the user terminal. Alternatively, the processor and storage medium can be implemented as separate components in the user terminal.

[0139] According to the present invention, a method for controlling a buffer memory using a controller (which includes a first processor, a second processor, and a buffer memory) and a system memory is provided. The method for controlling the buffer memory includes: sorting multiple sets of preliminary data based on the size of multiple sets of preliminary data generated by the first processor through processing multiple sensor data of detecting the surrounding environment; storing the multiple sets of preliminary data in the buffer memory in sorted order based on the capacity of the buffer memory; performing task processing on the preliminary data stored in the buffer memory by the second processor in sorted order to generate task data; storing the task data in the system memory if the processing of all the multiple sets of preliminary data is not completed; generating subsequent data by the first processor based on the task data based on the completion of the processing of all the multiple sets of preliminary data; and generating control instructions for controlling the device by the controller based on the subsequent data.

[0140] In some implementations, the first processor may be a neural processing device, and the second processor may be a central processing unit, a graphics processing unit, or a digital signal processor.

[0141] In some implementations, multiple sets of preliminary data may be stored in system memory, wherein sorting the preliminary data includes sorting the multiple sets of preliminary data sent from system memory.

[0142] In some implementations, the sorting order can be determined by descending order of the size of the preceding data.

[0143] In some implementations, storing multiple preceding data in a buffer may include storing at least a portion of the preceding data in the buffer in sorted order to match the buffer's capacity. Furthermore, generating task data may further include storing subsequent preceding data in the buffer in sorted order based on the buffer's capacity, in response to the presence of subsequent preceding data.

[0144] In some implementations, sorting multiple preceding data may further include assigning task identifiers to the preceding data.

[0145] In some implementations, the method may further include: deleting the task identifier associated with the task data based on the failure to complete all processing of multiple preceding data.

[0146] In some implementations, the method may further include determining a plurality of selected sensor data from the plurality of sensor data by considering an estimated path of the device based on previous sensor data of detecting the surrounding environment, prior to sorting the plurality of preceding data. Furthermore, the plurality of sensor data may include the plurality of selected sensor data.

[0147] In some implementations, the processing of the first processor generating preliminary data may include: executing some of the detailed modules belonging to the learning model in the first processor to process the sensor data.

[0148] In some implementations, the controller can be implemented as a system-on-a-chip (SoC) with a first processor, a second processor, and a buffer memory, and the system memory can be configured as volatile memory.

[0149] According to another example of the present invention, a vehicle is provided, comprising: a plurality of sensors, a memory, and a controller, wherein the plurality of sensors detect the surrounding environment and acquire a plurality of sensor data; the memory stores at least one instruction and includes a system memory; the controller executes at least one instruction stored in the memory and includes a first processor performing processing related to vehicle control, a second processor performing the processing, and a buffer memory managed by a buffer controller. The controller is configured to: sort the plurality of preliminary data based on the size of a plurality of preliminary data generated by the processing of the plurality of sensor data by the first processor; store the plurality of preliminary data in the buffer memory in sorted order based on the capacity of the buffer memory; perform task processing on the preliminary data stored in the buffer memory by the second processor in sorted order to generate task data; store the task data in the system memory if the processing of all the plurality of preliminary data is not completed; generate subsequent data by the first processor based on the task data based on the completion of the processing of all the plurality of preliminary data; and generate control instructions for controlling the device based on the subsequent data.

[0150] Furthermore, in autonomous driving systems, resource efficiency can be further improved by not processing unnecessary sensor data based on the estimated path.

[0151] According to the present invention, a method and vehicle for managing buffer memory can be provided, which improves resource efficiency and processing speed by using dynamic memory management of system cache.

[0152] The effects that can be obtained in this invention are not limited to those described above, and those skilled in the art will clearly understand from the following description other effects not mentioned.

[0153] Although the exemplary method of the present invention described above is represented as a series of operations for clarity of description, this is not intended to limit the order of the steps, and the steps may be performed simultaneously or in different orders as needed. To implement the method according to the invention, the described steps may further include other steps, including steps in addition to some steps, or may include additional steps in addition to some steps.

[0154] The various examples of the invention are not a list of all possible combinations, but are intended to describe representative examples of the invention, and the contents described in the various examples may be applied independently or in combination of two or more.

[0155] Furthermore, various embodiments of the present invention can be implemented in hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the present invention can be implemented using application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, etc.

[0156] The scope of the invention includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operation of methods according to various examples to be performed on a device or computer, and non-transitory computer-readable media on which such software or commands are stored and executable on a device or computer.

Claims

1. A method performed by a vehicle, the method comprising: The vehicle obtains sensor data related to its surrounding environment through its sensors. Sensor data is stored in the vehicle's system memory; Generate preliminary data based on sensor data; Sort the components of the preceding data based on the size of each component; Based on the capacity of the buffer memory, a portion of the preceding data is stored in the buffer memory in the order of the sorted portions, wherein the buffer memory has a higher access speed and a smaller storage capacity than the system memory. The tasks are processed in the order of the sorted portions stored in the buffer to generate task data; Since not all parts of the processing were completed, the task data was stored in the system memory. Based on the completion of all parts of the processing, control data is generated based on the task data; Control commands are output based on control data; The vehicle's control operations are based on control commands.

2. The method according to claim 1, wherein, The generation of preliminary data and control data is performed by a neural processing device, and task processing is performed by at least one of a central processing unit, a graphics processing unit, or a digital signal processor.

3. The method according to claim 1, wherein, The preceding data is generated in system memory before being sorted and stored in the buffer.

4. The method according to claim 1, wherein, The order of the sorted parts is determined by descending order of their size.

5. The method according to claim 1, wherein, Storing a portion of the preceding data into a buffer memory includes: storing at least a portion of the preceding data into the buffer memory sequentially to match the capacity of the buffer memory; The task data generation process includes: storing the remaining preceding data sequentially into the buffer memory based on the existence of the preceding data and the capacity of the buffer memory.

6. The method according to claim 1, wherein, Sorting the portions of the preceding data involves assigning task identifiers to each portion of the preceding data.

7. A vehicle comprising: Sensors configured to acquire sensor data indicating the vehicle’s surrounding environment; Main memory, configured to store sensor data; Local memory, configured to provide higher access speeds and smaller storage capacity than the main memory; as well as The processor circuitry is configured to enable the vehicle to: Determine the vehicle's path based on sensor data; Select sensor data related to the path from the sensor data; The selected sensor data is processed in the first stage to generate preliminary data, and the preliminary data is stored in the main memory; Sort the components of the preceding data based on the size of each component; Based on the capacity of the local memory, a portion of the sorted part is stored in the local memory; Perform second-stage processing on the portion stored in local memory; During the second stage of processing, it is determined whether the remainder of the preceding data exists in the main memory; Based on the existence of the remaining parts in main memory, the remaining parts are loaded from main memory into local memory in the order of sorting. The task data generated through the second stage of processing is processed in the third stage to generate control data; Output signals related to control data; Automated driving of vehicles based on signal control.

8. The vehicle according to claim 7, wherein, Sort the portions of the preceding data in descending order of their size.

9. The vehicle according to claim 7, wherein, During the sorting of portions of the preceding data, the processor circuit assigns a first task identifier to a first portion of the preceding data. After completing the second stage of processing of the first part, the processor circuit deletes the first task identifier assigned to the first part.

10. The vehicle according to claim 7, wherein, The processor circuitry and the local memory are located within the system-on-a-chip, and the main memory includes volatile memory.

11. A vehicle comprising: Multiple sensors, the multiple sensors being configured to acquire sensor data relating to the vehicle’s surrounding environment; System memory, configured to store sensor data; Buffer memory, configured to have higher access speed and smaller storage capacity than system memory; as well as processor; The system memory stores at least one instruction, which, when executed by the processor, is configured to cause the vehicle to: Generate preliminary data based on sensor data; Sort the components of the preceding data based on the size of each component; Based on the capacity of the buffer memory, the preceding data is stored in the buffer memory in the order of the sorted parts; The tasks are processed in the order of the sorted portions stored in the buffer to generate task data; Since not all parts of the processing were completed, the task data was stored in the system memory. Based on the completion of all parts of the processing, control data is generated based on the task data; Control commands are output based on control data; The vehicle's control operations are based on control commands.

12. The vehicle according to claim 11, wherein, The generation of preliminary data and control data is performed by a neural processing device, and task processing is performed by at least one of a central processing unit, a graphics processing unit, or a digital signal processor.

13. The vehicle according to claim 11, wherein, The preceding data is generated in system memory before being sorted and stored in the buffer.

14. The vehicle according to claim 11, wherein, The order of the sorted parts is determined by descending order of their size.

15. The vehicle according to claim 11, wherein, During the storage of a portion of the preceding data into the buffer memory, at least a portion of the preceding data is stored sequentially into the buffer memory to match the capacity of the buffer memory. During task processing, based on the presence of the remaining portion of the preceding data, the remaining portion of the preceding data is stored sequentially into the buffer memory.

16. The vehicle according to claim 11, wherein, During the sorting of the preceding data, a task identifier is assigned to each part of the preceding data.

17. The vehicle according to claim 16, wherein, After completing the task processing of a portion of the preceding data, delete the task identifier assigned to that portion.

18. The vehicle according to claim 11, wherein, When executed by the processor, the at least one instruction is configured to cause the vehicle to: Select multiple sensor data points corresponding to the estimated path of the vehicle from multiple sensor data points based on previous sensor data; Preliminary data is generated based on selected data from multiple sensors.

19. The vehicle according to claim 11, wherein, When executed by the processor, the at least one instruction is configured to cause the vehicle to: execute at least one layer of a neural network-based learning model based on sensor data to generate advance data.

20. The vehicle according to claim 11, wherein, The processor and the buffer memory are located within the system-on-a-chip, and the system memory includes volatile memory.

Images