Processor scheduling method and device, electronic equipment and computer program product

Abstract

The invention provides a processor scheduling method and device, electronic equipment and a computer program product. The method comprises the steps that a target task is determined according to the running state of a vehicle; the target task is dispatched to the multiple neural network processors, at least one of the multiple neural network processors processes the target task data corresponding to the target task, the target task is flexibly adjusted according to the actual running state of the vehicle, and the requirements of complex and changeable real scenes are met; the method is more suitable for complex road scene perception tasks in the ADAS, and the scheduling efficiency of the processor is improved.

Description

Technical Field

[0001] The present invention relates to the field of processor technology, and in particular to a processor scheduling method, device, electronic equipment and computer program product. Background Art

[0002] In the related art, the priority of each task is determined based on the running time and execution frequency of multiple tasks, and multiple tasks are classified into multiple priority queues from low to high priority, and multiple neural network processors (Neural-network Processing Unit, NPU) execute the tasks in the task queues in a fixed priority order.

[0003] However, as the real-time requirements for vehicle driving become increasingly higher, the static NPU scheduling method described above cannot adapt to the complex road scene perception tasks in the Advanced Driving Assistance System (ADAS) when faced with dynamic changes or environments that require flexible responses. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] To this end, an object of the present invention is to propose a processor scheduling method, which flexibly adjusts the target task according to the actual operating status of the vehicle, meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0006] Therefore, a second object of the present invention is to provide a processor scheduling device.

[0007] To this end, a third object of the present invention is to provide an electronic device.

[0008] To this end, a fourth object of the present invention is to provide a vehicle.

[0009] To this end, a fifth object of the present invention is to provide a computer-readable storage medium.

[0010] To this end, a sixth object of the present invention is to provide a computer program product.

[0011] In order to achieve the above-mentioned purpose, an embodiment of the first aspect of the present invention proposes a processor scheduling method, which includes: determining a target task according to the operating status of a vehicle; scheduling the target task to multiple neural network processors, at least one of the multiple neural network processors processing target task data corresponding to the target task.

[0012] According to the processor scheduling method of an embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0013] In some embodiments, determining a target task according to a running state of a vehicle includes: obtaining a task queue of a scene in which the vehicle is currently located; and determining a task with the highest priority in the task queue as a target task according to the running state.

[0014] In some embodiments, obtaining a task queue for a scene in which the vehicle is currently located includes: receiving a data frame of a scene in which the vehicle is currently located in response to a neural network processing control instruction; determining a forward-looking task, a surround-looking task, and a rearward-looking task according to the type of the data frame; and determining the forward-looking task, the surround-looking task, and the rearward-looking task as the task queue.

[0015] In some embodiments, the target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task, including: when the vehicle is in a forward state, determining that the target task is the forward-looking task in the task queue; scheduling the forward-looking task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward-looking task data corresponding to the forward-looking task.

[0016] In some embodiments, the target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes target task data corresponding to the target task, including: when the vehicle is in a turning state, determining that the target task is the forward view task and the surround view task in the task queue; scheduling the forward view task and the surround view task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward view task data corresponding to the forward view task and the surround view task data corresponding to the surround view task.

[0017] In some embodiments, the target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task, including: when the vehicle is in a reversing state, determining that the target task is the rear view task and the surround view task in the task queue; scheduling the rear view task and the surround view task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the rear view task data corresponding to the rear view task and the surround view task data corresponding to the surround view task.

[0018] In some embodiments, when at least one of the multiple neural network processors processes the target task data corresponding to the target task, it also includes: determining the overload number of the overloaded neural network processors among the multiple neural network processors, and when the overload number is a first preset number, scheduling the target task executed by the overloaded neural network processor to a non-overloaded neural network processor; when the overload number is a second preset number, the overloaded neural network processor suspends processing non-target task data, wherein the first preset number is less than the second preset number.

[0019] In some embodiments, the target task is scheduled to multiple neural network processors, at least one of the multiple neural network processors processes the target task data corresponding to the target task, and also includes: at least one of the multiple neural network processors executes a calling function to complete the current task and suspends the current task.

[0020] In some embodiments, the target task is scheduled to multiple neural network processors, at least one of the multiple neural network processors processes the target task data corresponding to the target task, and also includes: at least one of the multiple neural network processors resumes processing current task data.

[0021] In some embodiments, determining the target task according to the operating state of the vehicle also includes: scheduling the forward-looking task in the task queue to the first neural network processor, and scheduling the surround-looking task and the rear-looking task in the task queue to the second neural network processor.

[0022] In order to achieve the above-mentioned purpose, an embodiment of the second aspect of the present invention proposes a processor scheduling device, which includes: a determination module, used to determine a target task according to the operating status of a vehicle; a scheduling module, used to schedule the target task to multiple neural network processors, at least one of the multiple neural network processors processes target task data corresponding to the target task.

[0023] According to the processor scheduling device of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0024] In order to achieve the above-mentioned purpose, an embodiment of the third aspect of the present invention proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a processor scheduling program that can be executed by the at least one processor, and when the processor scheduling program is executed by the at least one processor, the at least one processor executes the processor scheduling method described in the above-mentioned embodiment.

[0025] According to the electronic device of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0026] In order to achieve the above-mentioned purpose, an embodiment of a fourth aspect of the present invention proposes a vehicle, wherein the vehicle includes the processor scheduling device described in the above-mentioned embodiment, or includes the electronic device described in the above-mentioned embodiment.

[0027] According to the vehicle of the embodiment of the present invention, under different vehicle operating conditions, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0028] In order to achieve the above-mentioned purpose, an embodiment of the fifth aspect of the present invention proposes a computer-readable storage medium, on which a processor scheduling program is stored. When the processor scheduling program is executed by the processor, the processor scheduling method described in the above-mentioned embodiment is implemented.

[0029] In order to achieve the above-mentioned purpose, an embodiment of the sixth aspect of the present invention proposes a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the processor scheduling method described in the above-mentioned embodiment is implemented.

[0030] According to the computer program product of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0033] Figure 1 is a structural block diagram of a processor scheduling system according to an embodiment of the present invention;

[0034] Figure 2 is a flowchart of a processor scheduling method according to an embodiment of the present invention;

[0035] Figure 3 is a flowchart of a processor scheduling method according to a specific embodiment of the present invention;

[0036] Figure 4 is a structural block diagram of a processor scheduling device according to an embodiment of the present invention;

[0037] Figure 5 is a structural block diagram of a vehicle according to an embodiment of the present invention.

[0038] Reference numerals: processor 1; connection circuit 2; first interrupt processing circuit 3; second interrupt processing circuit 4;

[0039] Processor scheduling device 5;

[0040] Determination module 11; Scheduling module 12;

[0041] Vehicle 10. DETAILED DESCRIPTION

[0042] Embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. Embodiments of the present invention are described in detail below.

[0043] With the widespread application of technologies such as autonomous driving, driver behavior recognition, pedestrian detection, and road condition analysis in automobiles, the demand for real-time driving in vehicles is getting higher and higher. These technologies are inseparable from the support of deep learning. Deep learning models require a lot of calculations, and running on traditional central processing units (CPUs) or graphics processing units (GPUs) is often inefficient and consumes a lot of power. The role of neural network processors in automotive chips is mainly to accelerate the operation of deep learning algorithms, provide real-time data processing capabilities, and help achieve more intelligent and safe driving. As hardware specially designed for neural networks, neural network processors can efficiently perform convolution, matrix multiplication, and other operations required by deep learning models, thereby greatly improving processing efficiency and reducing power consumption. In order to meet the needs of automotive chips to process big data scenarios, multiple neural network processors are usually configured in the automotive system-on-chip (System On Chip, SOC). A reasonable multi-core scheduling algorithm can greatly improve the data processing capabilities of the automotive system-on-chip.

[0044] However, in many on-board chip systems, neural network processor scheduling is performed only based on task priority. When multiple tasks arrive, the static NPU scheduling method first determines the priority of each task based on the running time and execution frequency of multiple tasks; multiple tasks are classified into multiple priority classes according to priority from low to high. When the neural network processor is running, it cyclically performs the following steps: execute all tasks to be executed in the second priority class in sequence; execute a task to be executed in the first priority class, wherein all tasks to be executed in the later priority class are executed in sequence at each preemption point of the task. The above method performs better when the predefined priority is clear and does not change with changes in the environment or state, but it may cause starvation. If high-priority tasks continue to enter the system, low-priority tasks will always be placed in the to-do queue and cannot be processed, and a priority inversion problem will occur, that is, high-priority tasks need to wait for low-priority tasks to release resources, which can easily lead to delays in high-priority tasks.

[0045] In addition, if high-priority tasks do not need to use all the neural network processor resources, and low-priority tasks cannot use the remaining resources, it will lead to a waste of resources. Finally, in some specific scenarios, such as multi-task learning or model parallel processing, simple priority scheduling may not be able to meet the dependencies between tasks, reducing processing efficiency.

[0046] Therefore, although pure priority scheduling is based on simple principles, it cannot fully meet the diverse needs in actual applications, and the scheduling strategy needs to be combined with other factors, such as the computing requirements of the task, input / output (I / O) requirements, and dependencies between tasks, in order to achieve better system performance. Tasks may be unevenly distributed on the neural network processors, which will cause some neural network processors to be overloaded and other neural network processor resources to be wasted.

[0047] Let's combine Figure 1 The processor scheduling system of an embodiment of the present invention is described. The processor scheduling system is used for accelerated processing of load neural networks and is applicable to the current mainstream advanced driving assistance system (ADAS) vehicle-mounted chips, which can be used in assisted driving systems as well as in intelligent driving systems.

[0048] The processor scheduling system includes a processor 1, a connection circuit 2, a first interrupt processing circuit 3, a second interrupt processing circuit 4, a first neural network processor NPU0 and a second neural network processor NPU1, wherein the connection circuit 2 is used to connect the first neural network processor NPU0 and the second neural network processor NPU1, the first interrupt processing circuit 3 is used to process the communication between the first neural network processor NPU0 and the processor 1, and the second interrupt processing circuit 4 is used to process the communication between the second neural network processor NPU1 and the processor 1; the first neural network processor NPU0 and the second neural network processor NPU1 can process multiple tasks in parallel, and these cores may run at the same or different frequencies to meet different needs, thereby processing simple or complex road scene perception tasks.

[0049] The connection circuit 2 connects the first neural network processor NPU0 and the second neural network processor NPU1 together, and combined with the first interrupt processing circuit 3 and the second interrupt processing circuit 4, the first neural network processor NPU0 and the second neural network processor NPU1 can collaborate to process a large neural network model. Processor 1 includes a scheduler and a task queue, which is one of the core parts of the system. It is responsible for deciding when, where and how to run tasks in the queue. When it has one or more task queues, it is used to store tasks to be processed. These queues may be distinguished according to the priority, type or other attributes of the task.

[0050] It can be seen from the above scheme that multiple neural network processors can process tasks in parallel, significantly improving the processing power and efficiency of the system, and can be dynamically adjusted according to changes in workload. The two neural network processors can be powered on and off independently, and only one core can be used to process tasks and idle cores can be shut down to save energy, or more cores can be started to handle high loads. Even if some cores have problems, the other cores can still work, ensuring the availability of the system.

[0051] Through reasonable processor scheduling strategies, neural network processors can be used more effectively, avoiding waste of resources and thus improving energy efficiency. The processor scheduling system can effectively improve system performance and efficiency and meet growing computing needs by reasonably and effectively managing and utilizing system resources.

[0052] Based on the structure of the above processor scheduling system, the following Figure 1-Figure 3 The processor scheduling method according to the embodiment of the present invention is described with examples.

[0053] like Figure 2 As shown, the processor scheduling method of the embodiment of the present invention at least includes step S1 and step S2.

[0054] Step S1, determining the target task according to the operating status of the vehicle.

[0055] Among them, the target task is the task with the highest priority corresponding to different operating states. By determining the target task, the processing requirements of visual data in different scenarios can be adapted.

[0056] In the embodiment, when the vehicle is started, there will be different operating states, such as moving, reversing, parking, turning, merging, etc. Each operating state needs to take into account the perception tasks under the current road scene, dynamically adjust the priority of the road scene perception tasks, and determine the task with the highest priority as the target task. By flexibly adjusting the target task according to the actual situation, it meets the needs of complex and changeable real scenes, and can enhance the stability and safety of the processor scheduling system, which is more suitable for the complex road scene perception tasks in ADAS.

[0057] Step S2, scheduling the target task to multiple neural network processors, at least one of the multiple neural network processors processes the target task data corresponding to the target task.

[0058] In an embodiment, after determining the target task, the task allocation objects and processing order of multiple neural network processors are dynamically adjusted, and the target task is scheduled to at least one of the multiple neural network processors, for example, including a first neural network processor NPU0 and a second neural network processor NPU1. After receiving the target task, the above-mentioned neural network processors give priority to executing the target task, that is, the first neural network processor NPU0 and the second neural network processor NPU1 give priority to reading and processing the target task data corresponding to the target task, including image data and video data detected by multiple cameras of the vehicle, etc., and by processing the target task data, the real-time execution of the target task is guaranteed, and by simultaneously utilizing the computing power of two or more neural network processors, the processing efficiency of the entire processor scheduling system is improved, which is suitable for ADAS scenarios and has high compatibility.

[0059] According to the processor scheduling method of an embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0060] In some embodiments, determining a target task according to the operating state of the vehicle includes: obtaining a task queue of a scene in which the vehicle is currently located; and determining the highest priority task in the task queue as the target task according to the operating state.

[0061] In an embodiment, in a vehicle-mounted system, a task queue of the current scene of the vehicle is obtained through relevant external components such as multiple cameras, and specific tasks are assigned to different cameras according to their fields of view and characteristics, wherein the multiple cameras include a front-view camera, a rear-view camera and a surround-view camera, and each camera is responsible for implementing different functions, thereby ensuring the intelligence and safety of the driving system, and task segmentation is performed based on the camera. These tasks are independent and have different characteristics, and tasks with higher real-time requirements can be processed in a timely manner; under different vehicle operating conditions, the priority of each task in the task queue will also change accordingly. At this time, task switching is performed according to the operating status, and the task with the highest priority in the task queue is set as the target task.

[0062] In some embodiments, obtaining a task queue for a scene in which the vehicle is currently located includes: receiving a data frame of a scene in which the vehicle is currently located in response to a neural network processing control instruction; determining a forward-looking task, a surround-looking task, and a rear-looking task according to the type of the data frame; and determining the forward-looking task, the surround-looking task, and the rear-looking task as a task queue.

[0063] In an embodiment, the CPU sends specific neural network processing control instructions to the processor scheduling system to control it to perform scene perception task operations, and at the same time controls related external components such as cameras to prepare data frames required by the processor scheduling system and save the data frames to the memory. The processor scheduling system responds to the neural network processing control instructions and receives data frames of the scene in which the vehicle is currently located, involving operations such as reading, transmitting, and caching of data frames.

[0064] When the processor scheduling system receives data frames from multiple cameras, it performs fine-grained task segmentation according to the camera type to which the data frames belong, that is, all image acquisition tasks are divided into multiple tasks, among which the data frame processing of the front camera, rear camera, and surround camera can be regarded as an independent task, namely, the front-view task, surround-view task, and rear-view task. If the data frame belongs to the front-view camera, the front-view task can be subdivided into: obstacle detection, road line recognition, and traffic signal recognition, etc. The above information is crucial for driving and requires a lot of computing resources; if the data frame belongs to the surround-view camera, the surround-view task can be subdivided into: vehicle tracking, pedestrian detection, and alarm notification, etc. This task needs to be carried out continuously throughout the vehicle's driving, but it is more important when turning, merging, reversing, and parking; if the data frame belongs to the rear-view camera, the rear-view task can be subdivided into: parking space detection, parking equipment detection, and environmental perception, etc., which are more important when reversing. By making full use of the characteristics and application requirements of each camera, more accurate information can be provided to the driver, and at the same time, it can be optimized according to the characteristics and requirements of the task, thereby improving the processing efficiency of the entire system and preparing for the subsequent dynamic scheduling.

[0065] In some embodiments, a target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes target task data corresponding to the target task, including: when the vehicle is in a forward state, determining that the target task is a forward-looking task in the task queue; scheduling the forward-looking task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward-looking task data corresponding to the forward-looking task.

[0066] In an embodiment, when the vehicle is in a forward state, the forward-looking task has the highest priority due to the need to check the road conditions ahead, and the target task is determined to be the forward-looking task in the task queue. At this time, as many resources as possible need to be allocated to processing the forward-looking task, and the forward-looking task is scheduled to at least one of the multiple neural network processors, such as the first neural network processor NPU0 and / or the second neural network processor NPU1. The first neural network processor NPU0 and / or the second neural network processor NPU1 give priority to processing the forward-looking task data corresponding to the forward-looking task to ensure clear forward-looking camera imaging.

[0067] In some embodiments, a target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes target task data corresponding to the target task, including: when the vehicle is in a turning state, determining that the target task is a forward-looking task and a surround-looking task in the task queue; scheduling the forward-looking task and the surround-looking task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward-looking task data corresponding to the forward-looking task and the surround-looking task data corresponding to the surround-looking task.

[0068] In an embodiment, when the vehicle is in a turning state, the forward-looking task and the surround-looking task have a higher priority due to the need to check for obstacles in front and on the sides, and the target task is determined to be the forward-looking task and the surround-looking task in the task queue. At this time, it is necessary to allocate as many resources as possible to processing the forward-looking task and the surround-looking task, and schedule the forward-looking task and the surround-looking task to at least one of the multiple neural network processors, such as the first neural network processor NPU0 and / or the second neural network processor NPU1. The first neural network processor NPU0 and / or the second neural network processor NPU1 give priority to processing the forward-looking task data corresponding to the forward-looking task and the surround-looking task data corresponding to the surround-looking task, to ensure clear and accurate forward-looking video and surround-looking video.

[0069] In some embodiments, the target task is dispatched to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task, including: when the vehicle is in a reversing state, determining that the target task is a rear view task and a surround view task in the task queue; dispatching the rear view task and the surround view task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the rear view task data corresponding to the rear view task and the surround view task data corresponding to the surround view task.

[0070] In an embodiment, when the vehicle is in a reversing state, the rear view task and the surround view task have a higher priority due to the need to check for rear and side obstacles, and the target task is determined to be the rear view task and the surround view task in the task queue; the rear view task and the surround view task are scheduled to at least one of the multiple neural network processors, such as the first neural network processor NPU0 and / or the second neural network processor NPU1, and the first neural network processor NPU0 and / or the second neural network processor NPU1 give priority to processing the rear view task data corresponding to the rear view task and the surround view task data corresponding to the surround view task, to ensure clear and accurate rear view warnings and surround view photography.

[0071] In some embodiments, when at least one of a plurality of neural network processors processes target task data corresponding to a target task, it also includes: determining the overload number of overloaded neural network processors among a plurality of neural network processors; when the overload number is a first preset number, scheduling the target task executed by the overloaded neural network processor to a non-overloaded neural network processor; when the overload number is a second preset number, the overloaded neural network processor suspends processing non-target task data, wherein the first preset number is less than the second preset number.

[0072] In an embodiment, when at least one of the multiple neural network processors processes the target task data corresponding to the target task, the monitoring module monitors the status and performance load of the multiple neural network processors in real time. If the multiple neural network processors, such as the first neural network processor NPU0 and the second neural network processor NPU1, are load balanced, no operation is performed, and the current scheduling situation is continuously executed until the running state of the vehicle changes; if the multiple neural network processors, such as the first neural network processor NPU0 and the second neural network processor NPU1, are load uneven, a load balancing operation is performed to determine the overload number of the overloaded neural network processors, and when the overload number is the first preset number 1 , it is considered that one of the first neural network processor NPU0 and the second neural network processor NPU1 is overloaded and the other is not overloaded, then the target task executed by the overloaded neural network processor is scheduled to the non-overloaded neural network processor; when the overload number is the second preset number 2, it is considered that both the first neural network processor NPU0 and the second neural network processor NPU1 are overloaded, and there are no available NPU computing resources at this time, then the first neural network processor NPU0 and the second neural network processor NPU1 suspend processing of non-target task data used to enhance driving experience but not directly affecting safety, such as environmental recognition data or facial recognition data, to ensure the normal progress of the target task.

[0073] For example, when the vehicle is in a turning state, the forward-looking task and the surround-looking task are dispatched to the first neural network processor NPU0 and / or the second neural network processor NPU1, and the task loads of the first neural network processor NPU0 and the second neural network processor NPU1 are monitored. If the load of the first neural network processor NPU0 increases with the increase of the forward-looking task and the surround-looking task and becomes overloaded, while the load of the second neural network processor NPU1 is relatively small, a part of the forward-looking task and the surround-looking task executed by the first neural network processor NPU0 will be automatically dispatched to the second neural network processor NPU1. It should be noted that although this will reduce the load burden of the first neural network processor NPU0, it is also necessary to ensure that the second neural network processor NPU1 can still quickly process the rear-looking task in an emergency situation, such as when emergency braking is required.

[0074] When the vehicle is in the reversing state, part of the rear-view task and the surround-view task are automatically transferred to the first neural network processor NPU0 to enhance the processing capability of the rear-view function; when surround-view parking assistance is required, the importance of the surround-view task is significantly increased. At this time, the forward-view task or the rear-view task is appropriately reduced, and more resources are allocated to the processing of the surround-view task to provide all-round parking assistance.

[0075] In the above example, the monitoring module can be software, hardware, or a combination of software and hardware, depending on the specific implementation plan, and is used to monitor the load of each neural network processor in real time, including the occupancy rate of the neural network processor, the occupancy of the memory, and the processing time of each task. If the load of the first neural network processor NPU0 is too large, some forward-looking or surround-looking tasks will be transferred to the second neural network processor NPU1 for processing in a timely manner; conversely, if the load of the second neural network processor NPU1 is too large, some rear-view tasks can be transferred to the first neural network processor NPU0 for processing. By real-time monitoring of the status and resource allocation of each neural network processor and switching the corresponding resource allocation mode, the load balancing of the processor scheduling system is ensured, overload of a single neural network processor is avoided, and it is flexible enough to cope with various changes, thereby ensuring safe and efficient operation of the vehicle.

[0076] In some embodiments, the target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task, and also includes: at least one of the multiple neural network processors executes a calling function to complete the current task and suspends the current task.

[0077] In an embodiment, when the compiled data instruction performs inference calculation on the NPU, it will be expressed as one or more calling functions, where the calling function is the execution granularity of the neural network processor, and multiple calling function tasks will be scheduled in sequence on the hardware queue of the neural network processor. When all the calling functions of a task are executed, then the task is also executed.

[0078] When a target task is scheduled to multiple neural network processors, it is considered that a task switch is executed. Prior to this, at least one of the multiple neural network processors is required to execute the calling function of the current task. After at least one of the multiple neural network processors executes the calling function of the current task, the current task is temporarily suspended and the context of the task is saved. After at least one of the multiple neural network processors processes the target task data, the processing of the current task data is resumed.

[0079] For example, when the vehicle is driving, the first neural network processor NPU0 is executing a forward-looking task. If the vehicle switches to the reverse state, the rear-view task has the highest priority, that is, the task of the rear-view camera needs to be processed first. If the resources of the second neural network processor NPU1 are scarce at this time, the first neural network processor NPU0 needs to temporarily suspend the forward-looking task after executing the calling function of the current forward-looking task, and execute the rear-view task instead. When the rear-view task is completed or the resources are no longer scarce, the execution of the forward-looking task is resumed.

[0080] In some embodiments, determining the target task according to the operating state of the vehicle also includes: scheduling the forward-looking task in the task queue to the first neural network processor, and scheduling the surround-looking task and the rear-looking task in the task queue to the second neural network processor.

[0081] In an embodiment, when the ADAS system is started, that is, the vehicle is started, the processor scheduling system is initialized, including powering on the two cores of the processor scheduling system, namely the first neural network processor NPU0 and the second neural network processor NPU1, allocating clocks, etc.

[0082] When the vehicle is ready to start, at this time, since the vehicle is still in a stationary state, task scheduling is relatively simple, and the environment and conditions around the vehicle body are relatively fixed, the task queue is assigned to the two neural network processors according to the static scheduling method. The front-view camera is responsible for tasks such as detecting the road conditions ahead, pedestrian detection, and traffic signal detection, and has the highest priority by default; in order to ensure the priority and real-time nature of the forward-looking task data, the priority of the forward-looking task in the task queue will be raised to the highest, and the forward-looking task will be scheduled to the first neural network processor NPU0, which processes the forward-looking task data for road segmentation; and the surround-view task and rear-view task in the task queue will be scheduled to the first neural network processor NPU0. The second neural network processor NPU1 processes the rear-view task data and the surround-view task data for other scene perception. By allocating the more important and complex forward-view task to the first neural network processor NPU0 with a larger load, and allocating the relatively lighter rear-view and surround-view tasks to the second neural network processor NPU1, it is ensured that more computing resources can be obtained when processing the forward-view task; the rear-view task and the surround-view task are relatively simple and more important when driving, so it is extremely necessary to allocate them independently to a neural network processor. The initial tasks are reasonably allocated by fixing the task priority, and static scheduling is achieved according to the characteristics of the tasks.

[0083] Reference below Figure 3 The processor scheduling method according to the embodiment of the present invention is described with examples.

[0084] like Figure 3 As shown, the processor scheduling method of the embodiment of the present invention at least includes steps S11 to S24.

[0085] Step S11, scheduling the forward-looking tasks in the task queue to the first neural network processor, and scheduling the surround-looking tasks and the rear-looking tasks in the task queue to the second neural network processor.

[0086] Step S12, determine whether the running state of the vehicle is a stationary state, if so, execute step S23; otherwise, execute step S13.

[0087] Step S13, in response to the neural network processing control instruction, receiving the data frame of the scene in which the vehicle is currently located; determining the forward-looking task, the surround-looking task and the rear-looking task as the task queue according to the type of the data frame.

[0088] Step S14: at least one of the multiple neural network processors executes a calling function to complete the current task and suspends the current task.

[0089] Step S15, when the vehicle is in a forward state, determine that the target task is a forward-looking task in the task queue; schedule the forward-looking task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward-looking task data corresponding to the forward-looking task.

[0090] Step S16, when the vehicle is in a turning state, determine that the target task is a forward-looking task and a surround-looking task in the task queue; schedule the forward-looking task and the surround-looking task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the forward-looking task data corresponding to the forward-looking task and the surround-looking task data corresponding to the surround-looking task.

[0091] Step S17, when the vehicle is in a reversing state, determine that the target task is a rear view task and a surround view task in the task queue; schedule the rear view task and the surround view task to at least one of the multiple neural network processors, and at least one of the multiple neural network processors processes the rear view task data corresponding to the rear view task and the surround view task data corresponding to the surround view task.

[0092] Step S18, determining the overload number of overloaded neural network processors among the multiple neural network processors.

[0093] Step S19, determining whether the overload quantity is a first preset quantity, if so, executing step S20; otherwise, executing step S21.

[0094] Step S20, scheduling the target task executed by the overloaded neural network processor to a non-overloaded neural network processor.

[0095] Step S21, determining whether the overload quantity is a second preset quantity, if so, executing step S22; otherwise, executing step S23.

[0096] Step S22: the overloaded neural network processor suspends processing of non-target task data.

[0097] Step S23: do not perform any operation.

[0098] Step S24, at least one of the multiple neural network processors resumes processing the current task data.

[0099] According to the processor scheduling method of an embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0100] Combine the following Figure 4 The processor scheduling device 5 according to the embodiment of the present invention is described.

[0101] like Figure 4 As shown, the processor scheduling device 5 of the embodiment of the present invention includes: a determination module 11 and a scheduling module 12, wherein:

[0102] The determination module 11 is used to determine the target task according to the operating status of the vehicle; the scheduling module 12 is used to schedule the target task to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task.

[0103] According to the processor scheduling device 5 of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the allocation object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0104] An electronic device according to an embodiment of the present invention comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a processor scheduling program executable by the at least one processor, and when the processor scheduling program is executed by the at least one processor, the at least one processor executes the processor scheduling method of the above embodiment.

[0105] According to the electronic device of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0106] Reference below Figure 5 A vehicle 10 according to an embodiment of the present invention will be described.

[0107] like Figure 5 As shown, the vehicle 10 of the embodiment of the present invention includes the processor scheduling device 5 of the above embodiment, or includes the electronic device of the above embodiment.

[0108] According to the vehicle 10 of the embodiment of the present invention, under different vehicle operating conditions, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0109] The following describes a computer-readable storage medium according to an embodiment of the present invention.

[0110] A processor scheduling program is stored on a computer-readable storage medium in an embodiment of the present invention. When the processor scheduling program is executed by a processor, the processor scheduling method of the above embodiment is implemented.

[0111] The following describes a computer program product according to an embodiment of the present invention.

[0112] The computer program product of the embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the processor scheduling method of the above embodiment is implemented.

[0113] According to the computer program product of the embodiment of the present invention, under the operating conditions of different vehicles, the perception task with the highest priority in the current road scene is adjusted to the target task, and the assignment object and processing order of the target task are determined to schedule the target task to multiple neural network processors. At least one of the multiple neural network processors preferentially executes the target task and processes the target task data corresponding to the target task. By flexibly adjusting the target task according to the actual operating status of the vehicle, it meets the needs of complex and changeable real scenes, is more adaptable to the complex road scene perception tasks in ADAS, and improves the scheduling efficiency of the processor.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0115] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A processor scheduling method, characterized in that: include: Determine the target task according to the vehicle's operating status; The target task is scheduled to multiple neural network processors, and at least one of the multiple neural network processors processes the target task data corresponding to the target task.

2. The processor scheduling method according to claim 1, characterized in that: Determine the target tasks according to the vehicle's operating status, including: Obtaining a task queue of a scene in which the vehicle is currently located; According to the running status, the task with the highest priority in the task queue is determined as the target task.

3. The processor scheduling method according to claim 2, characterized in that: Obtaining the task queue of the scene in which the vehicle is currently located, including: In response to the neural network processing control instruction, receiving a data frame of a scene currently located by the vehicle; Determining a forward-looking task, a surround-looking task, and a backward-looking task according to the type of the data frame; The forward-looking task, the surround-looking task, and the backward-looking task are determined as the task queue.

4. The processor scheduling method according to claim 3, characterized in that: Scheduling the target task to a plurality of neural network processors, wherein at least one of the plurality of neural network processors processes target task data corresponding to the target task, comprises: When the vehicle is in a forward state, determining the target task to be the forward-looking task in the task queue; The forward-looking task is scheduled to at least one of the plurality of neural network processors, and at least one of the plurality of neural network processors processes the forward-looking task data corresponding to the forward-looking task.

5. The processor scheduling method according to claim 3, characterized in that: Scheduling the target task to a plurality of neural network processors, wherein at least one of the plurality of neural network processors processes target task data corresponding to the target task, comprises: When the vehicle is in a turning state, determining the target task to be the forward-looking task and the surround-looking task in the task queue; The forward-looking task and the surround-looking task are dispatched to at least one of the plurality of neural network processors, and at least one of the plurality of neural network processors processes the forward-looking task data corresponding to the forward-looking task and the surround-looking task data corresponding to the surround-looking task.

6. The processor scheduling method according to claim 3, characterized in that: Scheduling the target task to a plurality of neural network processors, wherein at least one of the plurality of neural network processors processes target task data corresponding to the target task, comprises: When the vehicle is in a reversing state, determining the target task to be the rear-view task and the surround-view task in the task queue; The rear-view task and the surround-view task are dispatched to at least one of the plurality of neural network processors, and at least one of the plurality of neural network processors processes rear-view task data corresponding to the rear-view task and surround-view task data corresponding to the surround-view task.

7. The processor scheduling method according to claim 1, characterized in that: When at least one of the plurality of neural network processors processes the target task data corresponding to the target task, it also includes: determining an overload number of an overloaded neural network processor among the plurality of neural network processors; When the overload quantity is a first preset quantity, scheduling the target task executed by the overloaded neural network processor to a non-overloaded neural network processor; When the overload quantity is a second preset quantity, the overloaded neural network processor suspends processing non-target task data, wherein the first preset quantity is less than the second preset quantity.

8. The processor scheduling method according to claim 1, characterized in that: Scheduling the target task to a plurality of neural network processors, at least one of the plurality of neural network processors processing target task data corresponding to the target task, further comprising: At least one of the plurality of neural network processors executes a calling function to complete a current task and suspends the current task.

9. The processor scheduling method according to claim 1, characterized in that: Scheduling the target task to a plurality of neural network processors, at least one of the plurality of neural network processors processing target task data corresponding to the target task, further comprising: At least one of the plurality of neural network processors resumes processing current task data.

10. The processor scheduling method according to claim 1, characterized in that: Determine the target task according to the vehicle's operating status, and also include: The forward-looking tasks in the task queue are dispatched to the first neural network processor, and the surround-looking tasks and the backward-looking tasks in the task queue are dispatched to the second neural network processor.

11. A processor scheduling device, characterized in that: include: A determination module, used to determine a target task according to the operating status of the vehicle; A scheduling module is used to schedule the target task to multiple neural network processors, at least one of the multiple neural network processors processes the target task data corresponding to the target task.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a processor scheduling program that can be executed by the at least one processor. When the processor scheduling program is executed by the at least one processor, the at least one processor executes the processor scheduling method according to any one of claims 1 to 10.

13. A vehicle, characterized in that: Includes the processor scheduling device as claimed in claim 11, or includes the electronic device as claimed in claim 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a processor scheduling program, and when the processor scheduling program is executed by the processor, the processor scheduling method according to any one of claims 1 to 10 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the processor scheduling method according to any one of claims 1 to 10 is implemented.

Images