Algorithm scheduling method, heterogeneous system, equipment, chip system and storage medium

Abstract

The invention provides an algorithm scheduling method, a heterogeneous system, equipment, a chip system and a storage medium, and relates to the technical field of terminals. According to the scheme, in the method, when N algorithms need to be scheduled, data information of the N algorithms is written into a shared memory, and different memory blocks of the shared memory are used by data information of different algorithms. And the CPU sends instruction information carrying memory block information corresponding to each algorithm in the N algorithms to the DSP, so that the DSP can read data information of the N algorithms from memory blocks and execute tasks corresponding to the data information of the N algorithms. Therefore, the original N times of transmission (the information transmitted each time comprises the data information and the instruction information corresponding to the task of one algorithm) is reduced into one time of instruction information transmission and one time of data synchronization process, so that the scheduling overhead is reduced.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an algorithm scheduling method, heterogeneous system, device, chip system and storage medium. Background Technology

[0002] With the rapid development of technology, computing demands are becoming increasingly complex and diverse, making it difficult for traditional single processors to meet the needs of various application scenarios. Against this backdrop, heterogeneous systems have emerged.

[0003] Typically, heterogeneous systems consist of multiple processors integrated on a system-on-a-chip (SoC). Take, for example, a heterogeneous system composed of a central processing unit (CPU) and a digital signal processor (DSP). The CPU, as the core of the computer system's computation and control, is responsible for controlling and allocating all hardware resources and core scheduling. The DSP has high-efficiency numerical computation capabilities, suitable for performing image processing tasks such as image enhancement, data processing, and machine learning. Currently, self-developed algorithms or other algorithms use their own algorithm libraries for scheduling, with a single scheduling overhead from the CPU to the DSP of approximately 0.5 milliseconds. Since the scheduling time of each algorithm is controlled independently, if multiple algorithms initiate scheduling simultaneously, the cumulative scheduling overhead can be quite large. Summary of the Invention

[0004] This application provides an algorithm scheduling method, a heterogeneous system, a device, a chip system, and a storage medium, which solves the technical problem of high overhead caused by the CPU simultaneously scheduling the DSP to execute multiple algorithms.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, embodiments of this application provide an algorithm scheduling method. This method can be applied to electronic devices. The method can utilize a CPU and a DSP. The method may include:

[0007] The CPU determines N algorithms to be scheduled; the CPU sends an instruction to the DSP, which includes information about the memory block corresponding to each of the N algorithms, and these memory blocks belong to a shared memory; the CPU writes the data information of each algorithm into the memory block corresponding to each of the N algorithms; the DSP schedules the N algorithms to execute the tasks corresponding to the data information of the N algorithms. Here, N is a positive integer.

[0008] In the above scheme, the original N transmissions (each transmission including data and instruction information corresponding to an algorithm's task) are reduced to one instruction transmission and one data synchronization process. Since the data synchronization process has zero overhead, the scheduling overhead of this method is equal to the scheduling overhead of transmitting one instruction transmission, reducing the scheduling overhead from 0.5*N milliseconds to 0.5 seconds. It can be understood that the larger the value of N, the more significant the effect of this scheme in reducing scheduling overhead.

[0009] In one possible implementation, before scheduling N algorithms to execute tasks corresponding to the data information of the N algorithms via the DSP, the method may further include: reading data information from the memory block corresponding to each of the N algorithms via the DSP. It can be understood that since the CPU writes the data information of each algorithm into the memory block corresponding to each of the N algorithms, whenever the CPU updates the data information in the shared memory, the DSP can immediately obtain the updated data information and read the data information from the memory block corresponding to each algorithm; this process has zero overhead.

[0010] In one possible implementation, the instruction information may further include the priority corresponding to each of the N algorithms. Accordingly, scheduling the N algorithms to execute tasks corresponding to the data information of the N algorithms via the DSP can include: scheduling each of the N algorithms sequentially according to their priority from highest to lowest, executing the tasks corresponding to the data information of each algorithm. It can be understood that by setting a priority for each algorithm, higher-priority algorithms can be executed first, avoiding resource contention caused by executing multiple algorithms simultaneously.

[0011] In one possible implementation, the CPU has a first main algorithm library, and the DSP has a second main algorithm library. The first and second main algorithm libraries are used for unified scheduling of M preset algorithms. The M preset algorithms include N algorithms, where M is an integer greater than 1. Accordingly, sending an instruction message from the CPU to the DSP can include: sending an instruction message from the CPU's first main algorithm library to the DSP's second main algorithm library. The instruction message includes the scheduling policies for the M preset algorithms. The scheduling policy for each of the M preset algorithms includes: whether each preset algorithm is scheduled, the name of each preset algorithm, the name length of each preset algorithm, the priority of each preset algorithm, the starting sequence number of the memory block for each preset algorithm, and the number of memory blocks used by each preset algorithm. It can be understood that the scheduling policies for the above M preset algorithms are all carried in a single instruction message, and the size of the data carried in the instruction message does not affect the transmission speed of the instruction message. Taking a single scheduling overhead from CPU to DSP of approximately 0.5 milliseconds as an example, the CPU sends an instruction message to the DSP, so that the scheduling strategies of all preset algorithms are delivered to the DSP in one transmission, reducing the original N instruction message transmissions to 1 instruction message transmission, thereby shortening the scheduling overhead from the original 0.5*N milliseconds to 0.5 seconds.

[0012] In one possible implementation, the N algorithms are N modules in a second main algorithm library. Accordingly, scheduling the N algorithms to execute tasks corresponding to the data information of the N algorithms via the DSP includes: using the second main algorithm library of the DSP, based on the scheduling strategy of each preset algorithm, and in descending order of priority of the N algorithms, sequentially scheduling each module in the N modules to execute the tasks corresponding to the data information of each algorithm. It can be understood that in traditional scheduling schemes, the N algorithms on the DSP side are N individually executed algorithm modules. Self-developed algorithms cannot control the DSP's scheduling logic for these N algorithms. In some cases, less important algorithms may be prioritized, leading to some relatively important algorithms executing later, or in other cases, the N algorithms may compete for scheduling resources. However, in this application, by setting up a second main algorithm library on the DSP side, the N algorithms are N compilation units or N sub-modules in the second main algorithm library, and the second main algorithm library is responsible for the unified scheduling of the N algorithms.

[0013] In one possible implementation, using the DSP's second main algorithm library, based on the scheduling strategy of each preset algorithm, and in descending order of priority of the N algorithms, each module in the N modules is sequentially scheduled to execute the task corresponding to the data information of each algorithm. This can include: using the DSP's second main algorithm library, generating an algorithm scheduling table based on whether each preset algorithm in the scheduling strategy of M preset algorithms is scheduled or not, the algorithm scheduling table includes the N algorithms to be scheduled; then, in descending order of priority of the N algorithms in the algorithm scheduling table, each module in the N modules is sequentially scheduled to execute the task corresponding to the data information of each algorithm.

[0014] In one possible implementation, using the DSP's second main algorithm library, based on the scheduling strategy of each preset algorithm, and in descending order of priority among the N algorithms, each module in the N modules is sequentially scheduled to execute the task corresponding to the data information of each algorithm. This can include: using the DSP's second main algorithm library, scheduling the first module in the N modules to execute the task corresponding to the data information of the first algorithm in the first time period, and scheduling the second module in the N modules to execute the task corresponding to the data information of the second algorithm. The first and second algorithms have higher priorities than other unscheduled algorithms, and the DSP's thread resources support simultaneous scheduling of the first and second modules in the first time period. It can be understood that when the DSP's thread resources are sufficient to support simultaneous scheduling of algorithms, the DSP can improve task execution efficiency by using multi-threaded scheduling of multiple algorithms.

[0015] In one possible implementation, the CPU includes a camera scheduling module and a first main algorithm library. The CPU determines N algorithms to be scheduled, which may include: the camera scheduling module selecting N algorithms from M preset algorithms as the algorithms to be scheduled for processing a first image, where the first image is any frame from a video stream captured by the electronic device; and the CPU obtaining the N algorithms to be scheduled from the camera scheduling module. Different algorithms are scheduled to perform different tasks for different images.

[0016] In one possible implementation, before determining the N algorithms to be scheduled by the CPU, the method may further include: in response to a user's first operation, running a camera application and creating shared memory, the number of memory blocks included in the shared memory being determined based on M preset algorithms. It is understood that when an electronic device runs different applications, different applications will call different types and numbers of algorithms. During the development phase of the electronic device, developers can collect information on the various algorithms that may be called during the running of each application. This information may include, but is not limited to, the algorithm's name, name length, function, priority, and the amount of memory occupied by the algorithm's data. Based on this information, rules for creating shared memory when running each application can be formulated. As an example, the creation rules may include: the size of the shared memory region in physical memory, the number of memory blocks the shared memory is divided into, the starting sequence number of the memory block corresponding to each algorithm, and the number of memory blocks corresponding to each algorithm. Thus, when the electronic device starts running an application, shared memory can be created according to the rules corresponding to that application, so that when certain algorithms are called, the CPU can write the algorithm's data information into the memory block corresponding to that algorithm.

[0017] In one possible implementation, the method may further include: in response to a second user action, closing the camera application and reclaiming shared memory. It is understood that when the camera application is closed, the electronic device no longer needs to invoke the preset algorithm corresponding to the camera application to process image frames. Therefore, the electronic device can reclaim the shared memory with the camera application, thereby preventing the camera application from continuously occupying the shared memory and affecting other applications' creation and use of shared memory.

[0018] Secondly, this application provides a heterogeneous system, which may include a CPU and a DSP. The CPU is used to determine N algorithms to be scheduled; the CPU is also used to write data information for each of the N algorithms into a memory block corresponding to each algorithm, the memory blocks corresponding to the N algorithms belonging to a shared memory; the CPU is also used to send an instruction message to the DSP, the instruction message including the memory block corresponding to each of the N algorithms; the DSP is used to schedule the N algorithms to execute the tasks corresponding to the data information of the N algorithms.

[0019] Thirdly, this application provides an electronic device comprising a CPU, a DSP, and a memory. The memory is coupled to one or more processors and is used to store computer program code, including computer instructions. The CPU and DSP invoke the computer instructions to cause the electronic device to perform the methods provided in the first aspect and any possible implementation thereof.

[0020] Fourthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When executed on an electronic device, the computer instructions cause the electronic device to perform the method provided by the first aspect and any possible implementation thereof.

[0021] Fifthly, this application provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the method provided by the first aspect and any possible implementation thereof.

[0022] Sixthly, this application provides a chip system applied to an electronic device, the chip system including a CPU and a DSP, the CPU and DSP being used to invoke computer instructions to cause the electronic device to perform the methods provided in the first aspect and any possible implementation thereof.

[0023] It is understood that the beneficial effects that the heterogeneous system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip system of the sixth aspect can achieve can be referred to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0024] Figure 1 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of a CPU and DSP scheduling scheme provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0027] Figure 4 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0028] Figure 5 A schematic diagram illustrating the principle of shared memory provided in this application embodiment;

[0029] Figure 6 A schematic diagram of the preset algorithm and memory block provided in the embodiments of this application;

[0030] Figure 7 A schematic diagram illustrating the principle of the algorithm scheduling method provided in this application based on shared memory;

[0031] Figure 8 A flowchart illustrating the algorithm scheduling method provided in an embodiment of this application;

[0032] Figure 9A schematic diagram illustrating the processing of image frames in a video stream using various algorithms provided in the embodiments of this application;

[0033] Figure 10 A schematic diagram illustrating the unified scheduling of five algorithms by the second main algorithm library provided in this embodiment of the application;

[0034] Figure 11 This is a schematic diagram of a heterogeneous system provided in an embodiment of this application. Detailed Implementation

[0035] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or different treatments of the same object, rather than to describe a specific order of objects. Furthermore, the terms "comprising" and "having," and any variations thereof, mentioned in the description of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. In embodiments of this application, "multiple" includes two or more. In embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Additionally, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. In the description of this application, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone.

[0036] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Typically, heterogeneous systems consist of multiple processors integrated on a single System-on-a-Chip (SoC). Take, for example, a heterogeneous system composed of a CPU and a DSP. The CPU, as the computational and control core of the computer system, is responsible for controlling and allocating all hardware resources and core scheduling. DSPs possess high-efficiency numerical computation capabilities, making them suitable for image processing tasks such as image enhancement, data processing, and machine learning. DSP subsystems (such as aDSP, cDSP, dDSP, and mDSP) offer advantages like low power consumption and high efficiency, and are increasingly used in image enhancement, data processing, and machine learning operations, finding widespread application in fields such as cameras and audio. Many self-developed algorithms utilize DSPs for CV (computer vision) operators optimization, primarily for previewing algorithms. Previewing algorithms are characterized by short processing times, a large number of operators, and relatively consistent operator behavior.

[0038] For example, Figure 1This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application.

[0039] like Figure 1 As shown, electronic devices may include software platforms and hardware platforms.

[0040] The software platform can adopt a layered architecture, event-driven architecture, microkernel architecture, or microservice architecture. It can include various applications (such as camera applications, gallery applications, and image processing applications) and various algorithms that provide services to these applications (such as face detection algorithms, object tracking algorithms, single-target object tracking algorithms, human body tracking algorithms, face tracking algorithms, portrait exposure parameter adjustment algorithms, scene detection algorithms, and keypoint detection algorithms). During application execution, the application can invoke these algorithms to perform corresponding tasks.

[0041] The aforementioned software architecture runs on a hardware platform. In this embodiment, the hardware platform may include heterogeneous processors based on heterogeneous systems, displays (also called screens), depth estimation devices, and cameras, among other hardware structures. Heterogeneous processors can consist of multiple processors integrated onto a single System-on-a-Chip (SoC). For example, heterogeneous processors may include CPUs, DSPs, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). Typically, the CPU, as the computational and control core of the computer system, is responsible for controlling and allocating all hardware resources and scheduling core operations. DSPs possess highly efficient numerical computation capabilities. For computationally intensive batch processing tasks, such as image enhancement, data processing, and machine learning tasks, the CPU can deploy tasks to the DSP for execution. The CPU deploying an algorithm task to the DSP for execution is called a single scheduling process.

[0042] Currently, both self-developed algorithms and other algorithms use their own algorithm libraries for scheduling, with a single scheduling overhead from CPU to DSP of approximately 0.5 milliseconds. The algorithm library can be understood as an executable program; electronic devices can call the algorithm library to implement corresponding functions, and the implementation of the algorithm library depends on hardware resources such as CPU and DSP.

[0043] For example, Figure 2 This is a schematic diagram of a CPU and DSP scheduling scheme provided in an embodiment of this application.

[0044] like Figure 2As shown, each algorithm uses its own algorithm library for scheduling. For example, each algorithm can independently call the FastRPC framework to transmit instruction and data information. FastRPC is a remote procedure call (RPC) framework that enables calls between two programs, which can reside in different processes or on different chips. In this embodiment, FastRPC can enable calls between CPU-side algorithms (e.g., CPU-side algorithm 0) and DSP-side algorithms (e.g., DSP-side algorithm 0). During a single scheduling process for each algorithm, the CPU-side algorithm transmits instruction and data information to the DSP-side algorithm. If there are N algorithms, N scheduling processes are executed, with each scheduling process transmitting instruction and data information.

[0045] In the above process, since the scheduling time of each algorithm is controlled independently, if multiple algorithms initiate scheduling at the same time, the cumulative scheduling overhead of the multiple algorithms will reach 0.5*N milliseconds. For example, 10 algorithms will accumulate 5ms of scheduling overhead. It is understandable that the larger the value of N, the greater the cumulative scheduling overhead of the N algorithms, which may lead to resource contention issues between different schedulers.

[0046] In view of the above problems, this application provides an algorithm scheduling method. This method can be applied to electronic devices equipped with a CPU and a DSP. In this method, when N algorithms need to be scheduled, the data information of the N algorithms is written into a shared memory, with different memory blocks of the shared memory used by different algorithms. The CPU sends instruction information to the DSP carrying the memory block information corresponding to each of the N algorithms. The DSP can then read the data information of the N algorithms from the memory block and execute the tasks corresponding to the data information of the N algorithms. Thus, the original N transmissions (each transmission includes data information and instruction information corresponding to the task of one algorithm) are reduced to one instruction transmission and one data synchronization process. Since the data synchronization process has zero overhead, the scheduling overhead of this method is the same as the scheduling overhead of transmitting one instruction, reducing the scheduling overhead from the original 0.5*N milliseconds to 0.5 seconds. It can be understood that the larger the value of N, the more significant the effect of this scheme in reducing scheduling overhead.

[0047] The aforementioned electronic devices can be terminal devices. Terminal devices are also called terminals or user equipment (UE). For example, terminal devices can be mobile phones, personal computers (PCs), smart screens, smart TVs, tablets, wearable devices, computers with wireless transceiver capabilities, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, etc., or they can be other devices or apparatuses.

[0048] It should be noted that the embodiments in this application are illustrated by example of executing the algorithm scheduling method before the CPU and DSP, and do not constitute a limitation on this application. It is understood that the algorithm scheduling method provided in this application can also be used for other processors in heterogeneous systems, depending on actual usage requirements.

[0049] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0050] like Figure 3 As shown, the electronic device 100 may include an SOC 110, a memory 120, buttons 130, a sensor module 140, a display screen 150, an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, and a camera 170, etc.

[0051] The SOC 110 can be used to execute the algorithm scheduling method in the embodiments of this application. The SOC 110 may include one or more processing units, such as a CPU, DSP, GPU, application processor (AP), neural processing unit (NPU), image signal processor (ISP), etc. Different processing units can be independent devices; they can also be integrated into one or more processors, for example, the NPU can be set in the DSP.

[0052] The memory 120 can be used to store computer executable program code, which includes instructions. The SOC 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application (APP) required for a function, such as a camera application or a gallery application. The data storage area may store configuration files for each APP, as well as data created during the use of the electronic device 100, such as data information for algorithms to be invoked.

[0053] Buttons 130 include a power button, volume buttons, etc. Buttons 130 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100, such as key signal inputs to trigger camera 170 to capture images.

[0054] Sensor module 140 may include image sensors, touch sensors, etc. An image sensor, also known as a photosensitive element, is a device that uses the photoelectric conversion function of an optoelectronic device to convert a light image on a photosensitive surface into an electrical signal proportional to the light image. For example, an image sensor may be a complementary metal-oxide-semiconductor image sensor (CMOS image sensor, CIS). A touch sensor, also called a "touch panel," can be disposed on the display screen 150. The touch sensor and the display screen 150 together form a touch screen, also called a "touch screen." The touch sensor is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event and provide visual output related to the touch operation through the display screen 150. In some embodiments, the touch sensor may also be disposed on the surface of the electronic device 100, in a different position than the display screen 150. It should be noted that the above sensors can be independent functional modules in the electronic device or can be disposed within certain functional devices; for example, an image sensor can be integrated into a camera 170.

[0055] The display screen 150 includes a display panel for displaying the desktop, a camera preview interface, various images in the gallery, etc.

[0056] Electronic device 100 can implement audio functions, such as music playback and recording, through an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, and an application processor. The audio module 160 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 160 can also be used for encoding and decoding audio signals. The speaker 160A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through the speaker 160A. The receiver 160B, also called a "handpiece," is used to convert audio electrical signals into sound signals. When electronic device 100 answers a phone call or voice message, the receiver 160B can be brought close to the user's ear to hear the voice. The microphone 160C, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to the microphone 160C, inputting the sound signal into the microphone 160C. The headphone jack 160D is used to connect wired headphones.

[0057] Camera 170 is used to capture still images or videos. Light from an object passes through a lens to generate an optical image, which is then projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 170, where N is a positive integer greater than 1.

[0058] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0059] The following provides an example illustration of the software system of an electronic device. The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, or microservice architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system architecture of an electronic device.

[0060] For example, Figure 4 A schematic diagram of the software structure of the electronic device provided in an embodiment of this application is shown.

[0061] like Figure 4 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software architecture are divided from top to bottom as follows: application layer, application framework (FW) layer, system library (FWK LIB), hardware abstraction layer (HAL) layer, and kernel layer. This software architecture runs on top of the hardware layers, which may include components such as a System-on-a-Chip (SoC), a display screen, and a camera.

[0062] The application layer, also known as the application layer, can include a series of application packages. For example, the application layer can include a camera application, a gallery application, etc. The camera application is used to call the camera to capture tiny streams and image streams, and display the captured preview interface; the gallery application is used to store and display the final photos obtained through the camera application. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent business logic.

[0063] The application framework layer provides APIs and a programming framework for applications in the application layer. It includes predefined functions and may include an Activity Manager Service (AMS), a Window Manager Service (WMS), and a camera service. AMS manages the lifecycle of each application. WMS manages all windows in the system. The camera service may include a camera scheduling module (also known as an algorithm scheduling system). The camera scheduling module determines the algorithm to be scheduled for each frame in the preview video stream captured by the camera and executes the corresponding image processing task by calling the corresponding algorithm through the abstract interfaces of various algorithms in the HAL layer.

[0064] The system library can include multiple functional modules, such as a surface manager, media libraries, a 2D graphics engine (e.g., SGL), and a 3D graphics processing library (e.g., OpenGLES). The surface manager manages the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats. The 2D graphics engine is the drawing engine for 2D graphics. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0065] Within the system libraries, the Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part comprises the core Android libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection, among other functions.

[0066] The hardware abstraction layer has standard interfaces implemented by hardware vendors. For example, the hardware abstraction layer may include camera HAL and abstract interfaces for various algorithms. These abstract interfaces may include HAL interfaces for face detection algorithms, object tracking algorithms, single-target object tracking algorithms, human body tracking algorithms, face tracking algorithms, portrait exposure parameter adjustment algorithms, scene detection algorithms, and keypoint detection algorithms. It should be noted that this application does not specifically limit the types of algorithms and corresponding HAL interfaces that the electronic device can provide.

[0067] The kernel layer is the layer between hardware and software, and belongs to the lower level of the Android system. The kernel layer can contain various driver interfaces, such as camera drivers, display drivers, audio drivers, etc.

[0068] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0069] Below Figure 3 and Figure 4 Based on the provided functional modules, combined with Figures 5 to 11The specific implementation of the algorithm scheduling method provided in the embodiments of this application is illustrated with examples.

[0070] Referring to the description of the above embodiments, in traditional algorithm scheduling schemes, N data information corresponding to N algorithms is transmitted from the CPU to the DSP in N separate transmissions via the FastRPC framework. However, in the algorithm scheduling method of this application, the CPU writes the data information of the N algorithms into a shared memory, with different algorithm data information using different memory blocks of this shared memory. Then, the DSP can read the data information of the N algorithms from the memory blocks. This reduces the number of data transmissions from N to a single data synchronization process, significantly reducing scheduling overhead. For ease of understanding, the following will first illustrate this... Figure 5 The principles of shared memory will be introduced, and then... Figure 6 The principle of shared memory partitioning will be introduced, followed by... Figure 7 This paper introduces how to implement the algorithm scheduling method provided in this application based on shared memory.

[0071] For example, Figure 5 This is a schematic diagram illustrating the principle of shared memory provided in an embodiment of this application.

[0072] like Figure 5 As shown, shared memory is an inter-process communication (IPC) mechanism that allows multiple processes to access and manipulate the same physical memory region, thereby achieving data sharing and communication. In the use of shared memory, processes create shared memory segments through specific system calls, and then each process maps this shared memory segment into its own address space. Specifically, the steps for data exchange based on shared memory are as follows: First, the shared memory is created. The operating system first needs to allocate a region in physical memory as shared memory, which can be accessed by multiple processes. Second, memory mapping. Processes use interfaces provided by the operating system to map the shared memory region into their own address space (virtual memory), so that processes can access the shared memory as if it were their own private memory. Third, data access and synchronization. When one process writes data to shared memory, other processes sharing that memory region can immediately see the updated content. Shared memory is widely used in scenarios requiring high-speed data exchange due to its efficiency. The use of shared memory involves steps such as creation, mapping, reading, writing, and releasing memory, which usually require system calls or functions provided by the operating system.

[0073] In the algorithm scheduling method provided in this application, the data information of N algorithms to be scheduled is stored in a shared memory. In order to facilitate the DSP to distinguish the data information of each algorithm stored in the same shared memory, this application proposes a scheme to divide the shared memory into blocks, that is, to divide the shared memory into M memory bodies / blocks, and different algorithms use different memory blocks to store their data information.

[0074] Typically, when electronic devices run different applications, these applications invoke different types and numbers of algorithms. During the development phase of an electronic device, developers can collect information about the algorithms that might be invoked during the execution of various applications. This information can include, but is not limited to, the algorithm's name, name length, function, priority, and the amount of memory occupied by its data. Based on this information, they can then formulate rules for creating shared memory when running various applications. As an example, these rules might include: the size of the shared memory region in physical memory (determined based on the maximum number of algorithms that might be invoked during the execution of an application), the number of memory blocks the shared memory is divided into, the starting sequence number of each memory block corresponding to each algorithm, and the number of memory blocks corresponding to each algorithm (for example, some algorithms with less data may use one memory block, while some algorithms with more data may use multiple memory blocks). Thus, when an electronic device starts running an application, shared memory can be created according to the rules corresponding to that application, allowing the CPU to write the algorithm's data into the memory block corresponding to that algorithm when certain algorithms are invoked.

[0075] Taking the aforementioned N algorithms as examples of those scheduled during the execution of a camera application, the shared memory creation process, according to the rules corresponding to the camera application, is as follows: The system receives a user's action to launch the camera application, such as clicking the camera application icon; in response to the user's first action, the camera application is run, and shared memory is created according to the rules corresponding to the camera application. This shared memory is used to store data information involved in the camera application's invocation of various algorithms. Furthermore, the number of memory blocks included in the shared memory is determined based on the M preset algorithms corresponding to the camera application.

[0076] For example, Figure 6 This is a schematic diagram of the preset algorithm and memory block provided in the embodiments of this application.

[0077] The camera application corresponds to M preset algorithms, where M is an integer greater than 1. Taking M=32 as an example, if each memory block is 8MB and each preset algorithm corresponds to one memory block, then 32 memory blocks equal 256MB, meaning the shared memory is 256MB. For example... Figure 6As shown, when creating shared memory according to the rules corresponding to the camera application, each memory block is marked with a sequence number to distinguish it from other memory blocks. For example, the sequence number of the first memory block is Block1, the sequence number of the second memory block is Block2, the sequence number of the third memory block is Block3, and so on, with the sequence number of the 32nd memory block being Block32. Specifically, the first memory block can be used to store data for preset algorithm 1, the second memory block can be used to store data for preset algorithm 2, the third memory block can be used to store data for preset algorithm 3, and so on, with the 32nd memory block being used to store data for preset algorithm 32.

[0078] During the operation of a camera application, the electronic device may perform different processing operations on different image frames of the preview video stream, potentially invoking different algorithms. When deciding which algorithms to invoke, the CPU needs to write the data information for those algorithms into the corresponding memory blocks. For example, if image frame 1 requires the invocation of algorithms 1 and 2, the CPU writes the data information for algorithm 1 into memory block number 1 and the data information for algorithm 2 into memory block number 2. Similarly, if image frame 2 requires the invocation of algorithms 2, 5, and 6, the CPU writes the data information for algorithm 2 into memory block number 2, the data information for algorithm 5 into memory block number 5, and the data information for algorithm 6 into memory block number 6.

[0079] It should be noted that, Figure 6 This explanation uses the example of a preset algorithm being allocated one memory block, and does not limit the scope of the embodiments in this application. In other embodiments, a preset algorithm may be allocated multiple memory blocks. For example, some preset algorithms have a large amount of data, and one memory block may not be sufficient to meet the storage requirements. In this case, the electronic device can allocate multiple memory blocks based on the size of the data.

[0080] In some embodiments, as the number of image frames in the preview video stream increases, the data information of each memory block also increases. When the data information of calling an algorithm in the current frame is written to the memory block corresponding to the algorithm, if the memory block has already stored data information, the data information of the current frame can overwrite the original stored data information, so that the CPU can read the latest data information from the memory block and implement the corresponding function based on the data information.

[0081] In some embodiments, when the application terminates, such as when the electronic device receives a second user action, the electronic device can close the camera application and reclaim the shared memory with the camera application. It is understood that when the camera application is closed, the electronic device no longer needs to call the preset algorithm corresponding to the camera application to process image frames. Therefore, the electronic device can reclaim the shared memory with the camera application, thereby preventing the camera application from continuously occupying the shared memory and affecting other applications' creation and use of shared memory.

[0082] For example, Figure 7 This is a schematic diagram illustrating the principle of the algorithm scheduling method provided in this application based on shared memory.

[0083] like Figure 7 As shown, each algorithm's program consists of two parts: one part resides on the CPU side, and the other part resides on the DSP side. The CPU-side algorithm is responsible for sending instruction and data information, while the DSP-side algorithm is responsible for executing the corresponding task based on the instruction and data information. In the algorithm scheduling provided in this application, a first main algorithm library is set up on the CPU side, and a second main algorithm library is set up on the DSP side. The first main algorithm library is responsible for the unified scheduling of M preset algorithms on the CPU side, and the second main algorithm library is responsible for the unified scheduling of M preset algorithms on the DSP side. These M preset algorithms are all preset algorithms corresponding to the currently running application. These M preset algorithms include N algorithms to be scheduled, such as Algorithm 0, Algorithm 1, Algorithm 2... Algorithm N.

[0084] When N algorithms need to be scheduled, the data information of the N algorithms is written to the corresponding memory blocks of shared memory. For example, the CPU writes the data information Task0 of algorithm 0 to the memory block of algorithm 0, the data information Task1 of algorithm 1 to the memory block of algorithm 1, the data information Task2 of algorithm 2 to the memory block of algorithm 2, and so on, and writes the data information Task2 of algorithm N to the memory block of algorithm N.

[0085] On the CPU side, the first main algorithm library can collect instruction information for N algorithms and send all the information, including the memory block information corresponding to each of the N algorithms and the instruction information of the N algorithms, to the DSP through a single instruction.

[0086] On the DSP side, N algorithms can be understood as N compilation units or N sub-modules, and the second main algorithm library is responsible for the unified scheduling of these N algorithms. After receiving an instruction from the first main algorithm library, the second main algorithm library can generate an algorithm scheduling table, which includes the N algorithms to be scheduled. Then, according to the priority of the N algorithms in the scheduling table from high to low, each of the N modules is scheduled to execute the task corresponding to the data information of each algorithm. This can be referred to the description in the following embodiment, which will not be repeated here.

[0087] Based on the above embodiments, the specific process of the algorithm scheduling method provided in this application is described below.

[0088] For example, Figure 8 This is a flowchart illustrating the algorithm scheduling method provided in an embodiment of this application. This method can be applied to, for example... Figure 3 and Figure 4 The electronic device shown may include at least a CPU and a DSP.

[0089] like Figure 8 As shown, the method may include the following steps S101 to S105.

[0090] S101, the CPU determines N algorithms to be scheduled, where N is a positive integer.

[0091] Taking the aforementioned N algorithms as an example for camera application scheduling, the CPU is equipped with a camera scheduling module and a first main algorithm library. During the operation of the camera application, the camera captures and previews the video stream. Each time a frame is output, the camera scheduling module of the camera application can determine N algorithms corresponding to the current frame (e.g., the first image) from M preset algorithms, which are then used as the scheduling algorithms to process the first image. Thus, the first main algorithm library can obtain the N algorithms to be scheduled from the camera scheduling module. Furthermore, the first main algorithm library can also obtain the instruction information and data information of each of the N algorithms from the camera scheduling module. The instruction information of an algorithm indicates the specific action that the algorithm should perform, such as beautifying the skin of a person in an image; the data information of an algorithm can include the data used by the algorithm when performing the action, such as the storage address of the image to be processed and the beautification parameters.

[0092] For example, Figure 9 This diagram illustrates how various algorithms process image frames in a video stream according to their respective frame rates. For example... Figure 9As shown, the frame rate of the electronic device system is T. The frame rate corresponding to Algorithm 0 is T / 2, the frame rate corresponding to Algorithm 1 is T / 3, and the frame rate corresponding to Algorithm 2 is T / 4. According to the acquisition order of image frames in the video stream, the electronic device assigns a frame number to each image frame. This frame number is equivalent to a timestamp, used to identify the acquisition order of the image frames. For example, the electronic device acquires image frames M1, M2, M3, M4, M5, M6, M7, M8, and M9 in the order from front to back. Algorithm 0 processes image frames M1, M3, M5, M7, and M9 at a frame rate of T / 2; that is, image frames M1, M3, M5, M7, and M9 are scheduled using Algorithm 0. Algorithm 1 processes image frames M1, M4, and M7 at a frame rate of T / 3; that is, image frames M1, M4, and M7 are scheduled using Algorithm 1. Algorithm 2 processes image frames M1, M5, and M9 at a frame rate of T / 4; that is, Algorithm 2 is used to schedule image frames M1, M5, and M9. Thus, for different images in the preview video stream, the camera scheduling module determines different algorithms, and the CPU schedules different algorithms to execute different tasks. For example, the algorithms used to process image frame M1 include Algorithm 0, Algorithm 1, and Algorithm 2; the algorithm used to process image frame M3 includes Algorithm 0; the algorithm used to process image frame M4 includes Algorithm 4; and the algorithm used to process image frame M5 includes Algorithm 0 and Algorithm 2.

[0093] S102, the CPU sends an instruction message to the DSP, which includes information about the memory block corresponding to each of the N algorithms. The memory blocks corresponding to the N algorithms belong to a shared memory.

[0094] Based on the description of the above embodiments, each algorithm's program includes two parts: one part located on the CPU side and the other part located on the DSP side. The CPU-side algorithm is responsible for sending instruction information and data information, while the DSP-side algorithm is responsible for executing the corresponding task based on the instruction information and data information. In the algorithm scheduling provided in this application, a first main algorithm library is set up on the CPU side, and a second main algorithm library is set up on the DSP side. The first main algorithm library is responsible for the unified scheduling of M preset algorithms on the CPU side, and the second main algorithm library is responsible for the unified scheduling of M preset algorithms on the DSP side.

[0095] Specifically, "the CPU sending an instruction to the DSP" can include sending an instruction from the CPU's first main algorithm library to the DSP's second main algorithm library. This instruction can include scheduling policies for M preset algorithms. The scheduling policy for each of these M preset algorithms can include: whether each preset algorithm is scheduled (i.e., whether each preset algorithm is executed), the name of each preset algorithm, the name length of each preset algorithm, the priority of each preset algorithm, the starting sequence number of the memory block for each preset algorithm, and the number of memory blocks used by each preset algorithm, etc.

[0096] Accordingly, the "information on the memory block corresponding to each algorithm" may include the starting sequence number of the memory block for each preset algorithm, as well as the number of memory blocks used by each preset algorithm.

[0097] For example, the following is a partial code of a scheduling strategy that carries M preset algorithms in the instruction information:

[0098]

[0099] In this context, if Bool isAlgoRun is True, it indicates that the algorithm has been scheduled; if Bool isAlgoRun is False, it indicates that the algorithm has not been scheduled. Since only N out of the M preset algorithms are scheduled during a single scheduling process, the CPU indicates to the DSP whether each preset algorithm has been scheduled.

[0100] Algorithm priority represents the order in which algorithms are executed. A higher priority algorithm is executed earlier in the execution order, and a lower priority algorithm is executed later in the execution order.

[0101] One or more shared memory blocks can be used. By indicating the starting sequence number of the shared memory block and the number of shared memory blocks to be used, the DSP can read the data information of the algorithm from the memory block corresponding to the algorithm.

[0102] The SingleAlgoInfo algoInfos mentioned above are pointers to the Struct SingleAlgoInfo, which allows the scheduling policies of individual algorithms to be passed over, thereby summarizing the scheduling policies of all algorithms.

[0103] The number of algorithms is represented by M, which is the total number of preset algorithms.

[0104] It is understandable that the scheduling strategies of the aforementioned M preset algorithms are all carried in a single instruction message, and the size of the data carried in the instruction message does not affect the transmission speed of the instruction message. Taking the single scheduling overhead from the CPU to the DSP as approximately 0.5 milliseconds as an example, by sending an instruction message to the DSP, the CPU ensures that the scheduling strategies of all preset algorithms are delivered to the DSP in a single transmission, reducing the original N instruction message transmissions to a single instruction message transmission, thereby shortening the scheduling overhead from the original 0.5*N milliseconds to 0.5 seconds.

[0105] S103, the CPU writes the data information of each algorithm into the memory block corresponding to each of the N algorithms.

[0106] The memory blocks corresponding to the above N algorithms belong to a shared memory.

[0107] In some embodiments, before writing the data information of each algorithm into the memory block corresponding to each of the N algorithms, the method may further include: creating a shared memory. This shared memory is used to store the data information involved in the camera application's invocation of each algorithm, and the number of memory blocks included in the shared memory is determined based on the M preset algorithms corresponding to the camera application. For the implementation of the shared memory creation by the electronic device, please refer to the specific description of the above embodiments; it will not be repeated here.

[0108] In some embodiments, when N algorithms need to be scheduled, this can be controlled through a cache management interface. For example, the data information of the N algorithms can be written to the corresponding memory blocks in shared memory. For instance, the CPU writes the data information Task0 of algorithm 0 to the memory block of algorithm 0, the data information Task1 of algorithm 1 to the memory block of algorithm 1, the data information Task2 of algorithm 2 to the memory block of algorithm 2, and so on, and writes the data information Task2 of algorithm N to the memory block of algorithm N.

[0109] Based on the description of the above embodiments, the steps for data exchange based on shared memory are as follows: First, the shared memory is created. The operating system first needs to allocate a region in physical memory as shared memory, which can be accessed by multiple processes. Second, memory mapping. Processes map the shared memory region to their own address space through the interface provided by the operating system. This allows processes to access the shared memory as if it were their own private memory. In this embodiment, since the instruction information in S102 carries information about the memory block corresponding to each of the N algorithms, memory mapping can be achieved. Third, data access and synchronization. When one process writes data to the shared memory, other processes sharing that memory region can immediately see the updated content. In this embodiment, since the CPU writes the data information of each algorithm to the memory block corresponding to each of the N algorithms in S103, the DSP can immediately obtain the updated data information whenever the CPU updates the data information of the shared memory (i.e., S104 below).

[0110] S104, the DSP reads data information from the memory block corresponding to each of the N algorithms.

[0111] For example, the DSP can read data information from the memory block corresponding to each of the N algorithms through the second main algorithm library.

[0112] S105, the DSP schedules the N algorithms to execute the tasks corresponding to the data information of the N algorithms.

[0113] Different algorithms correspond to different tasks based on their data information.

[0114] Based on the description of S102 in the above embodiments, the scheduling strategy for each of the M preset algorithms may include: whether each preset algorithm is scheduled, the name of each preset algorithm, and the priority of each preset algorithm. The DSP can use this information to select N algorithms to be scheduled from the M preset algorithms and determine the priority of these N algorithms. If the DSP schedules multiple algorithms to execute corresponding tasks simultaneously, it may lead to resource contention scheduling overhead. To solve this problem, this application also proposes that the DSP schedule each of the N algorithms sequentially according to their priority from high to low, executing the task corresponding to the data information of each algorithm.

[0115] In traditional scheduling schemes, the N algorithms on the DSP side are N independent algorithm modules. Self-developed algorithms cannot control the DSP's scheduling logic for these N algorithms. In some cases, less important algorithms may be prioritized, leading to later execution of more important algorithms. Alternatively, in other cases, the N algorithms may compete for scheduling resources. To address this issue, this application proposes setting up a second main algorithm library on the DSP side. The N algorithms are defined as N compilation units or N sub-modules within this second main algorithm library, which is responsible for the unified scheduling of these N algorithms.

[0116] In some embodiments, after the second main algorithm library receives an instruction from the first main algorithm library, the second main algorithm library can generate an algorithm scheduling table algoList based on whether each preset algorithm is scheduled or not in the scheduling strategy of M preset algorithms. The algorithm scheduling table algoList can be used to record the N algorithms to be scheduled. Then, the second main algorithm library can schedule each of the N modules to execute the task corresponding to the data information of each algorithm in the order of priority from high to low of the N algorithms in the algorithm scheduling table.

[0117] In some embodiments, the second main algorithm library can be uniformly scheduled according to the current priority of each algorithm.

[0118] As an example, the second main algorithm library schedules the first module out of N modules to execute the task corresponding to the data information of the first algorithm during the first time period, and schedules the second module out of N modules to execute the task corresponding to the data information of the second algorithm. The first and second algorithms have higher priorities than other unscheduled algorithms, and the DSP's thread resources support simultaneous scheduling of the first and second modules during the first time period. For example, the second main algorithm library can use multi-threading to simultaneously schedule Algorithm 1 and Algorithm 2.

[0119] As another example, the second main algorithm library schedules the third module out of N modules in the second time period to execute the task corresponding to the data information of the first algorithm. The first algorithm has a higher priority than other unscheduled algorithms. For instance, the second main algorithm library can first use all the DSP's thread resources to execute the operators required by algorithm 0, ensuring that algorithm 0 is scheduled with the highest priority.

[0120] For example, Figure 10 This is a schematic diagram illustrating the unified scheduling of five algorithms by the second main algorithm library provided in this embodiment of the application.

[0121] Assume that Algorithm 0 has a higher priority than Algorithm 1, Algorithm 1 has a higher priority than Algorithm 2, Algorithm 2 has a higher priority than Algorithm 3, and Algorithm 3 has a higher priority than Algorithm 4. For example... Figure 10 As shown, in stage 1, the second main algorithm library can first use all the thread resources of the DSP to execute the operators required by algorithm 0, ensuring that algorithm 0 is scheduled with the highest priority; in stage 2, the second main algorithm library can use multiple threads to schedule algorithm 1 and algorithm 2 simultaneously; in stage 3, the second main algorithm library can use thread resources to execute the operators required by algorithm 3; in stage 4, the second main algorithm library can use thread resources to execute the operators required by algorithm 4.

[0122] It is understandable that the original scheduling was handled by the DSP operating system itself, but it has been changed to the second main algorithm library being responsible for the unified scheduling of N algorithms. This allows for unified scheduling based on the current priority of each task and the DSP system resource status, thereby avoiding additional scheduling overhead caused by system resource contention.

[0123] In the algorithm scheduling method provided in this application embodiment, when N algorithms need to be scheduled, the data information of the N algorithms is written into a shared memory, and different memory blocks of the shared memory are used for the data information of different algorithms. The CPU sends instruction information carrying the memory block information corresponding to each of the N algorithms to the DSP, so that the DSP can read the data information of the N algorithms from the memory block and execute the task corresponding to the data information of the N algorithms. In this way, the original N transmissions (each transmission includes data information and instruction information corresponding to the task of an algorithm) are reduced to 1 instruction transmission and 1 data synchronization process. Since the data synchronization process has zero overhead, the scheduling overhead of this method is the scheduling overhead of transmitting 1 instruction transmission, which reduces the scheduling overhead from the original 0.5*N milliseconds to 0.5 seconds. It can be understood that the larger the value of N, the more obvious the effect of this scheme in reducing scheduling overhead.

[0124] For example, Figure 11 This is a schematic diagram of a heterogeneous system provided in an embodiment of this application. For example... Figure 11 As shown, the heterogeneous system 00 includes a CPU 01 and a DSP 02. The CPU 01 and DSP 02 can implement the methods in the above embodiments.

[0125] In some embodiments, CPU 01 is used to determine N algorithms to be scheduled. CPU 01 is also used to write data information for each of the N algorithms into a memory block corresponding to each algorithm, where the memory blocks corresponding to the N algorithms belong to a shared memory. CPU 01 is also used to send an instruction message to the DSP, the instruction message including the memory block corresponding to each of the N algorithms. DSP 02 is used to schedule the N algorithms to execute the tasks corresponding to the data information of the N algorithms.

[0126] In other embodiments, DSP 02 can also be used to read data information from the memory block corresponding to each algorithm.

[0127] In other embodiments, DSP 02 can be specifically used to schedule each of the N algorithms to execute the task corresponding to the data information of each algorithm in descending order of priority.

[0128] In other embodiments, the CPU has a first main algorithm library, and the DSP has a second main algorithm library. The first and second main algorithm libraries are used for unified scheduling of M preset algorithms. The M preset algorithms include N algorithms, where M is an integer greater than 1. Specifically, CPU 01 can send an instruction message to the DSP's second main algorithm library through the CPU's first main algorithm library. The instruction message includes the scheduling strategy for the M preset algorithms. The scheduling strategy for each of the M preset algorithms includes: whether each preset algorithm is scheduled, the name of each preset algorithm, the name length of each preset algorithm, the priority of each preset algorithm, the starting sequence number of the memory block for each preset algorithm, and the number of memory blocks used by each preset algorithm.

[0129] In other embodiments, DSP 02 can be specifically used to generate an algorithm scheduling table based on the scheduling strategy of M preset algorithms, using the second main algorithm library of the DSP, and whether each preset algorithm is scheduled or not. The algorithm scheduling table includes N algorithms to be scheduled. Then, according to the priority of the N algorithms in the algorithm scheduling table from high to low, each of the N modules is scheduled to execute the task corresponding to the data information of each algorithm.

[0130] In other embodiments, DSP 02 can specifically be used to schedule a first module out of N modules to execute a task corresponding to the data information of a first algorithm, and to schedule a second module out of N modules to execute a task corresponding to the data information of a second algorithm, through the DSP's second main algorithm library. The first and second algorithms have higher priorities than other unscheduled algorithms, and the DSP's thread resources support simultaneous scheduling of the first and second modules during the first time period.

[0131] In other embodiments, the CPU is equipped with a camera scheduling module and a first main algorithm library. Specifically, the CPU 01 can be used to determine N algorithms from M preset algorithms through the camera scheduling module as the scheduling algorithms to process the first image, where the first image is any frame in the video stream captured by the electronic device; and to obtain the N algorithms to be scheduled from the camera scheduling module through the first main algorithm library.

[0132] This application also provides an electronic device, including a processor coupled to a memory, the processor being used to execute computer programs or instructions stored in the memory, so that the electronic device implements the methods in the above embodiments.

[0133] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the method described above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or can include one or more data storage devices such as servers or data centers that can be integrated with the medium. Available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0134] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0135] This application also provides a chip coupled to a memory. This chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0136] The electronic device, computer-readable storage medium, computer program product, and chip provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An algorithm scheduling method, characterized in that, The method is applied to an electronic device, the electronic device including a central processing unit (CPU) and a digital signal processor (DSP), the method comprising: The CPU determines N algorithms to be scheduled, where N is a positive integer; The CPU sends an instruction message to the DSP, the instruction message including information about the memory block corresponding to each of the N algorithms, and the memory blocks corresponding to the N algorithms belong to a shared memory. The CPU writes the data information of each algorithm into the memory block corresponding to each of the N algorithms; The DSP schedules the N algorithms to execute tasks corresponding to the data information of the N algorithms.

2. The method according to claim 1, characterized in that, Before scheduling the N algorithms to execute tasks corresponding to the data information of the N algorithms via the DSP, the method further includes: The DSP reads the data information from the memory block corresponding to each of the N algorithms.

3. The method according to claim 1, characterized in that, The instruction information also includes the priority corresponding to each of the N algorithms; the step of scheduling the N algorithms to execute tasks corresponding to the data information of the N algorithms via the DSP includes: The DSP schedules each of the N algorithms in descending order of priority to execute the task corresponding to the data information of each algorithm.

4. The method according to any one of claims 1 to 3, characterized in that, The CPU is equipped with a first main algorithm library, and the DSP is equipped with a second main algorithm library. The first main algorithm library and the second main algorithm library are used to uniformly schedule M preset algorithms. The M preset algorithms include the N algorithms, where M is an integer greater than 1. The step of sending an instruction message from the CPU to the DSP includes: The CPU sends an instruction message to the DSP's second main algorithm library through the first main algorithm library. The instruction message includes the scheduling policies of the M preset algorithms. The scheduling policy of each preset algorithm includes: whether the preset algorithm is scheduled, the name of the preset algorithm, the name length of the preset algorithm, the priority of the preset algorithm, the starting sequence number of the memory block of the preset algorithm, and the number of memory blocks used by the preset algorithm.

5. The method according to claim 4, characterized in that, The N algorithms are N modules in the second main algorithm library; the step of scheduling the N algorithms to execute tasks corresponding to the data information of the N algorithms through the DSP includes: Using the second main algorithm library of the DSP, based on the scheduling strategy of each preset algorithm, and in accordance with the priority of the N algorithms from high to low, each of the N modules is scheduled to execute the task corresponding to the data information of each algorithm.

6. The method according to claim 5, characterized in that, The step of using the second main algorithm library of the DSP, based on the scheduling strategy of each preset algorithm, and in descending order of priority of the N algorithms, sequentially scheduling each of the N modules to execute the task corresponding to the data information of each algorithm, includes: Using the second main algorithm library of the DSP, an algorithm scheduling table is generated based on whether each preset algorithm is scheduled or not in the scheduling strategy of the M preset algorithms. The algorithm scheduling table includes the N algorithms to be scheduled. Then, according to the priority of the N algorithms in the algorithm scheduling table from high to low, each of the N modules is scheduled to execute the task corresponding to the data information of each algorithm.

7. The method according to claim 5, characterized in that, The step of using the second main algorithm library of the DSP, based on the scheduling strategy of each preset algorithm, and in descending order of priority of the N algorithms, sequentially scheduling each of the N modules to execute the task corresponding to the data information of each algorithm, includes: Through the second main algorithm library of the DSP, the first module among the N modules is scheduled to execute the task corresponding to the data information of the first algorithm in the first time period, and the second module among the N modules is scheduled to execute the task corresponding to the data information of the second algorithm. The first and second algorithms have higher priorities than other unscheduled algorithms, and the DSP's thread resources support simultaneous scheduling of the first and second modules during the first time period.

8. The method according to any one of claims 1 to 7, characterized in that, The CPU is equipped with a camera scheduling module and a first main algorithm library; the process of determining N algorithms to be scheduled through the CPU includes: The camera scheduling module determines N algorithms from M preset algorithms as the scheduling algorithms to process the first image, which is any frame in the video stream captured by the electronic device. The N algorithms to be scheduled are obtained from the camera scheduling module through the first main algorithm library; Different image scheduling algorithms perform different tasks.

9. The method according to any one of claims 1 to 8, characterized in that, Before determining the N algorithms to be scheduled via the CPU, the method further includes: In response to the user's first action, the camera application is run and the shared memory is created, the number of memory blocks included in the shared memory being determined according to the M preset algorithms.

10. The method according to claim 9, characterized in that, The method further includes: In response to the user's second action, the camera application is closed and the shared memory is reclaimed.

11. A heterogeneous system, characterized in that, The heterogeneous system includes a CPU and a DSP; The CPU is used to determine the N algorithms to be scheduled; The CPU is also used to write the data information of each algorithm into the memory block corresponding to each of the N algorithms, and the memory blocks corresponding to the N algorithms belong to a shared memory. The CPU is also used to send an instruction message to the DSP, the instruction message including a memory block corresponding to each of the N algorithms; The DSP is used to schedule the N algorithms to execute tasks corresponding to the data information of the N algorithms.

12. An electronic device, characterized in that, The electronic device includes a CPU, a DSP, and a memory; The memory is coupled to the CPU and the DSP, and the memory is used to store computer program code, the computer program code including computer instructions, and the CPU and the DSP call the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

13. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including a CPU and a DSP, the CPU and DSP being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 10.

Images