A method, device and storage medium for multi-level program loading

Abstract

The present application provides a multi-level program loading method, device and storage medium, which relates to the field of computer technology and is used to solve the problem that the neural network model program is too large and the neural network model cannot complete the computing task. The solution includes: controlling the PDMA module to load the first-level loading content from the storage device to the program memory, the first-level loading content includes the second-level dynamic loading control code and the program of the first function group among the programs of N function groups, and the programs of the N function groups are obtained by segmenting the program of the target network model; after the program of the first function group is run, according to the second-level dynamic loading control code, controlling the PDMA module to execute N-1 second-level dynamic loading processes until the program of the Nth function group is loaded into the program memory, and after the program of the function group loaded into the program memory is run, the next second-level dynamic loading process is executed, and the programs of the N function groups have the same starting storage address in the program memory.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and storage medium for multi-level program loading. Background Art

[0002] A convolutional neural network (CNN) model consists of basic layers, each corresponding to an operation. These operations include convolution, pooling, element-wise operations, concatenation, fully-connected operations, and bath-normalization. A neural network processor (NNP) is a processor specifically designed to execute the computational tasks of CNN models. A simple NNP instruction fetch implementation uses the program DMA (PDMA) module to move program instruction encoding from DDR to program memory (PM). The instruction fetch (Ifetch) module then reads the instructions from the PM and distributes them to the various pipelines. In this scenario, instructions must already exist in the PM before they can be fetched. Therefore, if the model program exceeds the available memory space in the PM, it cannot be fully stored in the PM. If the model program cannot be fully stored in the PM, the NNP model program cannot be fully executed, and the NNP cannot complete the computational task. Summary of the Invention

[0003] The present application provides a program loading method, device and storage medium to solve the problem that the neural network model program is too large and the NNP cannot complete the computing task.

[0004] In a first aspect, an embodiment of the present application provides a multi-level program loading method, which is applied to a neural network processor, wherein the neural network processor includes a direct memory access module, and the method includes: controlling the direct memory access module to load a first-level loading content from a storage device into a program memory, wherein the first-level loading content includes a second-level dynamic loading control code and a program of a first function group among N function groups, wherein the programs of the N function groups are obtained by segmenting the programs of a target network model; and the storage space occupied by the program of any of the function groups is less than or equal to the remaining storage space of the program memory, where N is an integer greater than or equal to 2; after the program of the first function group is run, controlling the direct memory access module to execute N-1 second-level dynamic loading processes according to the second-level dynamic loading control code until the direct memory access module loads the program of the Nth function group into the program memory, wherein the next second-level dynamic loading process is executed after the program of the function group loaded into the program memory is run, and the programs of the N function groups have the same starting storage address of the program memory.

[0005] An embodiment of the present application provides a multi-level program loading method. In this method, a storage device stores N function group programs obtained by segmenting the target network model program. In this scheme, a neural network processor first controls a direct memory access module to load the first-level loading content from the storage device into the program memory through a first-level dynamic loading process. The first-level loading content includes a second-level dynamic loading control code and the program of the first function group. After the neural network processor runs the program of the first function group, it controls the direct memory access module to perform at least one second-level dynamic loading process according to the second-level dynamic loading control code. During the second-level dynamic loading process, after each function group program loaded into the program memory is executed, the next function group program is loaded from the storage device until the last function group program is loaded into the program memory. Each loaded function group program has the same starting storage address in the program memory as the previously loaded and executed function group program. The storage space occupied by each function group program is less than or equal to the remaining storage space in the program memory. Therefore, this scheme can solve the problem that when the neural network model program is too large, the model program cannot be fully executed by the NNP, and the NNP cannot complete the computing task.

[0006] In a possible implementation of the present application, according to the second-level dynamic loading control code, controlling the direct memory access module to perform at least N-1 second-level dynamic loading processes until the direct memory access module loads the program of the Nth function group into the program memory, including: determining, according to the second-level dynamic loading control code, second-level loading configuration information corresponding to the i-th second-level dynamic loading process, the second-level loading configuration information including: first information, and a first starting storage address of the i-th second-level loaded content in the program memory; i=1, 2, ..., N-1; the first information is used to determine a storage location of the i-th second-level loaded content in the storage device;

[0007] controlling the direct memory access module to use the program of the (i+1)th function group stored in the corresponding position of the storage device as the i-th secondary loading content according to the secondary loading configuration information, and load the program into the program memory according to the first starting storage address;

[0008] Determine whether to execute the (i+1)th secondary dynamic loading process;

[0009] When it is determined that the i+1th secondary dynamic loading process is to be executed, after executing the program of the i+1th function group, the process of controlling the direct memory access module to load the program of the i+2th function group from the storage device to the program memory is continued, and so on, until it is determined that the i+1th secondary dynamic loading process is not to be executed, then the secondary dynamic loading process is ended.

[0010] In one possible implementation of the present application, the secondary dynamic loading control code includes: a secondary dynamic loading count counter, and a target number of secondary dynamic loading times; the method provided in the embodiment of the present application further includes:

[0011] Each time the secondary dynamic loading process is executed, the value of the secondary dynamic loading times counter is increased by a preset value.

[0012] Correspondingly, after the i-th secondary dynamic loading process is executed, if the updated value of the secondary dynamic loading times counter is less than the target times of the secondary dynamic loading, it is determined to execute the i+1-th secondary dynamic loading process, otherwise, the secondary dynamic loading process is terminated.

[0013] In a possible implementation of the present application, the first information includes at least: a starting storage address of the i-th secondary loading content in the storage device;

[0014] The secondary dynamic loading control code includes: an address offset of the first secondary loading content compared to the starting storage address of the primary loading content in the storage device;

[0015] Accordingly, determining the first information corresponding to the i-th secondary dynamic loading process according to the secondary dynamic loading control code includes:

[0016] When i=1, determining the starting storage address of the i-th secondary loading content in the storage device according to the address offset and the starting storage address of the first-level loading content in the storage device;

[0017] When i≥2, the starting storage address of the i-th secondary loading content in the storage device is determined according to the starting storage address of the i-1th secondary loading content in the storage device and the target loading row number corresponding to the i-1th secondary loading.

[0018] In a possible implementation of the present application, the first information further includes the number of target program lines corresponding to the i-th secondary dynamic loading process. Accordingly, the method further includes:

[0019] The number of loaded lines corresponding to the value of the second-level dynamic loading counter in the second-level dynamic loading control code after the (i-1)th second-level dynamic loading is used as the number of target program lines corresponding to the (i)th second-level loading.

[0020] In a possible implementation of the present application, controlling a direct memory access module to load first-level load content from a storage device to a program memory includes: configuring first-level load configuration information for the direct memory access module so that the direct memory access module loads the first-level load content from the storage device to the program memory according to the first-level load configuration information, wherein the first-level load configuration information includes: the number of program lines dynamically loaded at the first level of the model program, the starting storage address of the first-level load content in the storage device, and the starting storage address of the first-level load content in the program memory.

[0021] In a possible implementation of the present application, a driver is also stored in the program memory, and the storage addresses of the driver, the secondary dynamic loading control code, and the program of any of the function groups in the program memory are continuous.

[0022] In a possible implementation of the present application, the storage addresses of the secondary dynamic loading control code and the programs of the plurality of function groups in the storage device are continuous.

[0023] In a second aspect, an embodiment of the present application provides a neural network processor, comprising a direct memory access module, a computing module, and a program memory, wherein:

[0024] The operation module is configured to: control the direct memory access module to load a first-level loading content from a storage device into the program memory, the first-level loading content including a second-level dynamic loading control code and a program of a first function group among N function groups, the programs of the N function groups being obtained by segmenting the programs of the target network model; and the storage space occupied by the programs of any one of the function groups is less than or equal to the remaining storage space of the program memory, where N is an integer greater than or equal to 2;

[0025] After the program of the first function group is run, the operation module is further configured to: execute the secondary dynamic loading control code, and control the direct memory access module to perform N-1 secondary dynamic loading processes until the program of the Nth function group is loaded into the program memory, wherein the next secondary dynamic loading process is executed after the program of the function group loaded into the program memory is run, and the programs of the N function groups have the same starting storage address in the program memory.

[0026] In a possible implementation of the present application, the operation module is specifically configured to execute a driver program stored in the program memory to control the direct memory access module to load the first-level loading content from the storage device to the program memory.

[0027] In one possible implementation of the present application, the neural network processor provided in the embodiment of the present application is also used to load a driver from a storage device into a program memory.

[0028] In a possible implementation of the present application, the computing unit is specifically configured to: determine, based on the second-level dynamic loading control code, second-level loading configuration information corresponding to an i-th second-level dynamic loading process, the second-level loading configuration information including: first information, and a first starting storage address of the i-th second-level loaded content in the program memory; i=1, 2, ..., N-1; the first information is used to determine a storage location of the i-th second-level loaded content in the storage device; and is further configured to: control the direct memory access module to use, based on the second-level loading configuration information, a program of the i+1th function group stored at a corresponding location in the storage device as the i-th second-level loaded content, and load the program into the program memory according to the first starting storage address;

[0029] The computing unit is also used to determine whether to execute the (i+1) second-level dynamic loading process;

[0030] The operation unit is also used to, when it is determined that the i+1th secondary dynamic loading process is to be executed, continue to control the direct memory access module to load the program of the i+2th function group from the storage device to the program memory after executing the program of the i+1th function group, and so on, until it is determined that the i+1th secondary dynamic loading process is not to be executed, and then end the secondary dynamic loading process.

[0031] In a possible implementation of the present application, the secondary dynamic loading control code includes: a secondary dynamic loading times counter, and a target number of secondary dynamic loadings; an operation unit is also used to increase the value of the secondary dynamic loading times counter by a preset value each time the secondary dynamic loading process is executed.

[0032] Correspondingly, after the i-th secondary dynamic loading process is executed, if the updated value of the secondary dynamic loading counter is less than the target number of secondary dynamic loadings, the operation unit is also used to determine whether to execute the i+1-th secondary dynamic loading process; otherwise, the operation unit is also used to end the secondary dynamic loading process.

[0033] In a possible implementation of the present application, the first information includes at least: a starting storage address of the i-th secondary loading content in the storage device;

[0034] The secondary dynamic loading control code includes: an address offset of the first secondary loading content compared to the starting storage address of the primary loading content in the storage device;

[0035] Accordingly, the computing unit is further configured to: when i=1, determine the starting storage address of the i-th secondary loading content in the storage device according to the address offset and the starting storage address of the primary loading content in the storage device;

[0036] When i≥2, the starting storage address of the i-th secondary loading content in the storage device is determined according to the starting storage address of the i-1th secondary loading content in the storage device and the target loading row number corresponding to the i-1th secondary loading.

[0037] In a possible implementation of the present application, the first information also includes the number of target program lines corresponding to the i-th secondary dynamic loading process. Accordingly, the operation unit is also used to use the number of loading lines corresponding to the value of the secondary dynamic loading counter in the secondary dynamic loading control code after the i-1-th secondary dynamic loading as the number of target program lines corresponding to the i-th secondary loading.

[0038] In a possible implementation of the present application, the computing unit is further used to configure first-level loading configuration information for the direct memory access module, so that the direct memory access module loads the first-level loading content from the storage device to the program memory according to the first-level loading configuration information, and the first-level loading configuration information includes: the number of program lines dynamically loaded at the first level of the model program, the starting storage address of the first-level loading content in the storage device, and the starting storage address of the first-level loading content in the program memory.

[0039] In a possible implementation of the present application, a driver is also stored in the program memory, and the storage addresses of the driver, the secondary dynamic loading control code, and the program of any of the function groups in the program memory are continuous.

[0040] In a possible implementation of the present application, the storage addresses of the secondary dynamic loading control code and the programs of the plurality of function groups in the storage device are continuous.

[0041] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory and a processor, wherein the memory is used to store a computer program, and when the processor executes the computer program, it implements the method described in the first aspect or various possible implementations of the first aspect.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in the first aspect or various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a first-level dynamic loading process provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the structure of a program loading system provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the structure of a neural network processor provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a program compilation method provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of a multi-level program loading method provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a multi-level program loading method provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a specific flow chart of a multi-level program loading method provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first function combination and the second function combination are merely used to distinguish different function combinations and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0052] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0054] like Figure 1 As shown, Figure 1 This is the process of loading a model program from a double data rate synchronous dynamic random access memory (DDR) to a neural network processor (NNP) in the prior art, such as Figure 1As shown: the driver and model program are stored in DDR. During the initialization loading process, NNP loads the driver from DDR into the program memory (PM) of NNP. After the initialization loading, NNP starts and executes the driver, and monitors in real time whether there are model tasks to be done. When a model task needs to be done, NNP configures PDMA in the driver to move the model program to PM, and the storage location of the model program in PM is behind the driver. The driver configuration of PDMA to move the model program is called first-level dynamic loading. After the first-level dynamic loading is completed, the driver jumps to the model program location through a subroutine call to start executing the model program. After the model program is executed, it returns to the driver to continue model task monitoring. The first-level dynamic loading of the model program is as follows Figure 1 However, first-level dynamic loading only works for a subset of models, meaning the model program size does not exceed the PM total space size minus the driver space size. If the model program is larger than the PM total space size minus the driver space size, using only first-level dynamic loading may not fully load the model program into the PM, preventing the NNP from completing its computational tasks.

[0055] Based on this, in order to solve the problem when the model program is larger than the remaining space of PM, this application solves it through secondary dynamic loading.

[0056] like Figure 2 As shown, Figure 2 A schematic diagram of a program loading system provided in an embodiment of the present application, wherein the program loading system includes: a storage device 100 and a neural network processor 200.

[0057] The storage device 100 is used to store multiple function group programs, drivers, and secondary dynamic loading control codes obtained after the network model program is cut. Each function group (func gp) program includes at least one function.

[0058] As an example, the programs of the multiple function groups are sequentially stored in the storage device 100 according to the order of the programs of the function groups in the program of the network model.

[0059] The network model involved in the embodiments of the present application may be a neural network model. As an example, the neural network model may be a convolutional neural network model. For example, the convolutional neural network model is a neural network model for face recognition.

[0060] As an example, the neural network processor (NNP) 200 is a processor specifically used to perform convolutional neural network model computing tasks.

[0061] As an example, Figure 3 As shown, the neural network processor 200 provided in the embodiment of the present application may include: a program direct memory access (PDMA) module 201, a program memory (PM) 202, an instruction read (Ifetch) module 203 and an arithmetic unit (ALU) 204. 。 The PDMA module 201 is connected to the PM 202 and the arithmetic unit (ALU), the PM 202 is connected to the instruction fetch (Ifetch) module 203 , and the instruction fetch (Ifetch) module 203 is connected to the arithmetic unit 204 .

[0062] Among them, the PDMA module 201 is responsible for moving programs (such as drivers or secondary dynamic loading control codes or programs of various function groups) from the storage device 100 to the PM. The read (Ifetch) module 203 is responsible for reading programs from PM202 and distributing instructions. The program of the neural network model needs to be stored in PM202 in advance before it can be read. The PDMA module 201 can receive the configuration of the NNP external interface, and can also receive the configuration of the NNP internal instructions. The embodiment of the present application does not limit this. The function of the arithmetic unit (ALU) 204 is to execute NNP scalar instructions, which include instructions for accessing software and hardware interface registers (obtaining software configuration information), instructions for configuring the PDMA module 201, and arithmetic and logical operation instructions.

[0063] Optionally, the program loading system may further include a compilation device 300. Compilation device 300 is connected to storage device 100, which is in turn connected to program loading device 200. Compilation device 300 is configured to divide the network model program into multiple function group programs. Compilation device 300 is then configured to store the resulting multiple function group programs in storage device 100.

[0064] It is understood that when the compiler device 300 segments the network model program, the storage space occupied by each function group is less than or equal to the remaining storage space of the PM. In other words, when the compiler device 300 segments the network model program, it needs to consider the remaining storage space of the PM.

[0065] Because the basic structure of a convolutional neural network typically consists of multiple consecutive layers, to reduce the DDR bandwidth requirements of the neural network processor, the compiler device 300 attempts to combine multiple consecutive layers into a layer-group or function (func) when segmenting the convolutional neural network. This process is called fusion. During the code generation phase, the NNP code for each func is generated. The top-level code of the model sequentially calls the subroutines of these funcs to generate the model code executed by the NNP.

[0066] As an example, the program of each function group is stored in the storage device 100 in the form of binary machine code. For example, the program of each function group is stored in the storage device 100 in the form of binary machine code.

[0067] It is worth noting that, in addition to storing the programs of the multiple function groups, the storage device 100 also needs to store the secondary dynamic loading control code.

[0068] Before introducing the embodiments of the present application, the process of how to generate the secondary dynamic loading control code in the embodiments of the present application is first introduced.

[0069] First, the compiling device 300 determines first information and second information, wherein the first information includes the total number of PM lines, the number of driver lines, and an initial value of the maximum number of secondary dynamic loading times.

[0070] The unit of row is the PM data bus width; each PM read or write is considered a PM row. The maximum number of second-level dynamic loads supported is an unknown variable. An initial value, such as 32, can be set as needed and may be modified during iterations. The maximum number of rows for first-level dynamic loads of the model program can be calculated from the total number of PM rows and the number of driver rows.

[0071] As an example, the second information includes the assembly file for each func and the assembler tool. During the model's second-level dynamic loading code generation process, all func assembly codes are combined at the func granularity to form the assembly code for multiple func gp files. The assembler generates binary machine code from the assembly code, thereby determining the program's footprint.

[0072] After the above conditions are met, the compiling device 300 can generate the secondary dynamic loading control code and the program of each func gp through the following steps 401 to 407.

[0073] Step 401: The compiling device 300 generates a secondary dynamic loading control code, including a macro definition file (generally referred to as a header file or header file, .h file) and an assembly file (.asm file).

[0074] Macro definition files are used because generating the secondary dynamic load control assembly file requires some information, such as the initial number of secondary dynamic loads actually required. However, this information is initially unknown, so it is represented as macros (arbitrary values ​​can be assigned to macros) and referenced in the assembly code. Using macro definitions stabilizes the number of instructions in the assembly code. This stable number of instructions makes it possible to calculate the number of lines in the secondary dynamic load control program, and thus determine the upper limit for the number of program lines for each func gp. The only unstable factor in the number of instructions in the secondary dynamic load control assembly code is the maximum number of secondary dynamic loads that can be supported. If the maximum number of secondary dynamic loads that can be supported increases, the number of instructions in the secondary dynamic load control code will also increase.

[0075] Step 402 : The compiling device 300 determines the number of program lines of the second-level dynamic loading control code according to the second-level dynamic loading control code generated in step 401 .

[0076] Step 403: The compiling device 300 determines the upper limit of the number of lines of each func gp program based on the total number of PM lines, the number of lines of the driver program, and the number of lines of the secondary dynamic loading control code.

[0077] As an example, the upper limit of the number of program lines for each func gp is less than or equal to a first threshold. The first threshold is calculated by subtracting the total number of PM lines from the number of lines in the driver program and subtracting the number of lines in the secondary dynamic loading control code. For example, if the total number of PM lines is M1, the number of lines in the driver program is M2, and the number of lines in the secondary dynamic loading control code is M3, then the upper limit of the number of program lines for the func gp program is less than or equal to M1 - M2 - M3.

[0078] It is worth noting that by limiting the upper limit of the number of program lines of each func gp program to be less than or equal to the first threshold, each func gp program can be completely stored in the PM after being moved from the storage device 100 to the PM.

[0079] In step 404, the compilation device 300 traverses the assembly files of all funcs of the target network model according to the upper limit of the number of program lines, groups them, and obtains the assembly file of each func gp and its program line number information, as well as the target secondary dynamic load count. The program line number of each func gp is less than or equal to the upper limit of the program line number.

[0080] It can be understood that the program line number information of an assembly file of a func gp is used to indicate the program line number after the assembly file of the func gp is compiled by the assembler.

[0081] For example, the upper limit of the number of program lines is a first threshold, such as 1000, and the number of program lines of each func gp is less than or equal to 1000.

[0082] The target number of secondary dynamic loading times is used to indicate the number of times the program loading device 200 needs to perform secondary loading from the storage device 100 .

[0083] Step 405: If the target number of secondary dynamic loading exceeds the maximum number of secondary dynamic loading that can be supported, the compilation device 300 will update the maximum number of secondary dynamic loading that can be supported to the target number of secondary dynamic loading and jump to step 401 for the next round of iteration; otherwise, it goes to step 406.

[0084] Step 406: Now that all unknown information has been determined, the compiler device 300 updates the macro definition file for the secondary dynamic loading control code. The updated macro definition file includes the target number of secondary dynamic loads required, the starting line (starting storage address) of the secondary dynamic load content in the PM, the starting PC corresponding to the secondary dynamic load (the jump instruction uses this PC to jump to the location of the secondary dynamically loaded program in the PM), the address offset of the first secondary dynamic load content compared to the primary load content in the storage device 100, and the number of program lines for each secondary dynamic load.

[0085] Step 407: The compiling device 300 generates the final binary machine code of the second-level dynamic loading control code and the binary machine code of each second-level dynamic loading func gp code.

[0086] Finally, the binary machine code of the second-level dynamic loading control code and the binary machine code of each second-level dynamic loading func gp code are stored in the storage device 100 .

[0087] The storage addresses of the machine code of the second-level dynamic loading control code and the machine code of each second-level dynamic loading func gp code in the storage device 100 are continuous, and all the second-level dynamic loading func gp codes are stored in the storage device 100 in sequence.

[0088] like Figure 5 FIG. 1 is a flow chart of a multi-level program loading method provided in an embodiment of the present application, the method comprising:

[0089] Step 501: The neural network processor controls the direct memory access module to load the first-level loading content from the storage device to the program memory.

[0090] The first-level loading content includes the second-level dynamic loading control code and the program of the first function group among the N function groups. The programs of the N function groups are obtained by segmenting the programs of the target network model. The storage space occupied by the programs of any function group is less than or equal to the remaining storage space of the program memory.

[0091] It is worth noting that the storage device is used to store at least the N function group programs obtained by segmenting the target network model program, and the second-level dynamic loading control code. The second-level dynamic loading control code and the program of the first function group of the N function group programs constitute the first-level loading content.

[0092] It is understood that the remaining storage space of the program memory is determined by the total storage space of the program memory and the storage space already occupied by the program memory. The storage space already occupied by the program memory refers to the storage space jointly occupied by the driver program and the secondary dynamic loading control code stored in the program memory.

[0093] As an example, the target network model may be a neural network model.

[0094] Among them, the secondary dynamic loading control code is used to manage the transportation and execution of the program of each function group.

[0095] As an example, the secondary dynamic loading control code includes: a first file and a second file, wherein the first file can be regarded as a macro definition file, the macro definition file includes the target number of secondary dynamic loading, the PM starting line of the secondary dynamic loading and the corresponding secondary dynamic loading starting PC, the address offset of the first secondary loading content compared to the first level loading content in the storage device, and the number of program lines loaded each time the secondary level is loaded.

[0096] It is understood that the address offset of the first secondary loading content in the storage device relative to the first primary loading content is used to determine the starting storage address of the first secondary loading content in the storage device. It is understood that in the embodiment of the present application, the program loading device loads the secondary loading content once, i.e., a program of a function group, from the storage device each time the secondary dynamic loading process is executed. Different secondary dynamic loading processes load different secondary loading contents.

[0097] The second file is an assembly file (.asm file). Knowing the number of program lines for each secondary dynamic load can determine the secondary load content to be loaded in combination with the starting storage address of each secondary load content in the storage device.

[0098] In the embodiment of the present application, the program of the target network model is divided into N function groups and stored in a storage device. The number of program lines in each function group has an upper limit, that is, it is less than or equal to a preset line number threshold.

[0099] For example, taking the target network model as a neural network model, the program of the neural network model is cut into programs of N function groups, and the programs of these N function groups are stored consecutively at the storage addresses in the storage device.

[0100] As an example, Figure 6 As shown, assuming that the target network model program is divided into five function groups (func gp0 program to func gp4 program), the program of the first function group is as follows Figure 6 The func gp0 program is shown. It can be understood that the storage location of the first function group program in the DDR is adjacent to the storage location of the secondary dynamic loading control code. In order to ensure that the NNP can smoothly read the programs of each function group from the storage device, the storage addresses of each function group in the DDR are continuous, for example, Figure 6 As shown, the storage addresses of the func gp0 program to the func gp4 program are continuous in DDR.

[0101] Optional, such as Figure 6 As shown, the driver is stored in the first storage space of the program memory.

[0102] Step 502: After the program of the first function group is loaded, the NNP controls the direct memory access module to execute N-1 secondary dynamic loading processes according to the secondary dynamic loading control code until the program of the Nth function group is loaded into the program memory, wherein the next secondary dynamic loading process is executed after the program of the function group loaded into the program memory is run, and the programs of the N function groups have the same starting storage address in the program memory.

[0103] As an example, the NNP may execute the program of the first function group in the following manner: the operation unit in the NNP executes the secondary dynamic loading control code to jump to the location of the program of the first function group, and executes the program of the first function group.

[0104] Exemplarily, the programs of all function groups have the same starting storage address in the program memory. For example, Figure 6 As shown, taking the example of all function group programs having the same starting storage address in the program memory, the starting storage addresses of the programs of func gp0 to func gp4 in the program memory are all address 1. Since the sizes of the programs of func gp0 to func gp4 may be different, the ending storage addresses of the programs of each func gp in the program memory may be different.

[0105] It is understandable that the programs of all function groups have the same starting storage address in the program memory. For example, if the program of the first function group is stored in storage space 2, and the starting storage address of storage space 2 is address 1, then the starting storage address of the program of the second function group in the program memory is also address 1. The ending storage address of the program of the second function group in the program memory is determined by the starting storage address of the program of the second function group in the program memory and the number of rows of the program of the second function group. Similarly, the starting storage address of the program of the last function group in the program memory is also address 1. This ensures that the program of a function group currently loaded into the program memory replaces the program of the function group last loaded into the program memory.

[0106] An embodiment of the present application provides a multi-level program loading method. In this method, since the storage device stores the programs of N function groups obtained after the program of the target network model is segmented, and in this scheme, the neural network processor first controls the direct memory access module to load the first-level loading content from the storage device into the program memory through a first-level dynamic loading process, the first-level loading content includes the program of the first function group and the second-level dynamic loading control code. After the neural network processor runs the program of the first function group, it controls the direct memory access module to perform at least one second-level dynamic loading process according to the second-level dynamic loading control code. Since each function group program loaded into the program memory is executed during the second-level dynamic loading process, the next function group program is loaded from the storage device until the last function group program is loaded into the program memory, and the program of each function group loaded each time has the same starting storage address in the program memory as the program of the previously loaded and executed function group. The storage space occupied by the program of each function group is less than or equal to the remaining storage space of the program memory. Therefore, this scheme can solve the problem that the neural network model program cannot be executed by the NNP when it is too large.

[0107] In a possible embodiment of the present application, the number of program lines of any function group is less than or equal to a first threshold, which is determined by the total number of lines of the program memory, the number of lines of the driver program, and the number of lines of the secondary dynamic loading control code.

[0108] For example, the first threshold is calculated by subtracting the number of lines in the driver program and the number of lines in the second-level dynamic load control code from the total number of lines in the program memory. For example, if the total number of lines in the program memory is 100, the number of lines in the driver program is 5, and the number of lines in the second-level dynamic load control code is 5, then the first threshold is 90. In other words, the number of program lines in any function group should be less than or equal to 90 and greater than or equal to 1.

[0109] It is understandable that the number of program lines of different function groups in the embodiments of the present application may be the same or different, but the number of program lines of each function group needs to be less than or equal to the first threshold value, so as to ensure that the storage space occupied by the program of the function group loaded into the program memory each time is less than or equal to the remaining storage space of the program memory. The number of function groups corresponding to the program of any target network model is obtained by the compilation device based on the assembly code of each func and the first threshold value search and assembly.

[0110] In a possible embodiment of the present application, the storage addresses of the programs of the multiple function groups stored in the storage device are continuous.

[0111] For example, Figure 6 As shown, taking a DDR as an example, the DDR stores the binary machine code of the driver (drv hex), the binary machine code of the second-level dynamic loading control code (l2ld), and the machine code of the N func gp programs. The second-level dynamic loading control code program machine code (l2ld) and the machine code of the N func gp programs are arranged consecutively in the DDR. For example, the storage address of the program file for func gp1 is located after the storage address of the program file for func gp0.

[0112] In a possible embodiment of the present application, the method provided in the embodiment of the present application may further include, before step 501: the PDMA in the neural network processor moves the driver program from the storage device to the program memory.

[0113] As a specific implementation, combined with Figure 3 PDMA can transfer drivers from storage devices to program memory in the following manner: Before the NNP boots up, it is necessary to configure PDMA to transfer drivers to the PM through the NNP external interface. Specifically, PDMA obtains driver configuration information through the NNP external interface. In response, PDMA transfers the driver from the storage device to the PM based on the configuration from the NNP external interface.

[0114] It can be understood that the process of the neural network processor moving the driver from the storage device to the program memory can be regarded as the initialization loading stage.

[0115] As an example, the configuration information related to the driver may include the storage location of the driver in the storage device and the storage starting row of the driver in the PM. For example, the storage location of the driver in the storage device may include the starting storage address of the driver in the storage device and the row number of the driver.

[0116] It is understandable that the driver is stored in the first storage space of the program memory. The first storage space is determined by the starting storage address (i.e., the starting row) and the ending storage address (i.e., the ending row). As an example, the starting storage address of the first storage space is the starting storage address of the program memory. Specifically, the PDMA can start writing the driver from the starting storage address of the program memory. Therefore, it is also understandable that the ending storage address of the first storage space is determined by the starting storage address of the program memory and the number of rows of the driver.

[0117] In a possible embodiment of the present application, step 501 in the embodiment of the present application can be implemented in the following manner:

[0118] After the NNP starts, the driver is executed through the NNP's computing unit, which monitors in real time whether there are any model tasks to be executed. When a model task is required, the NNP configures the PDMA module to load the first-level load content from the storage device to the program memory. For example, the NNP (specifically, the computing unit that executes the driver in the NNP) configures the PDMA module in the driver to move the first-level load content to the program memory. For example, the NNP configures the PDMA with the first-level load configuration information.

[0119] As an example, the PDMA module loads the first-level content from the storage device to the program memory, including:

[0120] The PDMA obtains the first-level loading configuration information, wherein the first-level loading configuration information includes: the number of first-level loading rows, the starting storage address of the first-level loading content in the storage device, and the starting storage address of the first-level loading content in the PM.

[0121] Based on the first-level load configuration information, the PDMA loads the contents stored at the target location in the storage device (i.e., the second-level load control code and the program of the first function group) into the PM as the first-level load contents. The target location is determined by the first-level load row number and the starting storage address of the first-level load contents in the storage device.

[0122] Optionally, the storage location of the first-level loading content in the PM is located after the storage location of the driver in the PM. As an example, the storage location of the second-level dynamic loading control code, func gp0 program, and driver in the PM can be as follows: Figure 6As shown, the driver's binary machine code (drv hex) is stored in the first storage space, the binary machine code of the second-level dynamic loading control code (l2ld hex) and the binary machine code of the first function group (model func gp0 hex) are stored in the second storage space. Specifically, (l2ld hex) is stored in storage space 1 of the second storage space, and modelfunc gp0 hex is stored in storage space 2 of the second storage space.

[0123] As an example, the first-level loaded content is stored in the second storage space in the program memory. Optionally, the storage address of the second storage space is continuous with the storage address of the first storage space, so that the storage space of the program memory can be saved as much as possible. For example, the end storage address (end row) of the first storage space is continuous with the start storage address (start row) of the second storage space. For example, the driver is stored in rows 0 to 399 of the PM, and the first-level loaded content is stored in rows 400 to 1500.

[0124] It is understood that the end storage address of the second storage space is less than or equal to the end storage address of the program memory. Of course, the storage address of the second storage space may also be discontinuous with the storage address of the first storage space, but in this case, it is necessary to ensure that the sum of the second storage space, the first storage space, and the gap storage space is less than or equal to the total storage space of the program memory.

[0125] It can be understood that the end storage address of the program memory determines the maximum storage space of the program memory.

[0126] As an example, the second storage space includes storage space 1 and storage space 2. Storage space 1 is used to store the secondary dynamic loading control code. Figure 6 As shown, the secondary dynamic loading control code l2ld hex is stored in lines 400 to 450. Storage space 2 is used to store the program of the first function group. For example, Figure 6 As shown, the program of the first function group funcgp0 is stored in lines 451 to 1500. The starting storage address of the storage space 1 and the ending storage address of the first storage space are continuous.

[0127] In one possible implementation of the present application, step 502 in the embodiment of the present application can be implemented in the following manner:

[0128] Step 5021: The neural network processor determines the secondary loading configuration information corresponding to the i-th secondary dynamic loading process based on the secondary dynamic loading control code, where the secondary loading configuration information includes: first information, and a first starting storage address of the i-th secondary loading content in the program memory; i = 1, 2, ..., N-1; the first information is used to determine the storage location of the i-th secondary loading content in the storage device.

[0129] As an example, the neural network processor determines the loading configuration information corresponding to the i-th secondary dynamic loading process according to the secondary dynamic loading control code, including:

[0130] When i=1, the neural network processor determines the starting storage address of the i-th secondary loading content in the storage device according to the address offset and the starting storage address of the first-level loading content in the storage device;

[0131] When i≥2, the neural network processor determines the starting storage address of the i-th secondary load content in the storage device according to the starting storage address of the i-th secondary load content in the storage device and the target load row number corresponding to the i-1th secondary load;

[0132] The number of loaded lines corresponding to the value of the second-level dynamic loading counter in the second-level dynamic loading control code after the i-1th second-level dynamic loading is used as the number of target program lines corresponding to the i-th second-level loading to obtain the first information in the loading configuration information.

[0133] It can be understood that, on the one hand, since the storage addresses of the programs of each function group in the storage device are continuous, and the secondary dynamic loading control code includes at least information 1 for determining the storage location (such as the starting storage address) of the programs of each function group in the storage device. For example, information 1 is used to determine the starting storage address of the program of func gp1 in the storage device. For example, information 1 is the address offset of the starting storage address of the program of func gp1 in the storage device compared to the starting storage address of the first-level loaded content in the storage device. For example, if the starting storage address of the first-level loaded content in the storage device is address X, and the address offset of the program of func gp1 relative to the first-level loaded content in the storage device is 4096, then the starting storage address of the program of func gp1 in the storage device is address (X+4096).

[0134] It is understandable that after the PDMA module transfers the program of each function group to the program memory, the neural network processor increments the number of program lines of the function group program transferred this time as the starting storage address of the program of the function group to be transferred next time in the storage device. For example, assuming that the target function group is the program of func gp1, the size of the program of func gp1 is 4096 bytes, and the starting storage address of the program of func gp1 in the storage device is address 256, then the starting storage address of the program of func gp2 to be transferred next time in the storage device is address 4352. As an example, in the embodiment of the present application, a line can have 16 bytes.

[0135] Step 5022: The neural network processor controls the direct memory access module to use the program of the i+1th function group stored in the corresponding position in the storage device as the i-th secondary loading content according to the secondary loading configuration information, and loads it into the program memory according to the first starting storage address.

[0136] Specifically, the neural network processor configures the secondary loading configuration information to the direct memory access module, so that the direct memory access module reads the program of the i+1th function group from the storage device as the i-th secondary loading content according to the storage location of the i-th secondary loading content in the storage device in the secondary loading configuration information, and then writes the read program of the i+1th function group into the program memory.

[0137] Step 5023: The neural network processor determines whether to execute the (i+1)th secondary dynamic loading process.

[0138] As an example, the secondary dynamic loading control code also includes: the target number of secondary dynamic loading. After the i-th secondary dynamic loading process is executed, if the value of the updated secondary dynamic loading number counter is less than the target number of secondary dynamic loading, it is determined to execute the i+1-th secondary dynamic loading process, otherwise, the secondary dynamic loading process is ended.

[0139] Step 5024: When it is determined that the i+1th secondary dynamic loading process is to be executed, the neural network processor continues to control the direct memory access module to execute the process of loading the program of the i+2 function groups from the storage device to the program memory, and so on, until it is determined that the i+1th secondary dynamic loading process is not to be executed, and then the secondary dynamic loading process is terminated.

[0140] It is worth noting that if the (i+1)th secondary dynamic loading process is not executed, it means that the program of the (i+1)th function group is the last function group program, that is, the (i)th secondary dynamic loading process is the last secondary dynamic loading process. If the (i+1)th secondary dynamic loading process is executed, it means that the program of the (i+1)th function group is not the last function group program, so the neural network processor needs to continue to control the PDMA to execute the secondary dynamic loading process.

[0141] In a possible implementation of the present application, the method provided in the embodiment of the present application may further include: each time the neural network processor executes a secondary dynamic loading process, the value of the secondary dynamic loading times counter in the secondary dynamic loading control code is increased by a preset value.

[0142] It is understood that the initial value of the secondary dynamic load count counter in the secondary dynamic load control code is 0. At this time, the target load row number corresponding to the first secondary dynamic load process, i.e., the load row number corresponding to the secondary dynamic load count index being 0, is 0. After the neural network processor controls the PDMA module to execute the first secondary dynamic load process, the neural network processor increments the value of the secondary dynamic load count counter in the secondary dynamic load control code by 1. That is, after the first secondary dynamic load process, the value of the secondary dynamic load count counter in the secondary dynamic load control code is 1. When the neural network processor executes the second secondary dynamic load process, the load row number for the second secondary dynamic load process is determined to be L2 based on the value of the secondary dynamic load count index being 1. The NNP loads the secondary dynamic load content into the PM based on L2 and the starting storage address of the second secondary dynamic load content. Thereafter, the neural network processor increments the value of the secondary dynamic load count counter in the secondary dynamic load control code by 1. That is, after the second secondary dynamic load process, the value of the secondary dynamic load count counter in the secondary dynamic load control code is 2, and so on.

[0143] In a possible embodiment of the present application, the storage addresses of the driver, the secondary dynamic loading control code, and the programs of any function group in the program memory are continuous.

[0144] In a possible embodiment of the present application, the storage addresses of the secondary dynamic loading control code and the programs of the N function groups in the storage device are continuous.

[0145] In a possible embodiment of the present application, the method provided by the embodiment of the present application may further include before 501: NNP obtains the size of the program of the original target network model and the size of the driver program, and NNP determines whether the size of the program of the original target network model is larger than the remaining storage space of the program memory (here the remaining storage space refers to the total size of the pm space minus the size of the driver program space) based on the size of the program of the target network model and the size of the driver program. If it is larger, the compilation device will segment the program of the target network model. It is understandable that the compilation device can then divide the program of the target network model according to the size of the program of the target network model and the size of the driver program. Figure 4 The process shown is to segment the target network model program. After segmentation, the programs of each function group are stored in the storage device in sequence when the software is deployed. If the size of the original target network model program is less than or equal to the remaining storage space of the program memory, the NNP can be Figure 1 The scheme shown loads the program of the target network model into the PM.

[0146] like Figure 7 As shown, Figure 7 A schematic diagram of a multi-level program loading method according to an embodiment of the present invention is provided. Figure 7 The embodiment shown takes the target network model as a neural network model, and the program of the neural network model is divided into 5 function groups, namely the program of func gp0 to the program of func gp4, and the storage device is DDR as an example. Figure 7 The method shown includes three stages, namely, an initialization loading stage (steps 701 to 702), a first-level dynamic loading process (steps 704 to 707), and a second-level dynamic loading process.

[0147] Step 701: During the initialization loading phase, the PDMA module obtains initialization configuration information through the NNP external interface. The initialization configuration information is used to configure the PDMA to transfer the driver program to the PM.

[0148] It is understandable that the initialization configuration information is used by the PDMA to determine the storage location of the driver in the DDR and the storage location of the driver in the PM.

[0149] For example, the initialization configuration information may include the number of program lines of the driver, the starting storage address of the driver in the DDR, and information about the first starting line corresponding to the driver in the PM.

[0150] Step 702: The PDMA module reads the driver program from the DDR according to the number of program lines of the driver program and the starting storage address of the driver program in the DDR, and writes the driver program into the PM according to the information of the first starting line.

[0151] It can be understood that various programs or codes in the embodiments of the present application are stored in the DDR or PM in the form of binary machine code, so reading or writing refers to reading the machine code of the program or code.

[0152] It is understandable that the PDMA module writes the driver into the PM according to the information of the first start line means that the PDMA module writes the driver into the PM starting from the first start line of the PM, that is, the starting storage address of the driver in the PM is the storage address corresponding to the first start line.

[0153] It is worth noting that after steps 701 and 702, the NNP starts up. After starting up, the NNP executes the driver program and monitors in real time whether there are any model tasks to be performed. When a model task is required, the NNP performs the following first-level dynamic loading phase.

[0154] Step 703: During the first-level dynamic loading phase, the PDMA module obtains the first-level loading configuration information through the NNP internal interface. The first-level loading configuration information is used by the PDMA module to determine the storage location of the first-level loading content in the DDR and the storage location of the first-level loading content in the PM.

[0155] For example, the first-level loading configuration information may include the number of program lines for the first-level loading, the starting storage address of the first-level loading content in the DDR, and the second starting row corresponding to the first-level loading content in the PM.

[0156] Step 704: The PDMA module reads the second-level dynamic loading control code and the program of func gp0 from the DDR as the first-level loading content according to the number of program lines loaded in the first level and the starting storage address of the first-level loading content in the DDR, and writes the second-level dynamic loading control code and the program of func gp0 into the PM according to the information of the second starting line.

[0157] Step 705: The computing unit in the NNP runs the program of func gp0.

[0158] After the first-level dynamic loading is complete, the driver jumps to the second-level dynamic loading control code through a subroutine call. The second-level dynamic loading control code also jumps to the func gp0 program through a subroutine call and begins executing the func gp0 program. After the fun gp0 program is executed, the driver returns to the second-level dynamic loading control code, and the second-level dynamic loading control process begins.

[0159] As an example, the func gp0 program is called by the secondary dynamic loading control code through a subroutine call, and the jump instruction pc points to the secondary dynamic loading start pc.

[0160] Step 706: During the first level 2 dynamic loading process, the NNP determines the starting storage address of the first level 2 loading content in the DDR according to the content of l2ld.

[0161] It's understandable that because the func gp1 program is the first secondary load, the initial value of the storage address of the first secondary load in DDR is the starting storage address of the func gp1 program in DDR, that is, the model base address + the size of the secondary dynamic load control code + the size of the func gp0 program. The size of the secondary dynamic load control code + the size of the func gp0 program is the DDR address offset of the first secondary load in the macro definition file compared to the primary load.

[0162] Step 707: The NNP determines the target number of loading rows of the first secondary loading content according to the initial value of the secondary dynamic loading times counter during the first secondary dynamic loading.

[0163] The initial value of the secondary dynamic loading times counter is 0.

[0164] It's understandable that when looking up the table, NNP uses the value of the secondary dynamic load counter as an index. The lookup table depth is the maximum number of secondary dynamic loads that can be supported. The lookup table information is the number of program lines for each secondary load, which comes from the macro definition file. The lookup table depth affects the length of the secondary dynamic load control code, as shown in Table 1 below:

[0165] Table 1

[0166] The value of the secondary dynamic loading counter The number of program lines per secondary load 0 L1 1 L2 2 L3 …… … N-1 LN

[0167] Taking the value of the secondary dynamic loading times counter as 0 as an example, the NNP can determine that the number of rows of the first secondary loading content is L1.

[0168] Step 708: The NNP configures the first level-2 loading configuration information for the PDMA module to trigger the PDMA module to perform the first level-2 dynamic loading. The level-2 loading configuration information is used by the PDMA module to determine the storage location of the first level-2 loading content in the DDR and the third starting row information corresponding to the first level-2 loading content in the PM.

[0169] For example, the secondary loading configuration information includes the starting storage address of the first secondary loading content in DDR, the target loading row number L1 of the first secondary loading, and the third starting row information corresponding to the first secondary loading content in PM.

[0170] As an example, the starting storage address of the first secondary loading content in DDR comes from the calculation in step 706. After the PDMA module moves the first secondary loading content to the PM, the PDMA module increments the starting storage address of the first secondary loading content in DDR and the number of program lines of the first secondary loading content as the starting storage address of the next secondary loading content in DDR, because all secondary loading contents, i.e., the programs of func gp1 to func gpN, have continuous addresses in DDR.

[0171] The information of the third starting line corresponding to the first secondary loading content in PM comes from the macro definition file, which is the number of program lines of the driver program + the number of program lines of the secondary dynamic loading control code. The target loading line number L1 comes from the calculation in step 707.

[0172] Step 709: The PDMA module reads the program of func gp1 from the DDR as the first level 2 dynamic loading content according to the starting storage address of the first level 2 loading content in the DDR and the L1, and writes the program of func gp1 into the PM according to the information of the third starting line.

[0173] Step 710: The NNP adds 1 to the value of the secondary dynamic loading times counter in the secondary dynamic loading control code.

[0174] It can be understood that since the program of func gp1 and the program of func gp0 have the same starting line in the PM, that is, they have the same starting storage address, when PDMA writes the program of func gp1 into the PM, the program of func gp0 stored in the PM is replaced by the program of func gp1.

[0175] Step 711: Run the program of func gp1.

[0176] As an example, the second-level dynamic loading control code calls the func gp1 program through a subroutine call, and the jump instruction pc points to the second-level dynamic loading start pc.

[0177] Step 712: After executing the program of func gp1, the NNP determines whether the value of the secondary dynamic loading counter is less than or equal to the target number of secondary dynamic loading (eg, 4). It is understood that after step 710, the value of the secondary dynamic loading counter becomes 1.

[0178] Step 713: Since the value of the secondary dynamic loading counter is less than the target number of secondary dynamic loading, the NNP determines the target number of loading rows corresponding to the second secondary loading process, namely L2, according to the current value of the secondary dynamic loading counter.

[0179] Step 714 : The NNP determines the starting storage address of the second level loading content in the DDR according to the starting storage address of the first level loading content in the DDR and the target loading row number corresponding to the first level loading content.

[0180] Step 715: The NNP configures the second secondary loading configuration information for the PDMA module again. The second secondary loading configuration information includes the starting storage address of the second secondary loading content in the DDR, the target loading row number corresponding to the second secondary loading process, and the third starting row information.

[0181] Step 716: The PDMA module reads the program of func gp2 from the DDR as the second secondary load content according to the starting storage address of the second secondary load content in the DDR and the target load row number corresponding to the second secondary load process, and writes it into the PM according to the information of the third starting row.

[0182] Step 717: The NNP adds 1 to the value of the secondary dynamic loading times counter in the secondary dynamic loading control code.

[0183] Step 718: Run the program of func gp2.

[0184] Referring to the process of steps 708 to 718 above, the NNP successively loads the programs of func gp3 to func gp4 into the PM and executes the programs of func gp. After executing the last program of func gp and the value of the secondary dynamic load count counter is greater than or equal to 4, the NNP executes the following step 719.

[0185] Step 719: End the secondary dynamic loading control process.

[0186] Figure 8 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0187] like Figure 8 As shown, the terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. The processor 80 is the neural network processor mentioned above. When the neural network processor 80 executes the computer program 82, it implements the steps in the above-mentioned multi-level program loading method embodiment, for example Figure 5 Steps S501 to S502 are shown.

[0188] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 82 in the terminal device 8.

[0189] The terminal device 8 can be a computing device such as a desktop computer, a notebook computer, a PDA, a smart phone, or a server. Those skilled in the art will understand that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 8 may also include input and output devices, network access devices, buses, etc.

[0190] The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 8. Furthermore, the memory 81 may include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device 8. The memory 81 may also be used to temporarily store data that has been output or is about to be output. The terminal device 8 may also include a communication module, which may provide communication solutions for network devices including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and the like. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have only one array element or an antenna array including multiple array elements. The communication module may receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module may also receive signals to be sent from the processor, frequency modulate and amplify them, and convert them into electromagnetic waves for radiation through the antenna.

[0191] The terminal device 8 may further include a power management module, which may receive input from an external power source, a battery, and / or a charger to power the processor, the memory, the communication module, and the like.

[0192] The terminal device 8 may further include a display module, which may be used to display information input by the user or information provided to the user. The display module may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Furthermore, a touch panel may cover the display panel. When the touch panel detects a touch operation on or near the touch panel, the touch panel transmits the information to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0194] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0195] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0196] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0199] An embodiment of the present application provides a computer program product. When the computer program product runs on a neural network processor, it enables the neural network processor to execute the steps of the above-mentioned multi-level program loading method.

[0200] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory)), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0201] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multi-level program loading method, characterized in that: The method is applied to a neural network processor, which includes a direct memory access module. The method includes: Controlling the direct memory access module to load first-level loading content from a storage device to a program memory according to first-level loading configuration information, wherein the first-level loading content includes a second-level dynamic loading control code and a program of a first function group among N function groups, where the programs of the N function groups are obtained by segmenting the programs of the target network model; and the storage space occupied by the programs of any of the function groups is less than or equal to the remaining storage space of the program memory, where N is an integer greater than or equal to 2; the first-level loading configuration information includes: the number of program lines of the model program first-level dynamic loading, the starting storage address of the first-level loading content in the storage device, and the starting storage address of the first-level loading content in the program memory; After the program of the first function group is executed, controlling the direct memory access module to execute N-1 secondary dynamic loading processes according to the secondary dynamic loading control code until the direct memory access module loads the program of the Nth function group into the program memory, wherein the next secondary dynamic loading process is executed after the program of the function group loaded into the program memory is executed, and the programs of the N function groups have the same starting storage address in the program memory; Among them, the secondary dynamic loading control code is used to manage the transfer and execution of the program of each function group; The secondary dynamic loading control code includes: the target number of secondary dynamic loading, the starting row in the program memory PM of the secondary dynamic loading and the corresponding secondary dynamic loading starting PC, the address offset of the first secondary loading content compared to the first level loading content in the storage device, and the number of program lines for each secondary loading.

2. The method according to claim 1, characterized in that The step of controlling the direct memory access module to perform N-1 secondary dynamic loading processes according to the secondary dynamic loading control code until the direct memory access module loads the program of the Nth function group into the program memory includes: Determining, based on the secondary dynamic loading control code, secondary loading configuration information corresponding to an i-th secondary dynamic loading process, the secondary loading configuration information comprising: first information, and a first starting storage address of the i-th secondary loading content in the program memory; i=1, 2, ..., N-1; the first information is used to determine a storage location of the i-th secondary loading content in the storage device; controlling the direct memory access module to use the program of the (i+1)th function group stored in the corresponding position of the storage device as the i-th secondary loading content according to the secondary loading configuration information, and load the program into the program memory according to the first starting storage address; Determine whether to execute the (i+1)th secondary dynamic loading process; When it is determined that the i+1th secondary dynamic loading process is to be executed, after executing the program of the i+1th function group, the process of controlling the direct memory access module to load the program of the i+2th function group from the storage device to the program memory is continued, and so on, until it is determined that the i+1th secondary dynamic loading process is not to be executed, then the secondary dynamic loading process is ended.

3. The method according to claim 2, characterized in that The secondary dynamic loading control code includes: a secondary dynamic loading times counter, and a target times of secondary dynamic loading; The method further comprises: Each time the secondary dynamic loading process is executed, the value of the secondary dynamic loading times counter is increased by a preset value; Correspondingly, after the i-th secondary dynamic loading process is executed, if the updated value of the secondary dynamic loading times counter is less than the target times of the secondary dynamic loading, it is determined to execute the i+1-th secondary dynamic loading process, otherwise, the secondary dynamic loading process is terminated.

4. The method according to claim 2, characterized in that The first information includes at least: a starting storage address of the i-th secondary loading content in the storage device; The secondary dynamic loading control code includes: an address offset of the first secondary loading content compared to the starting storage address of the primary loading content in the storage device; Accordingly, determining the first information corresponding to the i-th secondary dynamic loading process according to the secondary dynamic loading control code includes: When i=1, determining the starting storage address of the i-th secondary loading content in the storage device according to the address offset and the starting storage address of the primary loading content in the storage device; When i≥2, the starting storage address of the i-th secondary loading content in the storage device is determined according to the starting storage address of the i-1th secondary loading content in the storage device and the target loading row number corresponding to the i-1th secondary loading.

5. The method according to claim 4, characterized in that The first information also includes the number of target program lines corresponding to the i-th secondary dynamic loading process. Accordingly, the method further includes: The number of loaded lines corresponding to the value of the second-level dynamic loading counter in the second-level dynamic loading control code after the (i-1)th second-level dynamic loading is used as the number of target program lines corresponding to the (i)th second-level loading.

6. The method according to any one of claims 1 to 5, characterized in that The controlling the direct memory access module to load the first-level loading content from the storage device to the program memory includes: The first-level loading configuration information is configured for the direct memory access module, so that the direct memory access module loads the first-level loading content from the storage device to the program memory according to the first-level loading configuration information.

7. The method according to any one of claims 1 to 5, characterized in that The program memory further stores a driver program. The storage addresses of the driver program, the secondary dynamic loading control code, and any program of the function group in the program memory are continuous.

8. The method according to any one of claims 1 to 5, characterized in that The storage addresses of the secondary dynamic loading control code and the programs of the plurality of function groups in the storage device are continuous.

9. A neural network processor, characterized in that include: Direct memory access module, operation module, and program memory, wherein, The operation module is configured to: control the direct memory access module to load the first-level loading content from the storage device to the program memory according to the first-level loading configuration information, the first-level loading content including the second-level dynamic loading control code and the program of the first function group among the programs of N function groups, the programs of the N function groups being obtained by segmenting the programs of the target network model; and the storage space occupied by the programs of any one of the function groups is less than or equal to the remaining storage space of the program memory, where N is an integer greater than or equal to 2; After the program of the first function group is run, the operation module is further configured to: execute the second-level dynamic loading control code and control the direct memory access module to perform N-1 second-level dynamic loading processes until the program of the Nth function group is loaded into the program memory, wherein the next second-level dynamic loading process is performed after the program of the function group loaded into the program memory is run, and the programs of the N function groups have the same starting storage address in the program memory; the first-level loading configuration information includes: the number of program lines of the model program for first-level dynamic loading, the starting storage address of the first-level loaded content in the storage device, and the starting storage address of the first-level loaded content in the program memory; Among them, the secondary dynamic loading control code is used to manage the transfer and execution of the program of each function group; The secondary dynamic loading control code includes: the target number of secondary dynamic loading, the starting line of the PM of the secondary dynamic loading and the corresponding secondary dynamic loading starting PC, the address offset of the first secondary loading content compared to the first loading content in the storage device, and the number of program lines of each secondary loading.

10. A terminal device comprising a memory and a processor, wherein the memory is used to store a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images