Instruction generation method, image processing method and device for executing neural network operation

Abstract

Disclosed are an instruction generation method, an image processing method and a device for performing neural network operations. The instruction generation method for performing neural network operations includes: determining at least one intermediate layer set based on the first intermediate layer group in the neural network; the first intermediate layer group corresponds to the first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed; respectively determining the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set; generating instructions for performing neural network operations based on the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set. The embodiments of the present disclosure can perform operations according to the actual value of the number of image frames, so that redundant calculations can be effectively avoided while ensuring the normal processing of image frames, thereby reducing the waste of resources.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an instruction generation method, an image processing method and a device for executing neural network operations. Background Art

[0002] In actual scenarios, there is a neural network that can output the same number of results when a certain number of image frames are provided to it. Before inputting the image frames, the actual value of the number of image frames cannot be known, but only the maximum possible value of the number of image frames can be known. Therefore, regardless of the actual value of the number of image frames, the image frames are generally directly processed according to the maximum possible value, which will cause a waste of computing resources. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide an instruction generation method, an image processing method and a device for executing neural network operations.

[0004] According to one aspect of an embodiment of the present disclosure, there is provided a method for generating instructions for executing a neural network operation, comprising:

[0005] Determining at least one intermediate layer set based on a first intermediate layer group in the neural network; the first intermediate layer group corresponds to a first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed;

[0006] Respectively determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set;

[0007] Based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set, an instruction for executing the neural network operation is generated.

[0008] According to another aspect of the embodiments of the present disclosure, there is provided an image processing method, including:

[0009] Based on the target number of frames, and the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined according to the first intermediate layer group in the neural network, the number of operations to be performed is determined for each intermediate layer set respectively; the first intermediate layer group corresponds to the first input layer group in the neural network, the number of image frames used to input the first input layer group is not fixed, and the sum of the determined numbers of operations to be performed is equal to the target number of frames in value;

[0010] Through each set of intermediate layers, according to the determined number of operations to be performed, the instructions for neural network operations generated by the method provided in the previous aspect are executed on the target number of image frames, so as to perform operations on the target number of image frames.

[0011] According to another aspect of the embodiments of the present disclosure, there is provided an instruction generation device for executing a neural network operation, comprising:

[0012] A first determination module is used to determine at least one intermediate layer set based on a first intermediate layer group in the neural network; the first intermediate layer group corresponds to a first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed;

[0013] A second determination module, used to respectively determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set determined by the first determination module;

[0014] A generation module is used to generate instructions for executing the neural network operation based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set determined by the second determination module.

[0015] According to another aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0016] A determination module, configured to determine the number of operations to be performed for each intermediate layer set based on the target number of frames, and the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined according to a first intermediate layer group in the neural network; the first intermediate layer group corresponds to a first input layer group in the neural network, the number of image frames input to the first input layer group is not fixed, and the sum of the determined number of operations to be performed is numerically equal to the target number of frames;

[0017] A processing module is used to execute the instructions for neural network operations generated by the device provided in the previous aspect on the target number of image frames through each intermediate layer set according to the number of operations to be performed determined by the determination module, so as to perform operations on the target number of image frames.

[0018] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned instruction generation method for executing neural network operations, or to execute the above-mentioned image processing method.

[0019] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0020] processor;

[0021] a memory for storing instructions executable by the processor;

[0022] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned instruction generation method for performing neural network operations, or to implement the above-mentioned image processing method.

[0023] Based on the instruction generation method, image processing method, device, computer-readable storage medium and electronic device for performing neural network operations provided by the above-mentioned embodiments of the present disclosure, at least one intermediate layer set can be obtained based on the first intermediate layer group in the neural network. Next, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set can be determined respectively, and based on the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, instructions for performing neural network operations are generated for each intermediate layer set respectively. In this way, using the generated instructions for performing neural network operations, a reasonable number of operations to be performed for each intermediate layer set can be determined in combination with the number of image frames to be processed, and operations are performed accordingly. Therefore, the embodiments of the present disclosure can perform operations not according to the maximum possible value of the number of image frames, but according to the actual value of the number of image frames. This can effectively avoid redundant calculations while ensuring the normal processing of image frames, thereby reducing waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1-1 It is one of the schematic diagrams of the neural network to which the embodiments of the present disclosure are applicable.

[0026] Figure 1-2 This is the second schematic diagram of a neural network applicable to the embodiments of the present disclosure.

[0027] Figure 1-3 This is the third schematic diagram of a neural network applicable to the embodiments of the present disclosure.

[0028] Figure 2 It is a flowchart of a method for generating instructions for executing neural network operations provided by an exemplary embodiment of the present disclosure.

[0029] Figure 3It is a flowchart of a method for generating instructions for executing neural network operations provided by another exemplary embodiment of the present disclosure.

[0030] Figure 4 It is a flowchart of a method for generating instructions for executing neural network operations provided by another exemplary embodiment of the present disclosure.

[0031] Figure 5 It is a flowchart of a method for generating instructions for executing neural network operations provided by another exemplary embodiment of the present disclosure.

[0032] Figure 6 Schematic diagram of the splitting of the first intermediate layer group in an exemplary embodiment of the present disclosure.

[0033] Figure 7 It is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure.

[0034] Figure 8 It is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure.

[0035] Fig. 9 It is a flowchart of an image processing method provided by yet another exemplary embodiment of the present disclosure.

[0036] Fig.10 It is a schematic diagram of storage space allocation in an exemplary embodiment of the present disclosure.

[0037] Fig.11 is a schematic diagram of a repeat instruction in an exemplary embodiment of the present disclosure.

[0038] Fig.12 Schematic diagram of the behavior of repeating instructions in an exemplary embodiment of the present disclosure.

[0039] Fig.13 It is a schematic diagram of the structure of an instruction generation device for executing neural network operations provided by an exemplary embodiment of the present disclosure.

[0040] Fig.14 It is a schematic diagram of the structure of an instruction generation device for executing neural network operations provided by another exemplary embodiment of the present disclosure.

[0041] Fig.15 It is a structural diagram of an instruction generation device for executing neural network operations provided by yet another exemplary embodiment of the present disclosure.

[0042] Fig.16 It is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present disclosure.

[0043] Fig.17 is a schematic structural diagram of an image processing device provided by another exemplary embodiment of the present disclosure.

[0044] Fig.18 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0046] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0047] Those skilled in the art will appreciate that the terms "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meanings or necessarily logical orders. "Multiple" may refer to two or more, and "at least one" may refer to one, two, or more.

[0048] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0049] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0050] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0051] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0052] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0053] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0054] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0055] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0056] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0057] Application Overview

[0058] In actual scenarios, there is a process of simultaneously inputting a certain number of image frames into a neural network to obtain the same number of results. This number can be called the number of samples (batch). In the process of implementing the present disclosure, the inventor found that the actual value of the batch cannot be known in the compilation stage, but only the maximum possible value of the batch can be known. The compilation stage is generally calculated according to the maximum possible value of the batch. If the actual value of the batch is less than the maximum possible value of the batch, it will cause a waste of computing resources. How to avoid the waste of computing resources is an urgent problem to be solved for those skilled in the art.

[0059] Exemplary Systems

[0060] The embodiments of the present disclosure may mainly include two stages, namely a compilation stage and a running stage; wherein, in the compilation stage, the instruction generation method for performing neural network operations provided by the embodiments of the present disclosure may be executed to generate instructions for performing neural network operations; in the running stage, the image processing method provided by the embodiments of the present disclosure may be executed based on the instructions generated in the compilation stage to realize the operation processing of image frames, thereby avoiding the waste of computing resources.

[0061] It should be noted that the architecture of the neural network applicable to the embodiments of the present disclosure can be as follows: Figure 1-1 , Figure 1-2 as well as Figure 1-3 The neural network shown.

[0062] Figure 1-1 The neural network shown has i input groups and j output groups. Each input group can have several (e.g., a number from 0 to N0, a number from 0 to N1, etc.) or fixed 1 input (each input can be an image frame), and each output group can have several outputs (each output can be the result of the neural network performing calculations on an image frame), but each output group must be equal to the batch size of a certain input group and the input and output correspond one-to-one, and the output in this output group has no relationship with the input in other input groups whose batch size is not fixed to 1. For example, output 0 corresponds one-to-one to input 0, that is, the number of input 0s is equal to the number of output 0s (N0==P0), and the ath (0<=a<N0)th input 0 determines the ath output 0, different inputs 0 do not interfere with each other, and the values ​​of other inputs (e.g., input 1, input i-1) do not affect the value of output 0.

[0063] Figure 1-2 The neural network shown has 2 input groups and 2 output groups. Output 0 corresponds to input 0 one-to-one, and output 1 corresponds to input 1 one-to-one. In addition, the value of input 1 will not affect the value of output 0, and the value of input 0 will not affect the value of output 1. In this way, for several input 0s, the first input 0 corresponds to the first output 0, and the values ​​of other input 0s do not affect the value of the first output 0.

[0064] Compared to Figure 1-2 The neural network shown, Figure 1-3 The neural network shown has an additional input group that only includes input 2, but the batch of this input group is fixed to 1, and the results of all outputs of the neural network (that is, the value of each output 0 and the value of each output 1) will be affected by input 2.

[0065] By summary Figure 1-1 to Figure 1-3From the characteristics of the neural network shown, it can be seen that the neural network applicable to the embodiments of the present disclosure needs to meet the following conditions: if the input in each input group of batch>1 is dyed with one color, and then the associated (including direct or indirect) intermediate layer or output will be dyed with the same color, then each intermediate layer and each input of the neural network will have and will only be dyed with one color, the number of batches of the same color remains equal, and each batch corresponds one to one.

[0066] Exemplary Methods

[0067] Figure 2 It is a flowchart of a method for generating instructions for executing neural network operations provided by an exemplary embodiment of the present disclosure. Figure 2 The method shown includes step 201, step 202 and step 203, and each step is described below.

[0068] Step 201, based on the first intermediate layer group in the neural network, determine at least one intermediate layer set; the first intermediate layer group corresponds to the first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed.

[0069] Assume that the neural network involved in step 201 is Figure 1-2 The neural network shown in the figure has: The first intermediate layer group can be Figure 1-2 The first input layer group can be used to input Figure 1-2 The number of image frames used to input the first input layer group can be any positive integer less than or equal to N0; or the first intermediate layer group can be Figure 1-2 The first input layer group can be used to input Figure 1-2 The number of image frames used to input the first input layer group can be any positive integer less than or equal to N1.

[0070] Optionally, the number of intermediate layer sets determined based on the first intermediate layer group can be 1, 2, 3 or more than 3, each intermediate layer set can respectively include at least one intermediate layer in the first intermediate layer group, and the intermediate layers in all intermediate layer sets can constitute the first intermediate layer group.

[0071] Step 202 : Determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set respectively.

[0072] Here, each intermediate layer in the first intermediate layer group may have a corresponding number, so that based on the number of each intermediate layer in the first intermediate layer group and which intermediate layers in the first intermediate layer group are specifically included in each intermediate layer set, the intermediate layer number information of the intermediate layer set can be conveniently and reliably determined. In addition, the number of intermediate layers in the intermediate layer set can be obtained by counting the number of intermediate layers included in each intermediate layer set.

[0073] Step 203: Generate instructions for executing neural network operations based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set.

[0074] Here, based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set, instructions for executing neural network operations can be generated for each intermediate layer set. In this way, in the running stage, for each intermediate layer set, the generated corresponding instructions can be used to perform image frame operations, so as to realize the operation processing based on the actual value of the number of image frames (the specific implementation method can be seen in the following Figure 7 Description of the embodiment shown), thereby avoiding waste of computing resources.

[0075] exist Figure 2 Based on the embodiment shown, Figure 3 As shown, step 203 includes step 2031, step 2032 and step 2033.

[0076] Step 2031 : Based on the group identifier of the first input layer group, and the intermediate layer number information and the number of intermediate layers of each intermediate layer set, determine the triplet corresponding to each intermediate layer set respectively.

[0077] Here, each input layer group in the neural network may have a corresponding group identifier, and the group identifier of each input layer group may be recorded and stored in advance so as to obtain the group identifier of the first input layer group later, and each intermediate layer in the first intermediate layer group may share the group identifier. Optionally, the group identifier may be in digital form, Roman letter form, etc., and the group identifier may be specifically represented by h.

[0078] In step 2031, for any intermediate layer set, the group identifier of the first input layer group, the intermediate layer numbering information and the number of intermediate layers of the intermediate layer set can be combined to form a triplet corresponding to the intermediate layer set. Assuming that the intermediate layer numbering information of the intermediate layer set only includes the starting number of the intermediate layer and is represented by m, and the number of intermediate layers of the intermediate layer set is represented by k, the triplet corresponding to the intermediate layer set can be expressed as (h, m, k).

[0079] Step 2032, assign registers to the triples corresponding to each intermediate layer set.

[0080] Here, the register can be represented by r. Optionally, there can be a one-to-one correspondence between the triples and the registers; or there can be a non-one-to-one correspondence between the triples and the registers. For example, when the group identifiers, the intermediate layer number information, and the number of intermediate layers in the two triples are the same, the two triples can share the same register.

[0081] Step 2033, based on the registers allocated to the triples corresponding to each intermediate layer set, as well as the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, generate instructions for executing neural network operations.

[0082] Here, based on the registers assigned to the triples corresponding to each intermediate layer set, as well as the intermediate layer number information and the number of intermediate layers of each intermediate layer set, instructions for executing neural network operations can be generated for each intermediate layer set, so that in the running stage, for each intermediate layer set, image frame operations can be performed based on the use of corresponding registers, so as to realize operation processing based on the actual value of the number of image frames (the specific implementation method can be seen in the following Figure 8 Description of the embodiment shown), thereby avoiding waste of computing resources.

[0083] exist Figure 2 Based on the embodiment shown, Figure 4 As shown, step 203 includes step 2034 and step 2035.

[0084] Step 2034, allocate input storage space and output storage space of a single intermediate layer to each intermediate layer set respectively.

[0085] Here, for any set of intermediate layers, the input storage space allocated to each intermediate layer therein can be the same, the output storage space allocated to each intermediate layer therein can also be the same, the allocated input storage space of a single intermediate layer can be represented as a, and the allocated output storage space of a single intermediate layer can be represented as b.

[0086] Step 2035, based on the input storage space and output storage space of a single intermediate layer allocated to each intermediate layer set, as well as the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, generate instructions for executing neural network operations.

[0087] Here, based on the input storage space and output storage space of a single intermediate layer allocated to each intermediate layer set, as well as the intermediate layer number information and the number of intermediate layers of each intermediate layer set, instructions for executing neural network operations can be generated for each intermediate layer set, so that in the running stage, for each intermediate layer set, image frame operations can be performed based on the corresponding input storage space and output storage space, so as to ensure the orderly implementation of the operation processing (the specific implementation method can be seen in the following Fig. 9 1 and 2).

[0088] exist Figure 2 Based on the embodiment shown, Figure 5 As shown, step 201 includes step 2011, step 2012 and step 202.

[0089] Step 2011, determine whether the first intermediate layer group in the neural network meets the preset splitting condition; if the first intermediate layer group meets the preset splitting condition, execute step 2012; otherwise, execute step 2013.

[0090] Step 2012: split the first intermediate layer group into multiple intermediate layer sets.

[0091] Step 2013: The first intermediate layer group is regarded as an intermediate layer set.

[0092] Here, whether the first intermediate layer group meets the preset splitting condition can be determined based on the total number of intermediate layers in the first intermediate layer group. Optionally, if the total number of intermediate layers in the first intermediate layer group is large, it can be determined that the first intermediate layer group meets the preset splitting condition. In this case, the first intermediate layer group can be split into multiple intermediate layer sets, for example, Figure 6 As shown, if the total number of intermediate layers in the first intermediate layer group is 7, the first intermediate layer group can be split into 3 intermediate layer sets, wherein the first two intermediate layer sets respectively include 2 intermediate layers in the first intermediate layer group, and the last intermediate layer set includes the last 3 intermediate layers in the first intermediate layer group. If the total number of intermediate layers in the first intermediate layer group is not large, for example, the total number is only 2, it can be determined that the first intermediate layer group does not meet the preset splitting condition, and the first intermediate layer group can be directly determined as an intermediate layer set.

[0093] Of course, when determining whether the first intermediate layer group meets the preset splitting condition, in addition to considering the total number of intermediate layers in the first intermediate layer group, factors such as program running efficiency may also be considered.

[0094] It can be seen that in the embodiment of the present disclosure, according to whether the first intermediate layer group meets the preset splitting condition, at least one intermediate layer set can be obtained in a suitable manner.

[0095] Figure 7 It is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure. Figure 7 The method shown includes step 701 and step 702, and each step is described below.

[0096] Step 701, based on the target number of frames, and the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined according to the first intermediate layer group in the neural network, determine the number of operations to be performed for each intermediate layer set respectively; the first intermediate layer group corresponds to the first input layer group in the neural network, the number of image frames used to input the first input layer group is not fixed, and the sum of the determined numbers of operations to be performed is numerically equal to the target number of frames.

[0097] It should be noted that the introduction of the first intermediate layer group and the first input layer group can be referred to the relevant description of step 201 above, which will not be repeated here.

[0098] Here, the target frame number may be the number of image frames to be processed by the first intermediate layer group, and the target frame number may be numerically less than or equal to the number of intermediate layers included in the first intermediate layer group. For example, if the first intermediate layer group is Figure 1-2 The target number of frames can be less than or equal to N0; if the first intermediate layer group is Figure 1-2 For an intermediate layer group composed of several intermediate layers 1 in the middle, the target frame number can be less than or equal to N1.

[0099] In step 701, when determining the number of operations to be performed for each intermediate layer set based on the target number of frames and the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined according to the first intermediate layer group in the neural network, it is necessary to ensure that the sum of the determined number of operations is numerically equal to the target number of frames, so that the sum of the determined number of operations may be numerically equal to the number of intermediate layers included in the first intermediate layer group, or may be less than the number of intermediate layers included in the first intermediate layer group. Optionally, in step 701, it can also be ensured that the number of operations to be performed determined for any intermediate layer set is numerically less than or equal to the number of intermediate layers of the intermediate layer set, so as to avoid the number of operations to be performed determined for the intermediate layer set exceeding the computing power of the intermediate layer set.

[0100] Step 702, through each intermediate layer set, according to the determined number of operations to be performed, the instructions for neural network operation are executed on the target number of image frames (the instructions here are the above-mentioned Figure 2 The illustrated embodiment generates instructions for neural network operations) to perform operations on a target number of image frames.

[0101] Here, for each intermediate layer set, since the corresponding number of operations to be performed has been determined by executing step 701, when executing instructions for neural network operations on image frames of the target number of frames, the actual number of operations performed at each intermediate layer set can be the corresponding number of operations to be performed. In this way, when operations are performed on the target number of image frames, the sum of the actual number of operations performed will be numerically equal to the target number of frames. It can be determined that in the embodiments of the present disclosure, operations are not performed according to the maximum possible value of the number of image frames, but are performed according to the actual value of the number of image frames.

[0102] In an embodiment of the present disclosure, at least one intermediate layer set can be obtained based on the first intermediate layer group in the neural network. Next, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set can be determined respectively, and based on the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, instructions for performing neural network operations can be generated for each intermediate layer set respectively. In this way, using the generated instructions for performing neural network operations, a reasonable number of operations to be performed for each intermediate layer set can be determined in combination with the number of image frames to be processed, and operations can be performed accordingly. Therefore, the embodiment of the present disclosure may not perform operations according to the maximum possible value of the number of image frames, but may perform operations according to the actual value of the number of image frames. This can effectively avoid redundant calculations while ensuring normal processing of image frames, thereby reducing waste of resources.

[0103] exist Figure 7 Based on the embodiment shown, Figure 8 As shown, step 702 includes step 7021 and step 7022.

[0104] Step 7021, setting the storage value in the register allocated to the triplet corresponding to each intermediate layer set to the numerical value of the number of operations to be performed determined for the intermediate layer set.

[0105] Here, the registers assigned to the triples corresponding to each intermediate layer set can all be dynamically configurable registers for storing values. In this way, through the dynamic configuration of the storage values, it can be ensured that the storage value in each register is set to the numerical value of the number of operations to be performed determined for the corresponding intermediate layer set.

[0106] Step 7022, through each intermediate layer set, according to the storage value set in the register assigned to the triplet corresponding to each intermediate layer set, the instructions for neural network operation are executed on the target number of image frames (the instructions here are the above Figure 3 Instructions for neural network operations generated by the illustrated embodiment).

[0107] After dynamically configuring the storage values ​​in the registers, when executing instructions for neural network operations on the target number of image frames, the actual number of operations performed at each intermediate layer set can be numerically the storage value set in the register assigned to the corresponding triplet. In this way, the actual number of operations performed at each intermediate layer set is essentially the corresponding number of operations to be performed, thereby realizing operation processing based on the actual value of the number of image frames, thereby avoiding waste of computing resources.

[0108] exist Figure 7 Based on the embodiment shown, Fig. 9 As shown, step 702 includes step 7023.

[0109] Step 7023, executing the instructions for neural network operation on the target number of image frames through each intermediate layer set (the instructions here are the above Figure 4 When an instruction for a neural network operation is generated in the illustrated embodiment, after one operation is completed, the input storage address and the output storage address are offset according to the input storage space and the output storage space of a single intermediate layer allocated to the intermediate layer set.

[0110] Assuming that the number of operations to be performed for a certain intermediate layer set is 3 times, then 3 operations need to be performed at the intermediate layer set. If the input storage address for the first operation is X1, and the input storage address for the second operation is X2, then after the first operation is completed, the input storage address can be offset by a, and the output storage address can be offset by b. At this time, the input storage address can be updated to X1+a, and the output storage address can be updated to X2+b. Then, the image frame for the second operation can be obtained from the input storage address X1+a, and the result of the image frame after the operation is stored in the output storage address X2+b; after the second operation is completed, the input storage address can be offset by a again, and the output storage address can be offset by b again. At this time, the input storage address can be updated to X1+2a, and the output storage address can be updated to X2+2b. Then, the image frame for the third operation can be obtained from the input storage address X1+2a, and the result of the image frame after the operation is stored in the output storage address X2+2b.

[0111] The embodiments of the present disclosure, when performing actual operation processing, can better ensure the smooth and orderly implementation of each operation processing by offsetting the input storage address and the output storage address according to the allocated input storage space and the output storage space.

[0112] exist Figure 7 Based on the illustrated embodiment, the intermediate layers in each intermediate layer set are numbered according to a preset numbering rule; step 701 includes:

[0113] Based on the target number of frames, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined by the first intermediate layer group in the neural network, and the method for determining the number of operations that matches the preset numbering rule, the number of operations to be performed is determined for each intermediate layer set.

[0114] Here, the preset numbering rule may be: starting from 0, numbering according to an arithmetic progression, and the common difference is 1; or, the preset numbering rule may be: starting from 1, numbering according to an arithmetic progression, and the common difference is 2; or, the preset numbering rule may be: starting from 0, numbering according to an arithmetic progression, and the common difference is 2. When the preset numbering rule is known, the number of operations to be performed for each intermediate layer set can be determined accordingly.

[0115] In a specific implementation, each intermediate layer numbering information includes an intermediate layer starting number, and a preset numbering rule is: numbering starts from 0, numbering according to an arithmetic progression, and the common difference is 1;

[0116] For each intermediate layer set, the intermediate layer starting number in the intermediate layer number information is represented by m, the number of intermediate layers is represented by k, and the target frame number is represented by t, then:

[0117] When t is less than or equal to m, the value of the number of operations to be performed determined for this intermediate layer set is 0;

[0118] When t is greater than or equal to the sum of m and k, the value of the number of operations to be performed determined for this intermediate layer set is k;

[0119] When t is greater than m and less than the sum of m and k, the value of the number of operations to be performed determined for the current intermediate layer set is the difference between t and m.

[0120] In one example, the first intermediate layer group is Figure 1-2 The intermediate layer group is composed of several intermediate layers 0 in the first intermediate layer group, and the number of intermediate layers 0 is 7. According to the preset numbering rule, the numbers of these 7 intermediate layers 0 can be 0, 1, 2, 3, 4, 5, and 6 in sequence. Assume that the first intermediate layer group is split into Figure 6 For the three intermediate layer sets shown in the figure, the intermediate layer starting number m of the first intermediate layer set is specifically 0, and the number k of intermediate layers in the first intermediate layer set is specifically 2; the intermediate layer starting number m of the second intermediate layer set is specifically 2, and the number k of intermediate layers in the second intermediate layer set is specifically 2; the intermediate layer starting number m of the third intermediate layer set is specifically 4, and the number k of intermediate layers in the second intermediate layer set is specifically 3.

[0121] Assuming that the target number of frames t is 4, the first intermediate layer set satisfies that t is greater than or equal to the sum of m and k, the second intermediate layer set satisfies that t is greater than or equal to the sum of m and k, and the third intermediate layer set satisfies that t is less than or equal to m. Therefore, the number of operations to be executed determined for the first intermediate layer set is 2, the number of operations to be executed determined for the second intermediate layer set is 2, and the number of operations to be executed determined for the third intermediate layer set is 0.

[0122] Assuming that the target number of frames t is 6, the first intermediate layer set satisfies that t is greater than or equal to the sum of m and k, the second intermediate layer set satisfies that t is greater than or equal to the sum of m and k, and the third intermediate layer set satisfies that t is greater than m and less than the sum of m and k. Therefore, the number of operations to be executed determined for the first intermediate layer set is 2, the number of operations to be executed determined for the second intermediate layer set is 2, and the number of operations to be executed determined for the third intermediate layer set is 2.

[0123] In this implementation, by adopting a method for determining the number of operations that matches a specific numbering rule of starting from 0, numbering according to an arithmetic progression, and having a tolerance of 1, it can be ensured that the number of operations to be performed determined for each intermediate layer set does not exceed the computing power of the intermediate layer set, and the sum of the determined number of operations to be performed is exactly equal to the target number of frames in numerical value, thereby avoiding waste of computing resources.

[0124] It should be pointed out that when other numbering rules other than the specific numbering rule of starting from 0, numbering according to an arithmetic progression, and having a tolerance of 1 are used as preset numbering rules, the method for determining the number of operations needs to be adjusted accordingly. It is only necessary to ensure that the sum of the number of operations to be performed determined based on the method for determining the number of operations is numerically equal to the target number of frames.

[0125] In the embodiments of the present disclosure, by adopting a method for determining the number of operations that matches a preset numbering rule to determine the number of operations to be performed for each intermediate layer, the reasonable reliability of the determination result can be better guaranteed, thereby effectively avoiding the waste of computing resources.

[0126] In an optional example, it is possible to first determine whether the neural network belongs to the neural network applicable to the embodiments of the present disclosure. The specific determination method can refer to the description of the "Exemplary System" section and will not be repeated here. If the judgment result is yes, a corresponding model input number h can be determined for each input layer group in the neural network (the model input number can be 0, 1, 2, 3, 4, etc., which is equivalent to the group identifier mentioned above).

[0127] Next, all intermediate layer groups (which may include the first intermediate layer group mentioned above) are compiled according to the maximum possible value of the number of image frames. In order to ensure the efficiency of program operation, the intermediate layer groups can be split, specifically, as follows: Fig.10 As shown, the intermediate layer group can be split into two halves, and the hardware first calculates the first half of the intermediate layer group, and then calculates the second half of the intermediate layer group. According to the split result, storage space can also be allocated to each input / output / intermediate layer, and the space allocated to each input sample is the same size, and the space allocated to each output sample is the same size (this is equivalent to allocating a single intermediate layer input storage space and output storage space to each intermediate layer set in the above text). Specifically, the space size allocated to each input sample can be a (which is equivalent to the input storage space in the above text), and the space size allocated to each output sample can be b (which is equivalent to the output storage space in the above text). In addition, the number of intermediate layers k and the sample starting sequence number m (which is equivalent to the intermediate layer starting number in the above text) of each intermediate layer set can also be recorded.

[0128] Afterwards, for each intermediate layer set, a unique triple can be determined. The triple can be expressed as (h, m, k). A register r (which can also be called a repeat instruction register) can be assigned to the triple corresponding to each intermediate layer set. If the triplets of two intermediate layer sets are the same, they can share the same repeat instruction register.

[0129] After that, a repeat instruction can be generated according to (r, a, b) corresponding to each intermediate layer set. The repeat instruction can be specifically as follows: Fig.11 As shown, the (h, m, k) corresponding to each register a can also be recorded. Fig.11 As shown, each repetition instruction can be followed by a certain calculation instruction. The repetition instruction and the subsequent calculation instruction can form an instruction for neural network operation, and the behavior of the repetition instruction will affect the behavior of the subsequent calculation instruction, such as Fig.12 As shown, the behavior of repeating the instruction may specifically be:

[0130] (1) When executing a repeat instruction, read the value stored in register r in the repeat instruction, denoted as q; wherein a in register r represents the offset of each repetition of the input address, and b in register r represents the offset of each repetition of the output address;

[0131] (2) If q≤0, skip the calculation instruction following the repeat instruction and execute the next calculation instruction following the repeat instruction;

[0132] (3) If q=1, the calculation instruction following the repeat instruction is executed normally once;

[0133] (4) If q>1, the calculation instruction following the repeat instruction is executed repeatedly q times. After each calculation, the input storage address increases by a and the output storage address increases by b.

[0134] After the compilation phase, in the running phase, the value t of the actual number of samples of each input (which is equivalent to the target number of frames mentioned above) can be obtained, and then the storage value of register r can be set according to the value of (h, m, k) corresponding to each register r. Specifically, for each register r, t corresponding to the input h corresponding to it can be obtained. If t<=m, then the value of register r is set to 0, if t>=m+k, then the value of register r is set to k, if m<t<m+k, then the value of register r is set to tm. Finally, in the actual operation processing, the number of operations actually performed at each intermediate layer set can be numerically equal to the value set in the corresponding register r, so as to realize the operation processing according to the actual number of image frames to reduce the waste of resources.

[0135] Any instruction generation method for executing a neural network operation provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Alternatively, any instruction generation method for executing a neural network operation provided by the embodiments of the present disclosure can be executed by a processor, such as the processor executing any instruction generation method for executing a neural network operation mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be repeated below.

[0136] Any image processing method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any image processing method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any image processing method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0137] Exemplary Devices

[0138] Fig.13 It is a schematic diagram of the structure of an instruction generation device for executing neural network operations provided by an exemplary embodiment of the present disclosure. Fig.13 The device shown includes a first determining module 131 , a second determining module 132 and a generating module 133 .

[0139] A first determination module 131 is used to determine at least one intermediate layer set based on a first intermediate layer group in the neural network; the first intermediate layer group corresponds to a first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed;

[0140] The second determination module 132 is used to respectively determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set determined by the first determination module 131;

[0141] The generation module 133 is used to generate instructions for executing neural network operations based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set determined by the second determination module 132.

[0142] In an alternative example, Fig.14 As shown, the generating module 133 includes:

[0143] The first determination submodule 1331 is used to determine the triplet corresponding to each intermediate layer set based on the group identifier of the first input layer group and the intermediate layer number information and the number of intermediate layers of each intermediate layer set determined by the second determination module 132;

[0144] The first allocation submodule 1332 is used to allocate registers to the triples corresponding to each intermediate layer set determined by the first determination submodule 1331;

[0145] The first generation submodule 1333 is used to generate instructions for executing neural network operations based on the registers allocated to the triples corresponding to each intermediate layer set by the first allocation submodule 1332, and the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set determined by the second determination module 132.

[0146] In an alternative example, Fig.15 As shown, the generating module 133 includes:

[0147] The second allocation submodule 1334 is used to allocate input storage space and output storage space of a single intermediate layer to each intermediate layer set determined by the first determination module 131;

[0148] The second generation submodule 1335 is used to generate instructions for executing neural network operations based on the input storage space and output storage space of a single intermediate layer allocated to each intermediate layer set by the second allocation submodule 1334, and the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set determined by the second determination module 132.

[0149] In an alternative example, Fig.14 , Fig.15 As shown, the first determination module 131 includes:

[0150] The second determination submodule 1311 is used to determine whether the first intermediate layer group in the neural network meets the preset splitting condition;

[0151] The third determination submodule 1312 is used to split the first intermediate layer group into multiple intermediate layer sets if the second determination submodule 1311 determines that the first intermediate layer group meets the preset splitting condition; otherwise, treat the first intermediate layer group as one intermediate layer set.

[0152] Fig.16 It is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present disclosure. Fig.16 The device shown includes a determination module 161 and a processing module 162 .

[0153] A determination module 161 is used to determine the number of operations to be performed for each intermediate layer set based on the target number of frames and the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined according to the first intermediate layer group in the neural network; the first intermediate layer group corresponds to the first input layer group in the neural network, the number of image frames used to input the first input layer group is not fixed, and the sum of the determined number of operations to be performed is equal to the target number of frames in value;

[0154] The processing module 162 is used to perform the above-mentioned operations on the target number of image frames according to the number of operations to be performed determined by the determination module 161 through each intermediate layer set. Fig.13 The illustrated embodiment generates instructions for neural network operations to perform operations on a target number of image frames.

[0155] In an optional example, the intermediate layers in the first intermediate layer group are numbered according to a preset numbering rule;

[0156] The determination module 161 is specifically used for:

[0157] Based on the target number of frames, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined by the first intermediate layer group in the neural network, and the method for determining the number of operations that matches the preset numbering rule, the number of operations to be performed is determined for each intermediate layer set.

[0158] In an optional example, each intermediate layer numbering information includes an intermediate layer starting number, and a preset numbering rule is: numbering starts from 0, numbering according to an arithmetic progression, and the common difference is 1;

[0159] For each intermediate layer set, the intermediate layer starting number in the intermediate layer number information is represented by m, the number of intermediate layers is represented by k, and the target frame number is represented by t, then:

[0160] When t is less than or equal to m, the value of the number of operations to be performed determined for this intermediate layer set is 0;

[0161] When t is greater than or equal to the sum of m and k, the value of the number of operations to be performed determined for this intermediate layer set is k;

[0162] When t is greater than m and less than the sum of m and k, the value of the number of operations to be performed determined for the current intermediate layer set is the difference between t and m.

[0163] In an alternative example, Fig.17 As shown, the processing module 162 includes:

[0164] A setting submodule 1621 is used to set the storage value in the register allocated to the triplet corresponding to each intermediate layer set to the value of the number of operations to be performed determined for the intermediate layer set;

[0165] The processing submodule 1622 is used to perform the above processing on the target number of image frames according to the storage value set in the register assigned to the triplet corresponding to each intermediate layer set by the setting submodule 1621 through each intermediate layer set. Fig.14 The illustrated embodiment generates instructions for neural network operations.

[0166] In an optional example, the processing module 162 is specifically configured to:

[0167] After passing through each intermediate layer set, the target number of image frames is performed Fig.15 When the illustrated embodiment generates instructions for neural network operations, after one operation is completed, the input storage address and the output storage address are offset according to the input storage space and the output storage space of a single intermediate layer allocated to the intermediate layer set.

[0168] Exemplary Electronic Devices

[0169] Below, reference Fig.18 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0170] Fig.18 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0171] like Fig.18 As shown, the electronic device 180 includes one or more processors 181 and a memory 182 .

[0172] The processor 181 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 180 to perform desired functions.

[0173] The memory 182 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 181 may run the program instructions to implement the instruction generation method or image processing method for performing neural network operations of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0174] In one example, the electronic device 180 may further include: an input device 183 and an output device 184 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0175] For example, when the electronic device is the first device or the second device, the input device 183 may be a microphone or a microphone array. When the electronic device is a stand-alone device, the input device 183 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0176] In addition, the input device 183 may also include, for example, a keyboard, a mouse, and the like.

[0177] The output device 184 can output various information to the outside, including the determined distance information, direction information, etc. The output device 184 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0178] Of course, to simplify, Fig.18 Only some of the components related to the present disclosure in the electronic device 180 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 180 may also include any other appropriate components.

[0179] Exemplary computer program products and computer-readable storage media

[0180] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the instruction generation method or image processing method for performing neural network operations according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0181] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of an instruction generation method or an image processing method for performing neural network operations according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0183] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0184] The basic principles of the present disclosure are described above in conjunction with specific embodiments. It should be pointed out that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. The specific details disclosed above are only for the purpose of illustration and ease of understanding, not limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0185] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0187] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0188] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0189] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0190] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating instructions for executing a neural network operation, comprising: Determining at least one set of intermediate layers based on the first group of intermediate layers in the neural network; The first intermediate layer group corresponds to the first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed; Respectively determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set; Based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set, an instruction for executing the neural network operation is generated; wherein, in the running stage, for each intermediate layer set, the corresponding instructions that have been generated are respectively executed to perform the operation processing of the image frame, and the sum of the number of operations corresponding to each intermediate layer set is equal to the total number of image frames to be processed; The intermediate layers in the first intermediate layer group are numbered according to a preset numbering rule, and the number of operations corresponding to each intermediate layer set is based on the total number of image frames to be processed, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, and a method for determining the number of operations that matches the preset numbering rule.

2. The method according to claim 1, wherein: The generating, based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set, an instruction for executing the neural network operation comprises: Based on the group identifier of the first input layer group, and the intermediate layer number information and the number of intermediate layers of each intermediate layer set, respectively determine the triplet corresponding to each intermediate layer set; Assign registers to the triples corresponding to each intermediate layer set respectively; Instructions for executing the neural network operation are generated based on the registers allocated to the triples corresponding to each intermediate layer set, as well as the intermediate layer number information and the number of intermediate layers of each intermediate layer set.

3. The method according to claim 1, wherein: The generating, based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set, an instruction for executing the neural network operation comprises: Allocate input storage space and output storage space of a single intermediate layer to each set of intermediate layers respectively; Instructions for executing the neural network operation are generated based on the input storage space and output storage space of a single intermediate layer allocated to each intermediate layer set, as well as the intermediate layer number information and the number of intermediate layers of each intermediate layer set.

4. An image processing method, comprising: Based on the target number of frames, according to the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined by the first intermediate layer group in the neural network, and the method for determining the number of operations, the number of operations to be performed is determined for each intermediate layer set respectively; The intermediate layers in the first intermediate layer group are numbered according to a preset numbering rule, the method for determining the number of operations matches the preset numbering rule, the first intermediate layer group corresponds to a first input layer group in the neural network, the number of image frames used to input the first input layer group is not fixed, and the sum of the determined number of operations to be performed is numerically equal to the target number of frames; Through each set of intermediate layers, the instructions for neural network operations generated by claim 1 are executed on the target number of image frames according to the determined number of operations to be performed, so as to perform operations on the target number of image frames.

5. The method according to claim 4, wherein: The step of executing the instructions for neural network operation generated by claim 1 on the target number of image frames according to the determined number of operations to be performed through each intermediate layer set comprises: Setting the storage value in the register assigned to the triplet corresponding to each intermediate layer set to the value of the number of operations to be performed determined for the intermediate layer set; Through each intermediate layer set, the instructions for neural network operation generated by claim 2 are executed on the target number of image frames according to the storage values ​​set in the registers assigned to the triples corresponding to each intermediate layer set.

6. An instruction generation device for executing a neural network operation, comprising: A first determination module, configured to determine at least one intermediate layer set based on a first intermediate layer group in a neural network; The first intermediate layer group corresponds to the first input layer group in the neural network, and the number of image frames used to input the first input layer group is not fixed; A second determination module, used to respectively determine the intermediate layer number information and the number of intermediate layers of each intermediate layer set in the at least one intermediate layer set determined by the first determination module; A generating module, configured to generate instructions for executing the neural network operation based on the intermediate layer number information and the number of intermediate layers of each intermediate layer set determined by the second determining module; wherein, in the running stage, for each intermediate layer set, the corresponding instructions generated are respectively executed to perform operation processing of the image frame, and the sum of the number of operations corresponding to each intermediate layer set is numerically equal to the total number of image frames to be processed; The intermediate layers in the first intermediate layer group are numbered according to a preset numbering rule, and the number of operations corresponding to each intermediate layer set is based on the total number of image frames to be processed, the intermediate layer numbering information and the number of intermediate layers of each intermediate layer set, and a method for determining the number of operations that matches the preset numbering rule.

7. An image processing device, comprising: A determination module, configured to determine the number of operations to be performed for each intermediate layer set based on the target number of frames, according to the intermediate layer number information and the number of intermediate layers of each intermediate layer set in at least one intermediate layer set determined by the first intermediate layer group in the neural network, and a method for determining the number of operations; The intermediate layers in the first intermediate layer group are numbered according to a preset numbering rule, the method for determining the number of operations matches the preset numbering rule, the first intermediate layer group corresponds to a first input layer group in the neural network, the number of image frames used to input the first input layer group is not fixed, and the sum of the determined number of operations to be performed is numerically equal to the target number of frames; A processing module is used to execute the instructions for neural network operation generated by claim 6 on the target number of image frames through each set of intermediate layers according to the number of operations to be performed determined by the determination module, so as to perform operation processing on the target number of image frames.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the instruction generation method for executing a neural network operation as described in any one of claims 1 to 3, or to execute the image processing method as described in any one of claims 4 to 5.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the instruction generation method for performing neural network operations as described in any one of claims 1-3, or to implement the image processing method as described in any one of claims 4-5.

Images