Method, device and system for evaluating implementation resources of artificial intelligence chip design

Abstract

The invention relates to a method, device and system for evaluating implementation resources of artificial intelligence chip design. A method for evaluating implementation resources of artificial intelligence (AI) chip design is characterized by comprising: parsing a representation of a neural network model for the chip to generate network model data, the network model data comprising operators of layers of the neural network model, parameter sets associated with the operators, and connection relationships between the layers; and analyzing the network model data based on an operator library to determine resources required for realizing each layer of the neural network model, the operator library comprising at least one entry, the entry specifies a corresponding operator, a parameter set for the operator, and an associated algorithm for calculating resources required for hardware implementation corresponding to the operator.

Description

Technical Field

[0001] The present disclosure relates to methods, devices and systems for evaluating implementation resources of artificial intelligence (AI) chip designs, particularly artificial intelligence (AI) image signal processing (ISP) chips. Background Art

[0002] With the continuous development of artificial intelligence technology, the demand for artificial intelligence related chips is also increasing day by day. Applying AI technology to the field of ISP (image signal processing) has also become a development trend. Artificial intelligence image signal processing (AI-ISP) can give full play to the capabilities of AI processing on the basis of the traditional ISP architecture, especially in the fields of image processing such as noise reduction and high dynamic fusion (HDR), which has obvious effect advantages over traditional algorithms. Therefore, chip design based on AI-ISP has also become a hot topic of research.

[0003] In the process of designing the hardware architecture of the AI-ISP chip, the calculation method of each layer of artificial intelligence (such as a neural network or a neural network model) needs to be hardened (hardware) to implement. This process requires the algorithm engineer to submit the selected algorithm model to the chip engineer. The chip engineer needs to understand and design the implementation method of each layer, and then count the corresponding resources layer by layer, and then feed back the statistical results to the algorithm engineer for evaluation and subsequent optimization. This process is cumbersome. In particular, when chip engineers need to evaluate and compare multiple models, the entire work becomes extremely arduous and inefficient.

[0004] The present disclosure provides novel methods, devices, and systems for evaluating implementation resources for the design of artificial intelligence (AI) chips, particularly artificial intelligence (AI) image signal processing (ISP) chips. Summary of the invention

[0005] According to one aspect of the present disclosure, a method for evaluating implementation resources of an artificial intelligence (AI) chip design is provided, comprising: parsing a representation of a neural network model for the chip to generate network model data, the network model data comprising operators of each layer of the neural network model, parameter sets associated with the operators, and connection relationships between layers; analyzing the network model data based on an operator library to determine resources required to implement each layer of the neural network model, the operator library comprising at least one entry, the entry specifying a corresponding operator, a parameter set for the operator, and an associated algorithm for calculating resources required for a hardware implementation corresponding to the operator.

[0006] In some embodiments, the method further includes: training a neural network using the operator specified by the entry of the operator library and the parameter set for the operator to obtain a representation of a trained neural network model, wherein in the representation of the trained neural network model, the configuration of the operator and the related parameter set is consistent with the configuration of the operator and the related parameter set in the corresponding entry in the operator library.

[0007] In some embodiments, the method also includes: building an operator library, including: determining operations related to the chip design and corresponding operators and parameter sets; determining a hardware implementation method for performing operations corresponding to the operator with the parameter set on the chip; determining an algorithm for calculating resources corresponding to the hardware implementation method; and defining an entry for the operator, the entry specifying the corresponding operator, the parameter set for the operator, and the algorithm for calculating the resources required for the hardware implementation corresponding to the operator.

[0008] In some embodiments, the resources required for the hardware implementation corresponding to the operator include: the type and quantity of components required to implement the operator on a chip.

[0009] In some embodiments, the method is used to evaluate implementation resources of an AI image signal processing (ISP) chip design.

[0010] In some embodiments, the operator may be a basic operator or a combinatorial operator, and the combinatorial operator is composed of a plurality of basic operators.

[0011] In some embodiments, the neural network model is represented in ONNX format.

[0012] In some embodiments, the entry includes an initialization function and one or more of a resource calculation function and a delay calculation function, wherein the initialization function is used to initialize the operator using a given parameter set, the resource calculation function is used to calculate the hardware resources required by the operator under the corresponding parameter set, and the delay calculation function is used to calculate the resources required for the cache corresponding to the delay associated with the operator.

[0013] In some embodiments, the entry is implemented as a class or a compiled or executable version thereof.

[0014] In some embodiments, the method further includes: integrating the required resources and generating a resource report.

[0015] In some embodiments, analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model includes: performing initialization, including: based on the parameter settings of the initial image of the neural network model, the operators and related parameter sets of each layer, and the connection relationship between the layers, calling the corresponding entries of the operator library, generating instances of the corresponding operators of each layer, wherein the instances of the corresponding operators of each layer receive input from the instances of the operators of the corresponding layers based on the connection relationship between the layers.

[0016] In some embodiments, analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model also includes: calculating the resources required for the instances of each operator based on an algorithm for calculating the resources required for the hardware implementation corresponding to the operator in the entry of the operator library.

[0017] In some embodiments, analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model also includes: determining whether there is a layer with multiple inputs in the neural network model, receiving inputs from at least two paths at the layer with multiple inputs; determining whether there is a delay between the inputs of at least two paths at the layer with multiple inputs that needs to be processed; and determining the resources required for the cache corresponding to the delay.

[0018] In some embodiments, the method further includes: evaluating the neural network model based on required resources to select an optimized hardware implementation for the neural network model.

[0019] According to another aspect of the present disclosure, a computer system is provided, including: a processor; and a memory, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method described in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, there is also provided an apparatus for evaluating implementation resources of an artificial intelligence (AI) image signal processing (ISP) chip design, comprising: a network model data generating component for parsing a representation of a neural network model for an artificial intelligence (AI) chip to generate network model data, wherein the network model data comprises operators and related parameter sets of each layer of the neural network model and connection relationships between the layers; an operator library, wherein the operator library comprises at least one entry, wherein the entry specifies a corresponding operator, a parameter set for the operator, and an algorithm for calculating resources required for a hardware implementation corresponding to the operator, wherein the configuration of the operator and the related parameter set in the representation of the neural network model is consistent with the configuration of the operator and the related parameter set in the corresponding entry in the operator library; and an analyzing component for analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model.

[0021] In some embodiments, the analysis component includes: a component for initialization, including: based on the parameter settings of the initial image for the neural network model, the operators and related parameter sets of each layer, and the connection relationship between the layers, calling the entries of the operator library, and generating instances of the corresponding operators of each layer, wherein the instances of the corresponding operators of each layer receive input from the instances of the operators of the corresponding layer based on the connection relationship between the layers.

[0022] In some embodiments, the analysis component further comprises: a component for calculating resources required by instances of each operator based on an algorithm for calculating resources required for hardware implementation corresponding to the operator in an entry of the operator library.

[0023] In some embodiments, the analysis component also includes a delay calculation component, which is used to: determine whether there are multiple input layers in the neural network model, and receive inputs from at least two paths at the multiple input layers; determine whether there is a delay between at least two inputs at the multiple input layers; and determine the resources required for the cache corresponding to the delay.

[0024] In some embodiments, the device also includes at least one of the following: a resource report generation component, which is used to integrate the resources required for each layer and generate a resource report; a neural network training component, which is used to train the neural network using the operators specified by the entries of the operator library and the parameter sets for the operators to obtain a representation of the trained neural network model; and an evaluation component, which is used to evaluate the neural network model based on the required resources to select an optimized hardware implementation for the neural network model.

[0025] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0028] Figure 1 A schematic block diagram of implementation resources for evaluating artificial intelligence (AI) chip design according to an embodiment of the present disclosure is shown;

[0029] Figure 2A A schematic diagram showing an exemplary representation of a neural network in ONNX format according to an embodiment of the present disclosure;

[0030] Figure 2BShown with Figure 2A The operators specified in the operator library corresponding to the layers / nodes in the ONNX format representation shown;

[0031] Figure 3 A schematic diagram showing a generated network model according to an embodiment of the present disclosure is shown;

[0032] Figure 4 An exemplary code for calling corresponding operators on each layer of a network model according to an embodiment of the present disclosure is shown;

[0033] Figure 5 A schematic flow chart of a method for evaluating implementation resources of an AI chip design according to some embodiments of the present disclosure is shown;

[0034] Fig. 6A and 6B A block diagram showing an exemplary implementation of a convolution operator according to an embodiment of the present disclosure;

[0035] Fig. 7A A portion of an example code showing an entry or association structure of an operator library according to an embodiment of the present disclosure; and

[0036] Figure 7B Another part of the example code showing an entry or association structure of an operator library according to one embodiment of the present disclosure;

[0037] Figure 7C A portion of an example code showing an entry or association structure of an operator library according to an embodiment of the present disclosure;

[0038] Fig.7D Another part of the example code showing an entry or association structure of an operator library according to one embodiment of the present disclosure;

[0039] Fig. 7E Another part of the example code showing an entry or association structure of an operator library according to one embodiment of the present disclosure;

[0040] Figure 8 A schematic diagram of a composite operator according to an embodiment of the present disclosure is shown;

[0041] Fig. 9 An exemplary code of an entry or an association structure corresponding to the composite operator according to an embodiment of the present disclosure is shown;

[0042] Fig.10 A schematic diagram of a network model containing data hopping edges according to an embodiment of the present disclosure is shown;

[0043] Fig.11A schematic diagram of a network model containing data hopping edges according to another embodiment of the present disclosure is shown;

[0044] Fig.12 A computer system according to an embodiment of the present disclosure is shown; and

[0045] Fig.13 A device for evaluating implementation resources of an artificial intelligence (AI) image signal processing (ISP) chip design according to some embodiments of the present disclosure is shown.

[0046] Note that in the embodiments described below, the same reference numerals are sometimes used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In this specification, similar reference numerals and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0047] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure. In addition, techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered part of the authorized specification.

[0048] It should be understood that the following description of at least one exemplary embodiment is merely illustrative and does not constitute any limitation on the present disclosure and its application or use. It should also be understood that any implementation described exemplarily herein does not necessarily mean that it is preferred or advantageous over other implementations. The present disclosure is not limited by any stated or implied theory given in the above technical field, background technology, summary of the invention or specific embodiments.

[0049] Those skilled in the art will readily appreciate that an artificial neural network (ANN) may include a set of connected units or nodes, called artificial neurons. Signals are transmitted along the connections between artificial neurons, similar to synapses. That is, the artificial neuron that receives the signal processes the signal and then signals the connected neuron. Connection weights (also called synaptic weights) are associated with the connections and nodes. The weights are adjusted as learning proceeds. Many types of neural networks are known, including feedforward neural networks, multilayer perceptrons, deep neural networks, and convolutional neural networks (CNNs).

[0050] Accordingly, the algorithm of the neural network (e.g., a deep learning algorithm) can be composed of one computing unit after another. Generally, such a computing unit is called an operator in the relevant field. In the network model, an operator can correspond to a computing process, a logical process, and / or a function computing / mapping process in a layer. Here, various suitable processes for a value domain or a function domain or any other suitable type are collectively referred to as operation processes. In this article, the term "operator" is used to describe or represent various corresponding operation processes.

[0051] Those skilled in the art will readily understand that in the relevant field, "operator" can represent the name of the corresponding operation process, and can also represent the corresponding operation process itself. For example, convolution can be regarded as an operator, weight summation can be regarded as an operator, and activation function in a network model can also be regarded as an operator.

[0052] However, it should be understood that all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by ordinary technicians in the field to which the present disclosure belongs. It will be further understood that the terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and related technology, and unless explicitly defined herein, should not be interpreted in an idealized or overly formal sense.

[0053] In addition, certain terms may be used in the following description for reference purposes only and are not intended to be limiting. For example, the words "first," "second," and other such numerical words referring to structures or elements do not imply a sequence or order unless the context clearly indicates otherwise.

[0054] It should also be understood that when the term “include / comprises” is used in this document, it indicates the presence of the specified features, integers, steps, operations, units and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, units and / or components and / or their combinations.

[0055] The following will provide a detailed description with reference to the accompanying drawings.

[0056] Figure 1 A schematic block diagram of implementation resources for evaluating artificial intelligence (AI) chip design according to an embodiment of the present disclosure is shown. The schematic block diagram may also be applicable to a method or a module.

[0057] In box 101, a network model data generation module (which may be indicated by reference numeral 101) receives a representation of a trained neural network model for a chip. In some embodiments, the representation 102 of the neural network model may be data or a file in ONNX format. ONNX format is an open format for representing deep learning models proposed by Microsoft and Meta, which defines a set of standard formats that are independent of the environment and platform. ONNX files not only store the weights of the neural network model, but also store the structural information of the model, the input and output of each layer in the network, and other information.

[0058] The network model data generation module generates (101) network model data based on the representation of the neural network model for the chip. For example, the network model data generation module can, for example but not limited to, use a scripting language to parse the data or file in ONNX format, extract the model configuration parameters in the ONNX format model, so as to obtain the operators and related parameters used by each layer (or node) used by the model, and determine the topological relationship (connection relationship or input-output relationship) between the layers of the model. Thereby generating network model data that can be used by the subsequent modules. Here, the generated network model data can at least include the operators of each layer of the neural network model, the parameter sets related to the operators, and the connection relationship between the layers.

[0059] In box 103, the analysis module (which may also be indicated by reference numeral 103) analyzes (103) the generated network model data to determine the resources required to implement each layer of the neural network model. For example, the analysis module may receive the network model data, call the operator library 105, and analyze the network model layer by layer according to the topology or connection relationship between the layers of the network model to determine the resources required to implement each layer of the neural network model. The analysis module may also generate a configuration file for the corresponding resources.

[0060] The operator library 105 may include one or more entries, which may specify a corresponding operator, a parameter set for the operator, and an algorithm for calculating the resources required for the hardware implementation corresponding to the operator. For example, the entry may be implemented as an association structure. For example, in some embodiments, the association structure may be a class or a compiled version or executable version thereof. The algorithm for calculating the resources required for the hardware implementation corresponding to the operator may be implemented as a function, a method, or a calculation formula, for example, a function or method encapsulated in a class. However, the present disclosure is not limited to this; rather, it is sufficient to associate the operator, its parameter set, and the algorithm for calculating the required resources in any appropriate form.

[0061] Figure 2A Shows an ONNX representation of the layers / nodes of an exemplary network model. Figure 2AA layer or node in a graph is shown, and the layer / node can be configured to perform a corresponding operation process. Here, as an example, the corresponding operator type (op_type) is convolution (Conv), that is, the operator (and therefore, the layer / node) is used to perform a convolution operation.

[0062] Figure 2A The example in also defines the input "inputx" and weight (conv1.weight) of the node, as well as the output (output). It also defines several attributes, such as the attribute "dilations", the attribute "group", the attribute "kernal shape", the attribute "pads", and the attribute "strike", and defines the parameters (or parameters) and their types of each attribute. It should be understood that Figure 2A The ONNX format representation of the operator shown is only exemplary and not restrictive. Those skilled in the art will readily appreciate that the ONNX format or other appropriate formats can be used to describe various operators of a neural network model and its layers or nodes, as well as various parameter sets for the operators, etc.

[0063] Accordingly, an entry (e.g., class) for the operator of the corresponding convolution operation may be set in the operator library 105. Here, preferably, the format / configuration of the association structure for the operator in the operator library may be set to be the same as the format / configuration of the corresponding operator in the representation of the trained neural network model obtained. In some embodiments, preferably, in the representation of the neural network model, the configuration of the operator and the associated parameter set is consistent with the configuration of the operator and the associated parameter set in the corresponding entry in the operator library. In this way, the entry in the operator library (such as the aforementioned association structure (e.g., class)) may be easily called to automatically analyze the obtained network model data.

[0064] Here, as an example, Figure 2B As shown, the convolution operator Conv2d is defined in the operator library 105. Name represents the operator name. The operator name can be automatically generated, and the naming method can be the operator type plus the default number. Input_data is used to represent the input data of the operator, which is used to identify the topological connection relationship. Input_width represents the input size (width), and input_channel (or in_channel) / output_channel (or out_channel) represents the number of input channels / output channels. Ksize represents the kernel size (kernal size), that is, the size of the convolution kernel. Stride represents the step value. Scale represents the image scaling ratio.

[0065] In actual use, the input width mainly affects the size of the row cache resources and the corresponding row delay. For example, if the input image is 2048 pixels wide, the row delay and row cache will be twice the width of the 1024 image. On the other hand, for the scaling ratio, for example, assuming that the image size of a certain layer is 128*128*32 (width*height*number of channels (w*h*channel)), conv2d operation is performed, and assuming that the size after the downsampling operator is 64*64*32 (w*h*channel), conv2d operation is still performed. Because the length and width are reduced to 1 / 2 of the original, the overall scale parameter is 4. In terms of hardware resource design, the multiplier and other resources of this layer will be 1 / 4 of the previous layer.

[0066] right Figure 2A The representation of the network model in ONNX format is parsed, for example, through a script, the current convolution kernel size is extracted to be 3*3, the stride size is 2*2, and a row or column of data is added (pad) on the top, bottom, left, and right. According to the size of the weight conv1.weight, the input and output channels (channel) of the current conv can be obtained. In this way, the various parameters required by the Conv2d operator in the operator library can be obtained. As mentioned above, preferably, the configuration of the operators and their parameter sets defined in the operator library is consistent with the configuration of the corresponding operators and their parameter sets in the representation (or description file) of the neural network model.

[0067] Here, the operator may include a basic operator and / or a combination operator, and the combination operator is composed of a plurality of basic operators. Here, the basic operator is no longer subdivided. The basic operator may be understood as an operator whose operation properties make it impossible to further subdivide or unsuitable for further subdivision, or an operator whose further subdivision brings uneconomical or other disadvantages. Later, the combination operator will be further described below.

[0068] Back to Figure 1 In block 103, the analysis module may call corresponding entries or association structures in the operator library to determine the resources required to implement the operators of each layer of the neural network model. In some embodiments, the analysis module may also generate a configuration file for the corresponding resources.

[0069] The resources required for the hardware implementation corresponding to the operator include: the type and quantity of components required to implement the operator on a chip. For example, the components may be, for example, one or more of a memory, an adder, a multiplier, a divider, a buffer, etc., or a combination thereof. However, the present disclosure is not limited thereto, and for example, the resources may also be, for example, various types of logic gates to be used and their quantities.

[0070] Optionally, in block 107, a resource report generation module (which may also be indicated by reference numeral 107) integrates the resources required by each layer of the network model to generate a resource report. For example, the resource report generation module 107 may receive the required resource data or configuration file generated in block 103, parse and visualize the corresponding data, and generate more intuitive report data, such as displaying it in the form of a Microsoft Excel file or other visual forms.

[0071] Figure 3 A schematic diagram showing a visual representation of a test network model according to an embodiment of the present disclosure is shown; Figure 4 According to one embodiment of the present disclosure, Figure 3 The exemplary code generated by calling the corresponding operators of the corresponding scripts at each layer of the visualization model.

[0072] like Figure 3 As shown, the visualization model of the exemplary trained neural network model includes 6 layers (or nodes), conv_0 to conv_4 and pixelshuffle (pixel rearrangement), each corresponding to an operator. Layers conv_0 to conv_4 are each convolution operators, and the pixelshuffle layer corresponds to the pixelshuffle operator. For the pixelshuffle operator, it mainly realizes the conversion of channel dimension data and width and height (W, H) dimension data, which is similar to the upsampling function; the basic operation of this operator is as follows Fig. 7E as shown (to be further explained later).

[0073] Figure 4 Shows the Figure 3 The example code generated by calling the corresponding operators at each layer of the visualization model. Figure 4 As shown in the figure, the method / function _init_() of the class test_model defines the image parameter image, parameter img_width, instantiated objects conv_0 to conv_4, and object pixshuffle for initialization. The relevant parameters of the parameter image are defined. The parameter image is user-configured information, which is not automatically generated and needs to be configured by the user according to actual needs.

[0074] The parameter img_width defines the image width, which takes the value in the parameter image data structure. Objects conv_0 to conv_4 each call the operator Conv2d in the operator library and assign it with the specified parameters, while the object pixshuffle calls the operator PixelShuffle and assigns it with the specified parameters. Figure 4As shown in , these object codes contain the operator name or type, connection relationship (how many inputs there are and where the inputs come from), and other configuration parameters related to the operator.

[0075] It should be understood that in some embodiments, Figure 4 The code or its compiled version or executable version may be automatically generated. In addition, in this example, the code is written in the form of a class, but the present disclosure is not limited thereto and may also be written in other object-oriented or non-object-oriented program codes or structures. It should also be understood that Figure 4 The names or types of code elements such as various types, parameters, methods, functions, and operation algorithms in the code are merely schematic and illustrative and not restrictive.

[0076] Figure 5 A schematic flowchart of a method for evaluating implementation resources of an AI chip design according to some embodiments of the present disclosure is shown.

[0077] like Figure 5 As shown, the method 500 for evaluating the implementation resources of AI chip design according to an embodiment of the present disclosure may include: in step 501, generating network model data based on the representation of the neural network model for the chip, the network model data including operators of each layer of the neural network model, parameter sets related to the operators, and connection relationships between layers.

[0078] In some embodiments, the representation of the neural network model can be a representation of a trained neural network model. For example, the neural network model can be trained for a specific algorithm. As mentioned above, the trained model can be saved in, for example, ONNX format.

[0079] In some embodiments, method 500 may further optionally include: in step 503, constructing an operator library. Operations related to the chip design and corresponding operators and parameter sets may be first determined. In some implementations, for example, the initialization parameter type of each operator may be made consistent with the operator parameter type used when training the model, for example, the parameter configuration format under the PyTorch framework may be referred to, thereby ensuring universality of use.

[0080] For example, taking the commonly used convolution operation as an example (such as the operator Conv2d discussed above), the parameter set may include (but not limited to): operator name (name), input data (input_data), input size width (input_width), input number of channels (in_channel or input_channel), output number of channels (out_channel), etc. Here, it should be understood that the naming, type, implementation, etc. of operators, parameters, classes, methods / functions and their parameters and parameters given in this disclosure are exemplary and are only used for illustrative purposes, and are not intended to limit the scope of this disclosure.

[0081] Next, the resources required for the hardware implementation of the operation corresponding to the operator on the chip with the parameter set can be determined. For example, for a convolution operation, the hardware implementation can be various.

[0082] Fig. 6A and 6B A block diagram of an exemplary implementation of a convolution operator Conv2d according to an embodiment of the present disclosure is shown. The configuration parameters of the convolution operator are: input image width input_width=640, input channel in_channel=4, output channel out_channel=8, convolution kernel size kernal_size=3, stride=1, and scaling scale=4. Its hardware implementation is a circuit including a cache unit and a calculation unit.

[0083] Fig. 6A The cache unit is shown. As shown in the figure, the cache unit may include several (here, 2) line buffers (line_buffer) and corresponding control logic. Each line buffer can store pixel data of a corresponding channel. Four channels are shown here, and each channel has 640 pixel values. The output of the cache unit goes to the calculation unit.

[0084] Figure 6B Its computing part is shown. As shown in the figure, the computing part receives the output from the cache part, and the computing part may include several (here, 8) convolution computing units 601. Figure 6B The details of the convolution calculation unit 601 are also shown in the partially enlarged part 603. In this implementation, the convolution calculation unit includes 9 multipliers and 8 adders. Thus, in the calculation unit, 72 multipliers and 64 adders will be included. The calculation unit may also include a data management module (logic) to manage the provision of corresponding data to different convolution calculation units. Although embodiments of the present invention may focus mainly on hardware resources for operation processing, in some other embodiments, the hardware resources of these logic or logic modules may also be taken into account.

[0085] For other operators, their (optimized) hardware implementations can be determined similarly.

[0086] In this way, an algorithm for calculating resources corresponding to the optimized hardware implementation can be determined. In different embodiments, the algorithm can be implemented as a function, a method, a lookup table, and any suitable form.

[0087] An entry or an association structure for the operator may be defined in the operator library, and the entry (or association structure) may specify a corresponding operator, a parameter set for the operator, and an algorithm for computing hardware-implemented resources corresponding to the operator.

[0088] The construction method 500 further includes: in step 505, analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model. The operator library includes at least one entry (or association structure), and the association structure can specify / describe a corresponding operator, a parameter set for the operator, and an algorithm for calculating the resources required for the hardware implementation corresponding to the operator.

[0089] In one embodiment, analyzing the network model data (505) may include: in step S5051, initializing the operator. For example, based on the parameter settings of the initial image of the trained neural network model, the operators and related parameter sets of each layer, and the connection relationship between the layers, the corresponding association structure of the operator library may be called to generate the instance of the corresponding operator of each layer. The instance of the corresponding operator of each layer may receive input from the example of the operator of the corresponding layer based on the connection relationship between the layers.

[0090] Fig. 7A A portion of example code of an entry or associated structure for a two-dimensional convolution operator Conv2d in an operator library according to an embodiment of the present disclosure is shown. Figure 7B FIG. 4 shows a portion of an example code of an entry or associated structure for a two-dimensional convolution operator Conv2d in an operator library according to an embodiment of the present disclosure. Fig. 7A As shown, the operator Conv2d is defined as a class (or its compiled or executable version). Here, it is easy to see that Figure 2B The operator Conv2d defined in is actually Fig. 7A Part of the operator Conv2d defined in.

[0091] like Fig. 7A and 7BAs shown, the operator Conv2d is defined in the form of a class. The initialization method / function _init_() is used to initialize the operator (its corresponding class or the corresponding generated object) using the given parameters / parameters. For example, as mentioned above, input_data is used to represent the input data of the operator, which is used to identify the topological connection relationship; input_width represents the input size (width), in_channel (or in_ch) / out_channel (or out_ch) represents the number of input channels / output channels; ksize represents the kernel size (kernal size), that is, the size of the convolution kernel; stride represents the step value; scale represents the image scaling ratio.

[0092] like Fig. 7A and 7B As shown, in addition to defining the initialization method _init_() and class parameters, the operator Conv2d may further define a resource calculation function (module / method / function) calc_resource(), and optionally, a delay calculation function (module / method / function) calc_hv_delay() or calc_line_delay(). In some embodiments, the operator Conv2d may also define a resource report generation function (module / method / function) gen_res_json().

[0093] In a class-based programming language, these three functions or modules are implemented as methods or functions of a class, but the present disclosure is not limited to this. For example, in other embodiments, it can be implemented in other object-oriented or non-object-oriented languages; in this case, these three functions or modules can be defined or implemented in other ways, for example, they can also be defined not in the entry of the operator library, but in other encapsulation structures / classes / programs / functions. It should also be understood that although here, the association structure is implemented as a class to specify / describe the corresponding operator, the parameter set for the operator, and the algorithm for calculating the resources required for the hardware implementation corresponding to the operator; however, the present disclosure is not limited to this. For example, in other embodiments, the association structure can be loose, not encapsulated in the same information encapsulation structure (e.g., class), but as long as it can be called.

[0094] Step S505 may further include: in step S5053, performing resource calculation. For example, the resources required by the instance of each operator may be calculated based on an algorithm for calculating the resources required for the hardware implementation corresponding to the operator in the association structure of the operator library.

[0095] The resource calculation function (module / method / function) calc_resource() can be used to calculate the hardware resources required by the current operator under the input parameters, such as the size of the storage space, the number of basic computing devices (e.g., adders, multipliers, dividers, etc.), etc. Fig. 7A As shown in the example, the resource calculation function calc_resource() calculates the line buffer (represented by the parameter self.line_buffer) when ksize is greater than 1: the size of line_buffer is (ksize-1)*input_channel*input_width. And the number of multipliers (represented by the parameter self.multiplier) is also calculated: the multiplier is calculated as: ksize*ksize*input_channel*output_channel / (stride*stride) / scale.

[0096] In the case of other operators, it is also possible to calculate only or not the line buffer, and the type of computing hardware resources or the computing algorithm may be different. For example, Fig. 7E The calculation method of the relevant hardware resources of the PixelShuffle operator is shown in Figure 1. Since this operator does not involve calculations, but only rearranges pixels, only line buffer resources are needed to cache the current line data. In the resource calculation function (module / method / function) calc_resource(), only the relevant line buffer (e.g., row buffer) is calculated: the required buffer (line_buffer) resource is: self.line_buffer = 1*input_channel*input_width, and the corresponding line delay is 0.

[0097] As an example, Figure 7C FIG. 1 shows a portion of an example code of an entry or association structure of an operator library according to an embodiment of the present disclosure. Figure 7C In the resource calculation function (module / method / function) calc_resource() shown, the number (size) of line buffers, the number of multipliers, and the number of adders under different scaling (self.scale) are calculated.

[0098] Step S505 may further include: in step S5055, calculating the hardware resources required for the delay. For example, it may be determined whether there is a layer with multiple inputs in the neural network model, where the layer with multiple inputs receives inputs from at least two paths; determine whether there is a delay that needs to be processed between the inputs of at least two paths at the layer with multiple inputs; and determine the resources required for the cache corresponding to the delay.

[0099] The delay calculation function (module / method / function) calc_hv_delay() or calc_line_delay() can be used to calculate the size of the required line buffer (e.g., row buffer and / or column buffer) corresponding to the line delay (row delay and / or column delay) that may be introduced when using this operator. When using this operator, a delay may be introduced (e.g., when there is a delay between two inputs), such as a row direction delay and / or a column direction delay. As a result, an additional buffer is required to store the data for data alignment. Using the delay calculation function (module / method / function), the size of the required buffer corresponding to the delay can be calculated.

[0100] like Figure 7B As shown, a method / function calc_hv_delay() is defined. A parameter duty ratio duty_ratio is defined. Similarly, the column delay (vertical delay) v_delay is calculated as ksize>>1. The row delay (horizontal delay) is 0. The function returns the parameters of the relevant calculation. In other embodiments, the method / function calc_hv_delay() may not perform or need not perform calculations, or the line delay may be assigned to 0, such as Fig. 7E shown.

[0101] like Fig.7D As shown, a method / function calc_line_delay() is defined. In the resource calculation function (module / method / function) calc_line_delay(), only the relevant line delay (e.g., row delay) is calculated: line_delay=ksize>>1. Those skilled in the art will easily understand that ksize>>1 means shifting the parameter ksize by 1.

[0102] Resource report generation function (module / method / function) gen_res_json() is used to summarize the information required for the hardware implementation of operators at each layer of the network model, such as the number of adders / multipliers / dividers, storage space, line cache size, etc. This information can be used as data or can be stored in a file according to a predetermined format.

[0103] Figure 7B Also shown is a resource report generation function (module / method / function) gen_res_json() according to an embodiment of the present disclosure. Figure 7BAs shown in the exemplary, the parameter res_json array is defined in the resource report generation function (module / method / function) gen_res_json(), the relevant parameters of the operator are stored in the array, and the hardware resources calculated in the resource calculation function (module / method / function) calc_resource() are stored in the array, such as the number of adders (adder) self.adder, the number of multipliers (multiplier) self.multiplier, the number of dividers (divider) self.divider (here 0), the number of line buffers (line_buffer) self.line_buffer, and the weight size of the node self.weight_size. Those skilled in the art will easily understand that self is used to call the class itself or generate the object of the class itself. Here as an example, the information returned by this function is stored in a predetermined format as a json file, which can also adopt any suitable format known or customized.

[0104] Now back to Figure 5 , method 500 may further include: Optionally, in step S507, synthesizing the required resources to generate a resource report. In some embodiments, the data or file of the previous level (e.g., a file in json format) may be parsed to generate an Excel file or presented in any other visual display format. In the generated Excel table, there may be detailed information on each layer of network hardware resources, and an overall summary information may be generated for more intuitive display and comparison.

[0105] In some embodiments, method 500 may further include: Optionally, in step S509, the neural network is trained using the operator specified by the association structure of the operator library and the parameter set for the operator to obtain a representation of the trained neural network model. For example, the specified operator can be used in a certain layer of the neural network model, and the neural network can be trained using an existing or future developed neural network training method to obtain a trained neural network model and its representation. As an example, a CNN network model known in the art, such as a full CNN network model or a pyramid CNN model (PyNet), can be used, with the operator (basic operator or combined operator) specified in the operator library as the operator corresponding to the node in the network model, and an end-to-end ISP algorithm known in the art or future developed can be used to train the network model.

[0106] In some embodiments, method 500 may further include: Optionally, in step S511, evaluating the neural network model based on required resources to select an optimized hardware implementation for the neural network model.

[0107] According to an embodiment of the present disclosure, the operator library also supports the construction of combinatorial operators. In other words, the operator described in the embodiment of the present disclosure can be a basic operator or a combinatorial operator, and the combinatorial operator is composed of multiple basic operators. Figure 8 FIG. 4 is a schematic diagram of a composite operator according to an embodiment of the present disclosure. For example, a yellow convolution kernel structure (YellowConv) is conceived, such as Figure 8 As shown in the figure, it contains two basic operators Conv2d and one basic operator PixelShuffle. The operator PixelShuffle can be used to rearrange pixels. Then, the combined operator can be built in the operator library. After that, in the process of training the model and evaluating the model, the combined operator can be directly called.

[0108] Fig. 9 FIG. 4 shows an exemplary code of an entry or association structure corresponding to the composite operator according to an embodiment of the present disclosure. Fig. 9 As shown in the figure, the composite operator YellowConv is defined in the form of a class. The initialization method / function _init_() is used to initialize the operator (its corresponding class or the corresponding generated object) with the given parameters / parameters. For example, input_data is used to represent the input data of the operator, which is used to identify the topological connection relationship; in_w represents the size (width) of the input, in_ch / out_ch) represents the number of input channels / output channels; scale represents the image scaling ratio.

[0109] like Fig. 9 As shown, the operator YellowConv calls the existing basic operator Conv2d in the operator library to form objects or instances conv_0 and conv_1, and calls the existing basic operator PixelShuffle to form the object or instance shuffle_0. The input of the object or instance shuffle_0 comes from the operator conv_0. The input of the operator conv_1 comes from the operator shuffle_0.

[0110] In this way, during the model training and model evaluation process, the combination operator YellowConv can be directly called.

[0111] In the process of analyzing and summarizing resource data, there are usually two structural situations. One is Figure 3 In the form of the network model shown, the data flow is serial, so the hardware resources of each module can be directly accumulated.

[0112] Another situation is when data hops. For example, if the two paths of data are not aligned and there is a delay in different paths in the network model, data synchronization (also figuratively called hopping) is required. Fig.10 A schematic diagram of a network model containing data hopping edges according to an embodiment of the present disclosure is shown.

[0113] like Fig.10 As shown in the figure, the first input passes through three Conv2d operators, each of which generates a line delay of one line. The other input does not pass through these three Conv2d operators, so data synchronization is required. Here, it is necessary to generate a line delay of 3 lines in the other input path as well. The corresponding hardware resource is a line buffer of 3 lines to align the data. Then, the basic operator Concat is used to splice the two data in the channel dimension.

[0114] In the process of network structure analysis, since data edge hopping (alignment) does not belong to the basic operator, additional analysis and processing is performed on this edge hopping situation to obtain the row cache required for the edge hopping to ensure the accuracy of the calculation.

[0115] Fig.11 FIG. 2 shows a schematic diagram of a network model containing data hopping edges according to another embodiment of the present disclosure. Fig.11 As shown, operator conv1 and operator conv2 can be instances of the basic operator Conv2d, where operator conv2 forms two rows of line delay. In order to align the data on the path passing through operator conv2 with the data on the path not passing through operator conv2, two rows of line delay are required on the path not passing through operator conv2 to align the data. For this situation where data alignment is required, operator Addlayer is designed to calculate the corresponding line cache resources. Operator Addlayer can trace back the sources of the inputs of the two paths respectively and calculate the line delay resources required for data alignment. For example: assuming that the output of conv1 needs to be aligned with the output of conv2, conv2 will generate 2 lines of line delay. In order to align the outputs of conv1 and conv2, addlayer also needs two lines of line cache for data alignment in the edge jump part.

[0116] According to one aspect of the present disclosure, a computer system is also contemplated. Fig.12 FIG. 2 shows a computer system according to an embodiment of the present disclosure. Fig.12 As shown, the computer system may include: a processor; and a memory, wherein the memory stores a program, and when the program is executed by the processor, the processor executes the method described in any embodiment of the present disclosure.

[0117] According to one aspect of the present disclosure, a program product is also conceived, which is used to cause the processor to perform the method as described in any embodiment of the present disclosure when executed by the processor. For example, the program product may be a non-volatile computer-readable storage medium on which a program is stored, and when the program is executed by the processor, the processor performs the method as described in any embodiment of the present disclosure.

[0118] A device for evaluating implementation resources of an artificial intelligence (AI) image signal processing (ISP) chip design is also conceived. Fig.13 An apparatus for evaluating implementation resources of an artificial intelligence (AI) image signal processing (ISP) chip design according to some embodiments of the present disclosure is shown. Fig.13 As shown, the device 1300 may include: a network model data generating component 1301, which is used to parse the representation of a trained neural network model for an artificial intelligence (AI) chip to generate network model data, wherein the network model data includes operators and related parameter sets of each layer of the neural network model and connection relationships between layers.

[0119] The apparatus 1300 may further include: an operator library 1303, the operator library including at least one entry, the entry specifying a corresponding operator, a parameter set for the operator, and an algorithm for calculating resources required for a hardware implementation corresponding to the operator. The configuration of the operator and the associated parameter set in the representation of the neural network model is consistent with the configuration of the operator and the associated parameter set in the corresponding entry in the operator library.

[0120] The apparatus 1300 may further include: an analysis component 1305, for analyzing the network model data based on the operator library to determine the resources required for each layer of the neural network model. The apparatus 1300 may further include: a resource report generation component, for synthesizing the resources required for each layer and generating a resource report.

[0121] In some embodiments, the analysis component 1305 may include: an initialization component 13051 for initialization. The initialization component 13051 may include: based on the parameter settings of the initial image of the neural network model, the operators of each layer and the related parameter sets, and the connection relationship between the layers, calling the entries of the operator library to generate the instances of the corresponding operators of each layer. The instances of the corresponding operators of each layer receive input from the examples of the operators of the corresponding layer based on the connection relationship between the layers.

[0122] In some embodiments, the analysis component 1305 may further include: a resource calculation component 13053, which is used to calculate the resources required by the instance of each operator based on the algorithm for calculating the resources required for the hardware implementation corresponding to the operator in the entry of the operator library.

[0123] In some embodiments, the analysis component 1305 also includes a delay calculation component 13055, which is used to: determine whether there are multiple input layers in the neural network model, and receive inputs from at least two paths at the multiple input layers; determine whether there is a delay between at least two inputs at the multiple input layers; and determine the resources required for the cache corresponding to the delay.

[0124] In some embodiments, the apparatus 1300 may further include: a neural network training component 1309, which is used to train the neural network using the operator specified by the entry of the operator library and the parameter set for the operator to obtain a representation of the trained neural network model. And in some embodiments, the apparatus 1300 may further include: an evaluation component 1311, which is used to evaluate the neural network model based on the required resources to select an optimized hardware implementation for the neural network model.

[0125] According to the embodiments of the present disclosure, it is particularly suitable for AI-ISP chip design.

[0126] According to the embodiments of the present disclosure, algorithm engineers can flexibly adjust the network structure to train the network model and generate hardware resource evaluation reports in real time, so as to efficiently train and evaluate the AI ​​neural network model, greatly improving efficiency. Chip engineers only need to regularly maintain and update the hardware resource evaluation of the basic operator library, which can effectively reduce the workload of chip engineers.

[0127] According to the embodiments of the present disclosure, on the one hand, multiple network models can be evaluated and selected at the same time; on the other hand, a single model structure can be adjusted and optimized according to resource consumption. For example, if a certain layer of the network consumes a large amount of multiplier resources, it can be optimized by reducing the number of convolution channels. In this way, the chip engineers and algorithm engineers are relatively separated, ensuring that the resource evaluation of the model can be carried out efficiently.

[0128] In addition, according to the embodiments of the present disclosure, the combinatorial operator can be realized by combining the basic operators, which enables the training and evaluation of the combinatorial operator to be completed directly at the software level, thereby greatly reducing the workload and improving the efficiency of AI model training and chip design.

[0129] The embodiments of the present disclosure may be implemented as methods, devices, systems, and / or computer program products. A computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the embodiments of the present disclosure.

[0130] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0131] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0132] The computer-readable program instructions for performing the operation of the embodiments of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of the latter, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider through the Internet). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be executed by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit, to perform the various aspects of the embodiments of the present disclosure.

[0133] Various aspects of the embodiments of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It will be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0134] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes or modules of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other device to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article containing instructions, which implements aspects of the functions / actions specified in one or more boxes or modules of the flowchart and / or block diagram.

[0135] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operation blocks / steps are executed on the computer, other programmable device, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in one or more blocks or modules of the flowchart and / or block diagram.

[0136] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the embodiment of the present disclosure.In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure.For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of each frame of the block diagram and / or flow chart illustration and the frame in the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0137] References in the specification to "one embodiment" or "an embodiment" and other variations of the present principles mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" and any other variations in various places throughout the specification do not necessarily all refer to the same embodiment.

[0138] Those skilled in the art will appreciate that the boundaries between the operations (or steps) described in the above-described embodiments are merely illustrative. Multiple operations can be combined into a single operation, which can be distributed in additional operations, and the operations can be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of specific operations, and the order of operations may be changed in other various embodiments. However, other modifications, variations, and replacements are equally possible. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.

[0139] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. The various embodiments disclosed herein may be combined in any manner without departing from the spirit and scope of the present disclosure. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for evaluating implementation resources of an artificial intelligence (AI) chip design, characterized in that: include: Parsing a representation of a neural network model for the chip to generate network model data, the network model data including operators of each layer of the neural network model, parameter sets associated with the operators, and connection relationships between layers; The network model data is analyzed based on an operator library to determine the resources required to implement each layer of the neural network model, the operator library including at least one entry, the entry specifying a corresponding operator, a parameter set for the operator, and an associated algorithm for calculating the resources required for a hardware implementation corresponding to the operator.

2. The method according to claim 1, characterized in that Also includes: Training a neural network using an operator specified by an entry of the operator library and a parameter set for the operator to obtain a representation of a trained neural network model, In the representation of the trained neural network model, the configuration of the operator and the related parameter set is consistent with the configuration of the operator and the related parameter set in the corresponding entry in the operator library.

3. The method according to claim 1, characterized in that Also includes: Build the operator library, including: Determining operations related to the chip design and corresponding operators and parameter sets; Determining a hardware implementation method for performing an operation corresponding to the operator with the parameter set on the chip; determining an algorithm for computing resources corresponding to the hardware implementation; and An entry for the operator is defined, the entry specifying a corresponding operator, a parameter set for the operator, and an algorithm for calculating resources required for a hardware implementation corresponding to the operator.

4. The method according to claim 1, characterized in that The resources required for the hardware implementation corresponding to the operator include: the type and quantity of components required to implement the operator on a chip.

5. The method according to claim 1, characterized in that The method is used to evaluate the implementation resources of AI image signal processing (ISP) chip design.

6. The method according to claim 1, characterized in that The operator is: Basic operator; or A combinatorial operator, wherein the combinatorial operator is composed of a plurality of basic operators.

7. The method according to claim 1, characterized in that The neural network model is represented in ONNX format.

8. The method according to claim 1, characterized in that The entry includes an initialization function and one or more of a resource calculation function and a delay calculation function, The initialization function is used to initialize the operator using a given parameter set. The resource calculation function is used to calculate the hardware resources required by the operator under the corresponding parameter set. The delay calculation function is used to calculate the resources required for the cache corresponding to the delay associated with the operator.

9. The method according to claim 1, characterized in that: The entry is implemented as a class or a compiled or executable version thereof.

10. The method according to any one of claims 1 to 9, characterized in that The network model data is analyzed based on the operator library to determine the resources required to implement each layer of the neural network model, including: Initialization, including: Based on the parameter settings of the initial image for the neural network model, the operators of each layer and the related parameter sets, and the connection relationship between the layers, the corresponding entries of the operator library are called to generate instances of the corresponding operators of each layer, The instances of the corresponding operators of the layers receive inputs from the instances of the operators of the corresponding layers based on the connection relationship between the layers.

11. The method according to claim 10, characterized in that Analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model also includes: The resources required by each operator instance are calculated based on the algorithm for calculating the resources required for the hardware implementation corresponding to the operator in the entry of the operator library.

12. The method according to claim 1, characterized in that Analyzing the network model data based on the operator library to determine the resources required to implement each layer of the neural network model also includes: Determine whether there is a layer with multiple inputs in the neural network model, where the layer with multiple inputs receives inputs from at least two paths; Determining whether there is a delay between inputs of at least two paths at the layer having multiple inputs that needs to be processed; Resources required for caching corresponding to the delay are determined.

13. The method according to any one of claims 1 to 9 and 11 to 12, characterized in that Also includes: The neural network model is evaluated based on the required resources to select an optimized hardware implementation for the neural network model.

14. A computer system, characterized in that: include: processor; as well as A memory, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to any one of claims 1 to 13.

15. A device for evaluating implementation resources of an artificial intelligence (AI) image signal processing (ISP) chip design, characterized in that: include: A network model data generating component, used to parse the representation of the neural network model for an artificial intelligence (AI) chip to generate network model data, wherein the network model data includes operators and related parameter sets of each layer of the neural network model and connection relationships between layers; an operator library, the operator library comprising at least one entry, the entry specifying a corresponding operator, a parameter set for the operator, and an algorithm for computing resources required for a hardware implementation corresponding to the operator, wherein a configuration of the operator and the associated parameter set in the representation of the neural network model is consistent with a configuration of the operator and the associated parameter set in a corresponding entry in the operator library; The analysis component is used to analyze the network model data based on the operator library to determine the resources required to implement each layer of the neural network model.

16. The device according to claim 15, characterized in that The analysis component comprises: Components used for initialization include: Based on the parameter settings of the initial image for the neural network model, the operators of each layer and the related parameter sets, and the connection relationship between the layers, the entries of the operator library are called to generate components of instances of corresponding operators of each layer, The instances of the corresponding operators of the layers receive inputs from the instances of the operators of the corresponding layers based on the connection relationship between the layers.

17. The device according to claim 16, characterized in that The analysis component also includes: A component for calculating the resources required by an instance of each operator based on an algorithm for calculating the resources required for the hardware implementation corresponding to the operator in an entry of the operator library.

18. The device according to claim 16, characterized in that The analysis component also includes a delay calculation component, which is used to: Determine whether there is a layer with multiple inputs in the neural network model, where the layer with multiple inputs receives inputs from at least two paths; determining whether there is a delay between inputs from at least two paths at the layer having multiple inputs; Resources required for caching corresponding to the delay are determined.

19. The device according to any one of claims 15 to 18, characterized in that Also includes one or more of the following: Resource report generation component, used to integrate the resources required by each layer and generate resource reports; A neural network training component, used to train a neural network using the operator specified by the entry of the operator library and the parameter set for the operator to obtain a representation of a trained neural network model; as well as An evaluation component is used to evaluate the neural network model based on required resources to select an optimized hardware implementation for the neural network model.

Images