Heterogeneous chip model calling method and heterogeneous chip
By segmenting and combining the operator nodes of heterogeneous chips, forming a collection of fusion nodes, and using multiple processors to coordinate processing, the problem of inefficiency of heterogeneous chips when calling the target neural network model is solved, and more efficient data processing is achieved.
Patent Information
- Application Number
- CN202510639299.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-08-01
AI Technical Summary
When calling the target neural network model, heterogeneous chips cannot fully call and coordinate different types of processors, resulting in a decrease in data processing efficiency and an increase in latency.
Through the division strategy, the operator nodes of the target neural network model are divided and combined to form a collection of fusion nodes, and they are assigned to the processing parts corresponding to different types of processors of heterogeneous chips, and the data is processed together using multiple processors.
It effectively saves processing time and improves the data processing efficiency of heterogeneous chips on the target neural network model.
Smart Images
Figure CN120409702A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of heterogeneous chip model calls, and particularly to a heterogeneous chip model call method and a heterogeneous chip. Background Art
[0002] A heterogeneous chip has multiple different types of processors. For example, the processors may include: a Vision Digital Signal Processor (VDSP), a Neural Processing Unit (NPU), and a Central Processing Unit (CPU). A target neural network model can use the heterogeneous chip for data processing. Currently, although the heterogeneous chip call pipeline uses a parallel call method, it can improve the inference frame rate of the target neural network model to a certain extent. However, due to the inability to fully call and coordinate each different type of processor, the data processing efficiency of the heterogeneous chip will decrease, and the input and output delays of the associated target neural network model will increase. Summary of the Invention
[0003] An embodiment of this application provides a heterogeneous chip model call method, including:
[0004] Determine a target neural network model corresponding to the heterogeneous chip, where the target neural network model has multiple operator nodes, the target neural network model includes a preprocessing part, an inference part, and a postprocessing part, and the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the preprocessing part, the inference part, and the postprocessing part;
[0005] Using a partitioning strategy, partition and combine the operator nodes to form a corresponding set of fused nodes;
[0006] Assign the set of fused nodes to the preprocessing part, the inference part, and the postprocessing part respectively;
[0007] Use multiple different types of the processors to jointly process the data respectively associated with the adjusted preprocessing part, inference part, and postprocessing part.
[0008] Optionally, the preprocessing part includes preprocessing operator nodes, the postprocessing part includes postprocessing operator nodes, and the using a partitioning strategy to partition and combine the operator nodes to form a corresponding set of fused nodes includes:
[0009] Partition the operator nodes in the inference part to form a first part of nodes, a second part of nodes, and a third part of nodes;
[0010] Combining the first part of nodes with at least one operator node in the pre-processing part to form a corresponding first fusion node set;
[0011] Determining a corresponding second fusion node set based on the second part of nodes;
[0012] The third part of nodes and the post-processing nodes are combined to form a corresponding third fusion node set.
[0013] Optionally, the assigning the fusion node set to the pre-processing part, the reasoning part, and the post-processing part respectively includes:
[0014] Assigning the first fusion node set to the pre-processing part, wherein the pre-processing part corresponds to the visual digital signal processor of the heterogeneous chip;
[0015] assigning the second fusion node set to the reasoning part, wherein the reasoning part corresponds to the neural network processor of the heterogeneous chip;
[0016] The third fusion node set is assigned to the post-processing part, where the post-processing part corresponds to the central processing unit of the heterogeneous chip.
[0017] Optionally, the partitioning strategy includes:
[0018] Splitting and combining the operator nodes based on the logical operation relationship between the operator nodes; and,
[0019] The operator nodes are divided and combined based on a processing speed of the processor on the operator nodes.
[0020] Optionally, the splitting and combining the operator nodes based on the logical operation relationship between the operator nodes includes:
[0021] Using the convolutional layer in the target neural network model as a segmentation marker;
[0022] Based on the segmentation identifier, the convolution layer and its subsequent operator nodes are divided into the same fusion node set, wherein there is a logical operation relationship between the convolution layer and its subsequent operator nodes, and the subsequent operator nodes include at least one of the following: an activation function layer, a batch normalization layer, and a pooling layer in the target neural network model.
[0023] Optionally, the dividing and combining the operator nodes based on a processing speed of the processor on the operator nodes includes:
[0024] Determine the processing speeds of the memories in multiple ones of the processors when processing the operator nodes respectively;
[0025] Partition the operator nodes into the processing part of the target neural network model data corresponding to the processor of the memory with the highest processing speed.
[0026] Optionally, the method further includes:
[0027] When the set of fusion nodes is a model sub-graph of the target neural network model, obtain, through the board side, each of the model sub-graphs compiled on the development machine, where the heterogeneous chip is disposed on the board side;
[0028] Determine the inference performance data of the processor for processing the model sub-graph;
[0029] Based on the inference performance data, determine whether it is necessary to re-partition and re-combine the model sub-graph.
[0030] Optionally, the using multiple processors of different types to collaboratively process the data respectively associated with the adjusted pre-processing part, the inference part, and the post-processing part includes:
[0031] Process the data associated with the pre-processing part through the vision digital signal processor of the heterogeneous chip;
[0032] Process the data associated with the inference part through the neural network processor of the heterogeneous chip;
[0033] Process the data associated with the post-processing part through the central processor of the heterogeneous chip, where the processing times of the vision digital signal processor, the neural network processor, and the central processor at least partially overlap.
[0034] Optionally, the method further includes:
[0035] During the process of using multiple processors of the heterogeneous chip to collaboratively process the target neural network model, determine whether the multiple processors are load-balanced;
[0036] In the case of determining that the processor is not load-balanced, re-partition and re-combine the operator nodes.
[0037] An embodiment of the present application further provides a heterogeneous chip, including:
[0038] A determination module configured to determine a target neural network model corresponding to a heterogeneous chip, where the target neural network model has multiple operator nodes, and the target neural network model includes a preprocessing part, an inference part, and a postprocessing part, and the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the preprocessing part, the inference part, and the postprocessing part;
[0039] A partitioning module configured to use a partitioning strategy to split and combine the operator nodes to form corresponding sets of fused nodes; and assign the sets of fused nodes to the preprocessing part, the inference part, and the postprocessing part respectively;
[0040] A processing module configured to use the multiple different types of processors to collaboratively process the data respectively associated with the adjusted preprocessing part, inference part, and postprocessing part.
[0041] The heterogeneous chip model calling method according to the embodiments of the present application uses a partitioning strategy set with the goal of balancing the processing time and / or efficiency of multiple processors of a heterogeneous chip to re - split and combine the operator nodes of a target neural network model, and re - forms a preprocessing part, an inference part, and a postprocessing part. And uses multiple heterogeneous processors to process the data respectively associated with the re - determined preprocessing part, inference part, and postprocessing part. When the heterogeneous chip processes the data input to the target neural network model, it effectively saves the processing time and improves the processing efficiency. Description of the Drawings
[0042] Figure 1 It is a flowchart of the heterogeneous chip model calling method according to the embodiments of the present application;
[0043] Figure 2 It is for the embodiments of the present application Figure 1 A flowchart of an embodiment of step S200 therein;
[0044] Figure 3 It is for the embodiments of the present application Figure 1 A flowchart of an embodiment of step S300 therein;
[0045] Figure 4 It is a flowchart of a specific embodiment of the heterogeneous chip model calling method according to the embodiments of the present application;
[0046] Figure 5 It is a flowchart of another specific embodiment of the heterogeneous chip model calling method according to the embodiments of the present application;
[0047] Figure 6 It is a schematic diagram of the partitioning and combination of the operator nodes according to the embodiments of the present application;
[0048] Figure 7 Schematic diagram of the connection structure between the development machine and the board end in the embodiment of the present application;
[0049] Figure 8 Schematic diagram of converting the graph neural network in the embodiment of the present application into a fusion graph;
[0050] Figure 9 Schematic diagram of the division of multiple operator nodes in one embodiment of the embodiment of the present application;
[0051] Figure 10 Block diagram of the heterogeneous chip in the embodiment of the present application. Detailed implementation manners
[0052] Reference is made herein to the accompanying drawings to describe various aspects and features of the present application.
[0053] It should be understood that various modifications can be made to the embodiments claimed herein. Accordingly, the above description should not be regarded as limiting, but merely as exemplifications of embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the present application.
[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0055] These and other features of the present application will become apparent from the following description of the preferred forms of the embodiments given by way of non - limiting example with reference to the accompanying drawings.
[0056] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art can definitely implement many other equivalent forms of the present application.
[0057] When taken in conjunction with the accompanying drawings, the above and other aspects, features, and advantages of the present application will become more apparent in view of the following detailed description.
[0058] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of the present application and can be implemented in various ways. Well - known and / or repetitive functions and structures have not been described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely as a basis for the claims and a representative basis for teaching those skilled in the art to use the present application in substantially any suitable detailed structure in a variety of ways.
[0059] This specification may use phrases such as "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", all of which may refer to one or more of the same or different embodiments according to the present application.
[0060] A heterogeneous chip model calling method according to an embodiment of the present application. This method can be applied to a heterogeneous chip, which in turn can be applied to various scenarios, such as being applied to vehicles, terminal devices, etc. A heterogeneous chip is a chip or technology that integrates different types of processors. For example, the processors can be a VDSP (Vision Digital Signal Processor), an NPU (Neural Processing Unit), a CPU (Central Processing Unit), etc. respectively. By combining the advantages of various processors, a heterogeneous chip can achieve more efficient computing and lower power consumption.
[0061] A target neural network model is a computational model that mimics the structure and function of a biological neural network. It has a large number of interconnected neurons and can handle complex pattern recognition, classification, regression, etc. problems. For example, it can be used to identify objects, scenes, people, etc. in an image. A heterogeneous chip can serve as the hardware foundation, and the target neural network model can use the heterogeneous chip to process data related to the input to the target neural network model (such as the image to be processed input to the target neural network model).
[0062] The following will describe the heterogeneous chip model calling method according to an embodiment of the present application in detail with reference to the accompanying drawings. Figure 1 is a flowchart of the heterogeneous chip model calling method according to an embodiment of the present application, as Figure 1 shown and in conjunction with Figure 6 , the method includes the following steps:
[0063] S100, determine the target neural network model corresponding to the heterogeneous chip, where the target neural network model has multiple operator nodes, the target neural network model includes a preprocessing part, an inference part, and a postprocessing part, and the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the preprocessing part, the inference part, and the postprocessing part.
[0064] Exemplarily, the heterogeneous chip can be set in devices such as terminals, development boards, etc., and can be used to process data related to its corresponding target neural network model. For example, when image data input to the target neural network model needs to be processed, the heterogeneous chip related to the target neural network can process the image data.
[0065] In this embodiment, the identity of the target neural network can be determined based on the identifier of the target neural network model. Thus, the heterogeneous chip can determine its own working mode and / or the calling mode of the target neural network model. Of course, if the target neural network model is adjusted, the working mode of the heterogeneous chip and / or the calling mode of the target neural network model can also be adjusted accordingly.
[0066] The target neural network model has multiple operator nodes. The operator node is a logical computing unit, and the operator node can include the operation layer or computing unit of the target neural network model. Each operator node can perform logical operations. The target neural network model includes a preprocessing part (preprocess), an inference part (inference), and a postprocessing part (postprocess). Each processing part can include at least one operator node and other types of nodes, and can use the heterogeneous chip to perform data processing in their respective corresponding stages.
[0067] The heterogeneous chip includes multiple different types of processors. For example, it can include a VDSP (Vision Digital Signal Processor), an NPU (Neural Processing Unit), and a CPU (Central Processing Unit). The different types of each processor make the heterogeneous chip have heterogeneous characteristics. In this embodiment, the multiple processors respectively correspond to the preprocessing part, the inference part, and the postprocessing part. So that each processor can process the processing part corresponding to it. For example, the VDSP corresponds to the preprocessing part of the target neural network model and can process the data associated with the preprocessing part. The NPU corresponds to the inference part of the target neural network model and is used to process the data associated with the inference part. The CPU corresponds to the postprocessing part of the target neural network model and is used to process the data associated with the postprocessing part.
[0068] S200, using a partitioning strategy, split and combine the operator nodes to form a corresponding set of fusion nodes.
[0069] Exemplarily, operator nodes can be combined so as to use the same processor in the heterogeneous chip as a whole to process relevant data. The operator nodes can be split and combined to form multiple sets of fused nodes, and each set of fused nodes can include at least one operator node and other nodes. A processor in the heterogeneous chip can process the relevant data of the set of fused nodes corresponding to it. For example, the first set of fused nodes is used to process the first input data, and the vision digital signal processor corresponds to the first set of fused nodes, so that the vision digital signal processor can be used to process the first input data.
[0070] In this embodiment, all operator nodes in the target neural network model are split and then combined by using a partitioning strategy to form multiple sets of fused nodes. The partitioning strategy can be set with the goal of balancing the processing time and / or efficiency of multiple processors. In one embodiment, the partitioning strategy can be preset according to comprehensive factors such as the structure and features of the target neural network model, the characteristics of the operator nodes, and / or the characteristics of each different type of processor in the heterogeneous chip.
[0071] S300, assign the sets of fused nodes to the preprocessing part, the inference part, and the postprocessing part respectively.
[0072] Exemplarily, a set of fused nodes with the original preprocessing part as the main body can be assigned to the current preprocessing part. A set of fused nodes with the original inference part as the main body can be assigned to the current inference part. A set of fused nodes with the original postprocessing part as the main body can be assigned to the current postprocessing part.
[0073] Illustratively, the original preprocessing part includes a preprocessing operator node pre node. The original inference part includes operator nodes node 1, operator node 2, operator node 3, …, operator node n-2, operator node n-1, operator node n. The original postprocessing part includes a postprocessing operator node post node. After partitioning and combining each operator node based on the partitioning strategy, the first set of fused nodes formed includes the preprocessing operator node pre node and operator node 1. The second set of fused nodes includes operator nodes node 2, operator node 3, …, operator node n-2. The third set of fused nodes includes operator nodes node n-1, operator node n, and the postprocessing operator node post node. The first set of fused nodes is determined as the current preprocessing part, the second set of fused nodes is determined as the current inference part, and the third set of fused nodes is determined as the current postprocessing part.
[0074] The S400 uses multiple processors of different types to cooperate in processing the data respectively associated with the adjusted preprocessing part, the inference part, and the postprocessing part.
[0075] Exemplarily, multiple processors of different types on a heterogeneous chip can respectively process the data associated with their corresponding processing parts. For example, a vision digital signal processor corresponds to the preprocessing part and can be used to process the data associated with the preprocessing part. A neural network processor corresponds to the inference part and can be used to process the data associated with the inference part. A central processing unit corresponds to the postprocessing part and can be used to process the data associated with the postprocessing part.
[0076] During the data processing, multiple processors can cooperate with each other. For example, multiple processors can run in parallel and process their respective tasks simultaneously, thus saving processing time. In addition, since the partitioning strategy can be set with the goal of balancing the processing time and / or efficiency of multiple processors. Therefore, the time for processing the data currently associated with the preprocessing part, the inference part, and the postprocessing part is the same or similar. Thus, the data processing time is saved during the invocation process of the target neural network model, and the processing efficiency is improved.
[0077] The method for invoking a heterogeneous chip model according to an embodiment of the present application uses a partitioning strategy set with the goal of balancing the processing time and / or efficiency of multiple processors on a heterogeneous chip to re-partition and combine the operator nodes of the target neural network model, and re-forms the preprocessing part, the inference part, and the postprocessing part. And multiple heterogeneous processors are used to respectively process the data associated with the re-determined preprocessing part, inference part, and postprocessing part. When the heterogeneous chip processes the data input to the target neural network model, the processing time is effectively saved and the processing efficiency is improved.
[0078] In an embodiment of the present application, the preprocessing part includes preprocessing operator nodes, the postprocessing part includes postprocessing operator nodes, and using the partitioning strategy, the operator nodes are partitioned and combined to form corresponding sets of fusion nodes, as Figure 2 shown, including the following steps:
[0079] S210, partition the operator nodes in the inference part to form a first part of nodes, a second part of nodes, and a third part of nodes;
[0080] S220, combine the first part of nodes with at least one operator node in the preprocessing part to form a corresponding first set of fusion nodes;
[0081] S230, determine a corresponding second set of fusion nodes based on the second part of nodes;
[0082] S240: Combine the third part of nodes with the post-processing nodes to form a corresponding third fusion node set.
[0083] Exemplarily, the pre-processing component includes a pre-processing operator node (pre node), which performs preliminary conversion or cleaning of input data, optimizing data representation and improving model performance. The post-processing component includes a post-processing operator node (post node), which performs logical operations on input data.
[0084] In one embodiment, the inference portion takes a long time to process associated data using a processor, so the operator nodes in the inference portion can be divided into a first portion of nodes, a second portion of nodes, and a third portion of nodes. The first portion of nodes is combined with at least one operator node and other nodes (such as pre-processing operator nodes) in the pre-processing portion to form a corresponding first fusion node set; the second portion of nodes is determined to be a second fusion node set; and the third portion of nodes is combined with the post-processing node post node to form a corresponding third fusion node set.
[0085] In conjunction with the above embodiment, for example, before the division, the operator nodes in the reasoning part include operator node 1, operator node 2, operator node 3, ..., operator node n-2, operator node n-1, and operator node n. The operator nodes in the reasoning part are divided into a first part of nodes, a second part of nodes, and a third part of nodes. The first part of nodes includes operator node 1 and operator node 2, the second part of nodes includes operator node 3, ..., operator node n-2. The third part of nodes includes operator node n-1 and operator node n.
[0086] The preprocessing operator node pre node, operator node node 1 and operator node node 2 are combined to form a first fusion node set; the operator nodes node3, ..., operator node node n-2 are combined to form a second fusion node set; the operator node node n-1, operator node node n and the post-processing node post node are combined to form a third fusion node set.
[0087] In one embodiment of the present application, the fusion node set is respectively assigned to the pre-processing part, the reasoning part and the post-processing part, such as Figure 3 Shown, including:
[0088] S310, assign the first set of fusion nodes to the pre - processing part, where the pre - processing part corresponds to the vision digital signal processor of the heterogeneous chip.
[0089] Exemplarily, continuing with the above - mentioned embodiments for illustration, the first fusion set can be assigned to the pre - processing part of the target neural network model, or the first fusion set can be determined as the pre - processing part as a whole. The pre - processing part corresponds to the vision digital signal processor of the heterogeneous chip. The vision digital signal processor can process the data associated with the pre - processing part, such as processing the data associated with the pre - processing operator node pre node, operator node node 1, and operator node node 2 in the above - mentioned embodiments.
[0090] S320, assign the second set of fusion nodes to the inference part, where the inference part corresponds to the neural network processor of the heterogeneous chip.
[0091] Similarly, the second fusion set can be assigned to the inference part of the target neural network model, or the second fusion set can be determined as the inference part as a whole. The inference part corresponds to the neural network processor of the heterogeneous chip. The neural network processor can process the data associated with the inference part, such as processing the data associated with operator node node3,..., operator node node n - 2 in the above - mentioned embodiments.
[0092] S330, assign the third set of fusion nodes to the post - processing part, where the post - processing part corresponds to the central processor of the heterogeneous chip.
[0093] Similarly, the third fusion set can be assigned to the post - processing part of the target neural network model, or the third fusion set can be determined as the post - processing part as a whole. The post - processing part corresponds to the central processor of the heterogeneous chip. The central processor can process the data associated with the post - processing part, such as processing the data associated with operator node noden - 1, operator node node n, and post - processing node post node in the above - mentioned embodiments.
[0094] In an embodiment of the present application, the partitioning strategy includes: splitting and combining the operator nodes based on the logical operation relationship between the operator nodes; and splitting and combining the operator nodes based on the processing speed of the processor for the operator nodes.
[0095] Exemplarily, there may be a logical operation relationship between operator nodes. For example, if two operator nodes need to cooperate with each other to jointly complete a task, then there is a logical operation relationship between the two operator nodes. Operator nodes with a relatively low degree of combination of logical operation relationships can be separated, and operator nodes with a relatively high degree of combination of logical operation relationships can be grouped together. This can increase the coherence of the work of operator nodes.
[0096] The partitioning strategy also includes splitting and combining operator nodes based on the processing speed of the processor for the operator nodes. The processing speeds of each processor for the operator nodes are different. Some processors have a relatively high processing speed. When splitting the operator nodes, relatively more operator nodes can be partitioned into the fusion node set corresponding to the processor with stronger processing capabilities, thereby reducing the burden on the processor with weaker processing capabilities and coordinating the load balancing work of each processor.
[0097] In one embodiment, when the target neural network model processes data, its inference time occupies the main time. Therefore, when setting the partitioning strategy, the operator nodes in the inference part can be appropriately partitioned into the pre-processing part and / or the post-processing part, so that during the process of processing the data associated with each processing part of the target neural network model, the processing times of all processors of the heterogeneous chip are balanced or the processing times are close to the same.
[0098] In one embodiment of the present application, the splitting and combining of the operator nodes based on the logical operation relationship between the operator nodes includes:
[0099] Taking the convolutional layer in the target neural network model as the splitting identifier;
[0100] Based on the splitting identifier, the convolutional layer and its subsequent operator nodes are partitioned into the same fusion node set, where there is a logical operation relationship between the convolutional layer and its subsequent operator nodes, and the subsequent operator nodes include at least one of the following: the activation function layer, the batch normalization layer, and the pooling layer in the target neural network model.
[0101] Exemplarily, the target neural network model includes multiple convolutional layers and other layers, such as including an activation function layer, a batch normalization layer, and a pooling layer. Each layer can be used as an operator node. Since there is a logical operation relationship between the convolutional layer and its subsequent operator nodes, the convolutional layer and its subsequent operator nodes can be partitioned into the same fusion node set, thereby effectively improving the logical operation efficiency during the process of processing the data associated with the same fusion node set.
[0102] Combine Figure 8For example, the operator nodes included in the graph neural network (target neural network model) are respectively: the first convolutional layer Conv, the first activation function layer Relu, the second convolutional layer Conv, the first batch normalization layer BN, the third convolutional layer Conv, the second batch normalization layer BN, the second activation function layer Relu, the third convolutional layer Conv, and the pooling layer pooling. The first convolutional layer Conv and the first activation function layer Relu can be determined as the same fusion node set to form the fusion node set Conv+Relu. The second convolutional layer Conv and the first batch normalization layer BN are determined as the same fusion node set to form the fusion node set Conv+BN. The third convolutional layer Conv, the second batch normalization layer BN, and the second activation function layer Relu are determined as the same fusion node set to form the fusion node set Conv+BN+Relu. The third convolutional layer Conv and the pooling layer pooling are determined as the same fusion node set to form the fusion node set Conv+pooling.
[0103] In one embodiment of the present application, the splitting and combining of the operator nodes based on the processing speed of the operator nodes by the processor includes:
[0104] Determine the processing speeds when the memories in multiple processors process the operator nodes respectively;
[0105] Divide the operator nodes into the processing part of the target neural network model data corresponding to the processor of the memory with the highest processing speed.
[0106] Exemplarily, the processing speeds when the memories in multiple processors process the data of the operator nodes are different. For example, the read and write data speeds are different. The processing speeds of the memories of some processors are relatively high, while the processing speeds of the memories of other processors are relatively low. When dividing the operator nodes, the operator nodes can be preferentially divided into the pre-processing part, the inference part, or the post-processing part of the target neural network model data corresponding to the processor of the memory with the highest processing speed. For example, an NPU usually has a certain amount of SRAM, and its read and write speed is much faster than that of the DDR used by other processors. Therefore, when dividing the operator nodes, the division can be performed at the operator node or operator node set that writes data to the DDR. This makes the processing times of multiple processors balanced or nearly the same.
[0107] Combined with Figure 9For example, an NPU usually has a certain amount of SRAM, whose read and write speeds are much faster than those of the DDR used by other processors. For relatively small data processing tasks (such as small feature maps), the internal SRAM can be directly used for caching without writing to the DDR, saving unnecessary memory bandwidth. For example, multiple operator nodes are respectively: operator node Conv+Relu, operator node Conv+BN, operator node Conv+BN+Relu, operator node Conv+pooling. The operator node Conv+Relu and the operator node Conv+BN can be divided into the first group, and the operator node Conv+BN+Relu and the operator node Conv+pooling can be divided into the second group. Ensure that the operator nodes in the first group can be written into the SRAM instead of the DDR, improving the processing efficiency.
[0108] In an embodiment of the present application, as Figure 4 shown and in combination with Figure 7 , the method further includes the following steps:
[0109] S500, when the set of fusion nodes is a model subgraph of the target neural network model, obtain each of the model subgraphs compiled on the development machine through the board side, where the heterogeneous chip is disposed on the board side.
[0110] S600, determine the inference performance data of the processor for processing the model subgraph;
[0111] S700, based on the inference performance data, determine whether it is necessary to re-segment and combine the model subgraph.
[0112] Exemplarily, the target neural network model may include multiple model subgraphs, where the set of fusion nodes is the model subgraph. For example, the model subgraphs may be sub gragh1, sub gragh2, and sub gragh3. Among them, sub gragh1 corresponds to the VDSP, sub gragh2 corresponds to the NPU, and sub gragh3 corresponds to the CPU.
[0113] In one embodiment, the heterogeneous chip is installed on the development board, and the development board can interact with the development machine through Remote Procedure Call (RPC). The target neural network model can be compiled on the development machine, and the development board processes the associated data. During this process, the development board obtains each model sub-graph compiled on the development machine, uses multiple processors of the heterogeneous chip to process the data associated with the model sub-graph, and determines the inference performance data of the model sub-graph according to the processing results. If the inference performance data meets the preset requirements, it can be determined that the method of splitting and combining the operator nodes in the model sub-graph meets the requirements; otherwise, the operator nodes in the model sub-graph can be re-split and combined to improve the efficiency of data processing.
[0114] In one embodiment of the present application, the use of multiple different types of the processors to cooperate in processing the data associated with the adjusted pre-processing part, the inference part, and the post-processing part respectively includes:
[0115] Processing the data associated with the pre-processing part through the vision digital signal processor of the heterogeneous chip;
[0116] Processing the data associated with the inference part through the neural network processor of the heterogeneous chip;
[0117] Processing the data associated with the post-processing part through the central processing unit of the heterogeneous chip, wherein the processing times of the vision digital signal processor, the neural network processor, and the central processing unit at least partially overlap.
[0118] Exemplarily, the heterogeneous chip includes multiple different types of processors, which are respectively a vision digital signal processor (VDSP), a neural processing unit (NPU), and a central processing unit (CPU). The vision digital signal processor is used to process the data associated with the pre-processing part; the neural network processor is used to process the data associated with the inference part; the central processing unit is used to process the data associated with the post-processing part. The vision digital signal processor, the neural network processor, and the central processing unit can work in parallel so that the processing times at least partially overlap, enabling load balancing among the vision digital signal processor, the neural network processor, and the central processing unit, thereby improving the data processing efficiency.
[0119] In one embodiment of the present application, asFigure 5 As shown in the figure, the method further includes the following steps:
[0120] S800, during the process of using multiple processors of the heterogeneous chip to co-process the target neural network model, determine whether the multiple processors are load-balanced.
[0121] Exemplarily, on the one hand, during the process of using multiple processors of the heterogeneous chip to co-process the target neural network model, when the operator nodes of each processing part in the target neural network model are not segmented and combined, the load of the processors may be unbalanced. On the other hand, after the operator nodes of each processing part in the target neural network model are segmented and combined using the partitioning strategy, the data processed by the processors changes, which may cause the load of each processor to change. In this embodiment, at a preset time, it is possible to determine whether the multiple processors are load-balanced by detecting the running states of each processor.
[0122] S900, in the case where it is determined that the processors are not load-balanced, re-segment and combine the operator nodes.
[0123] Exemplarily, if the processors are not load-balanced, on the one hand, it indicates that it may be due to the fact that the operator nodes of each processing part in the target neural network model are not segmented and combined, resulting in the lack of load balancing when the processors process the associated data of the corresponding processing parts. On the other hand, it may also be that the way of segmenting and combining the operator nodes of each processing part in the target neural network model is not adapted to the current running state of the heterogeneous chip. Thus, the operator nodes can be re-segmented and combined to achieve load balancing when the processors process the data associated with each processing part of the target neural network model.
[0124] The embodiment of the present application also provides a heterogeneous chip, as Figure 10 shown, including:
[0125] A determination module configured to determine the target neural network model corresponding to the heterogeneous chip, where the target neural network model has multiple operator nodes, the target neural network model includes a pre-processing part, an inference part, and a post-processing part, and the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the pre-processing part, the inference part, and the post-processing part.
[0126] Exemplarily, the heterogeneous chip can be disposed in devices such as terminals and development boards, and can be used to process data related to its corresponding target neural network model. For example, when the image data input to the target neural network model needs to be processed, the heterogeneous chip related to the target neural network can process the image data.
[0127] In this embodiment, the determination module can determine the identity of the target neural network based on the identifier of the target neural network model. Thus, the working mode of the heterogeneous chip itself and / or the calling mode of the heterogeneous chip to the target neural network model can be determined. Of course, if the target neural network model is adjusted, the working mode of the heterogeneous chip and / or the calling mode to the target neural network model can also be adjusted accordingly.
[0128] The target neural network model has multiple operator nodes. The operator node is a logical computing unit. The operator node can include the operation layer or computing unit of the target neural network model, and each operator node can perform logical operations. The target neural network model includes a preprocess part, an inference part, and a postprocess part. Each processing part can include at least one operator node and other types of nodes, and can use the heterogeneous chip to perform data processing in their respective corresponding stages.
[0129] The heterogeneous chip includes multiple different types of processors, such as it can include VDSP (Vision Digital Signal Processor), NPU (Neural Processing Unit), and CPU (Central Processing Unit). The different types of each processor make the heterogeneous chip have heterogeneous characteristics. In this embodiment, the multiple processors respectively correspond to the preprocess part, the inference part, and the postprocess part. So that each processor can process the processing part corresponding to it. For example, the VDSP corresponds to the preprocess part of the target neural network model and can process the data associated with the preprocess part. The NPU corresponds to the inference part of the target neural network model and is used to process the data associated with the inference part. The CPU corresponds to the postprocess part of the target neural network model and is used to process the data associated with the postprocess part.
[0130] A partitioning module, configured to use a partitioning strategy to split and combine the operator nodes to form corresponding sets of fused nodes; and assign the sets of fused nodes to the preprocess part, the inference part, and the postprocess part respectively.
[0131] Exemplarily, operator nodes can be combined so that the same processor in the heterogeneous chip can be utilized as a whole to process relevant data. Operator nodes can be split and combined to form multiple sets of fused nodes, and each set of fused nodes can include at least one operator node and other nodes. A processor in the heterogeneous chip can process the relevant data of the set of fused nodes corresponding to it. For example, the first set of fused nodes is used to process the first input data, and the vision digital signal processor corresponds to the first set of fused nodes, so that the vision digital signal processor can be used to process the first input data.
[0132] In this embodiment, the partitioning module uses a partitioning strategy to split and then combine all the operator nodes in the target neural network model to form multiple sets of fused nodes. The partitioning strategy can be set with the goal of balancing the processing time and / or efficiency of multiple processors. In one embodiment, the partitioning strategy can be preset according to comprehensive factors such as the structure and features of the target neural network model, the characteristics of operator nodes, and / or the characteristics of each different type of processor in the heterogeneous chip.
[0133] In this embodiment, the partitioning module can assign the set of fused nodes with the original preprocessing part as the main body to the current preprocessing part. Assign the set of fused nodes with the original inference part as the main body to the current inference part. Assign the set of fused nodes with the original postprocessing part as the main body to the current postprocessing part.
[0134] For example, the original preprocessing part includes the preprocessing operator node pre node. The original inference part includes operator nodes node 1, operator node 2, operator node 3, …, operator node n - 2, operator node n - 1, operator node n. The original postprocessing part includes the postprocessing operator node post node. After the partitioning module partitions and combines each operator node based on the partitioning strategy, the first set of fused nodes formed includes the preprocessing operator node prenode and operator node 1. The second set of fused nodes includes operator nodes node 2, operator node 3, …, operator node n - 2. The third set of fused nodes includes operator nodes node n - 1, operator node n, and the postprocessing operator node post node. The first set of fused nodes is determined as the current preprocessing part, the second set of fused nodes is determined as the current inference part, and the third set of fused nodes is determined as the current postprocessing part.
[0135] A processing module configured to use multiple processors of different types to cooperate in processing the data respectively associated with the adjusted pre - processing part, the inference part, and the post - processing part.
[0136] Exemplarily, the processing module can use multiple processors of different types in a heterogeneous chip to process the data respectively associated with their corresponding processing parts. For example, a vision digital signal processor corresponds to the pre - processing part and can be used to process the data associated with the pre - processing part. A neural network processor corresponds to the inference part and can be used to process the data associated with the inference part. A central processing unit corresponds to the post - processing part and can be used to process the data associated with the post - processing part.
[0137] During the process of processing data, multiple processors can cooperate with each other. For example, multiple processors can run in parallel and process their respective tasks simultaneously, thus saving processing time. In addition, since the partitioning strategy can be set with the goal of balancing the processing time and / or efficiency of multiple processors. Therefore, the time for processing the data associated with the current pre - processing part, inference part, and post - processing part is the same or similar. Thus, the processing time of data is saved during the process of invoking the target neural network model, and the processing efficiency is improved.
[0138] An embodiment of the present application also provides a storage medium carrying one or more computer programs, and when the one or more computer programs are executed by a processor, the steps of the method described above are implemented.
[0139] An embodiment of the present application also provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the method described above are implemented.
[0140] In various embodiments of the present application, the magnitude of the sequence numbers of the above - mentioned processes does not mean the order of execution is prior or subsequent. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
[0141] Those of ordinary skill in the art can realize that the various illustrative logical blocks (ILBs) and steps described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.
[0142] In several embodiments provided by this application, it should be understood that the disclosed methods, heterogeneous chips, storage media, and computer program products can be implemented in other ways. For example, the above-described embodiments are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.
[0143] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0144] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.
[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc.
[0146] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.
Claims
1. A heterogeneous chip model calling method, characterized in that, Including: Determine a target neural network model corresponding to a heterogeneous chip, where the target neural network model has multiple operator nodes, the target neural network model includes a pre-processing part, an inference part, and a post-processing part, and the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the pre-processing part, the inference part, and the post-processing part; Using a partitioning strategy, split and combine the operator nodes to form corresponding sets of fused nodes; Assign the sets of fused nodes to the pre-processing part, the inference part, and the post-processing part respectively; Use the multiple different types of processors to cooperate in processing the data respectively associated with the adjusted pre-processing part, inference part, and post-processing part.
2. The heterogeneous chip model calling method according to claim 1, wherein The pre-processing part includes pre-processing operator nodes, the post-processing part includes post-processing operator nodes, and the using a partitioning strategy to split and combine the operator nodes to form corresponding sets of fused nodes includes: Partition the operator nodes in the inference part to form a first part of nodes, a second part of nodes, and a third part of nodes; Combine the first part of nodes with at least one operator node in the pre-processing part to form a corresponding first set of fused nodes; Based on the second part of nodes, determine a corresponding second set of fused nodes; Combine the third part of nodes with the post-processing nodes to form a corresponding third set of fused nodes.
3. The heterogeneous chip model calling method according to claim 2, wherein, The assigning the sets of fused nodes to the pre-processing part, the inference part, and the post-processing part respectively includes: Assign the first set of fused nodes to the pre-processing part, where the pre-processing part corresponds to the vision digital signal processor of the heterogeneous chip; Assign the second set of fused nodes to the inference part, where the inference part corresponds to the neural network processor of the heterogeneous chip; Assign the third set of fused nodes to the post-processing part, where the post-processing part corresponds to the central processing unit of the heterogeneous chip.
4. The heterogeneous chip model calling method according to claim 1, wherein The partitioning strategy includes: Based on the logical operation relationship between the operator nodes, split and combine the operator nodes; and, Based on the processing speed of the operator nodes by the processors, split and combine the operator nodes.
5. The heterogeneous chip model calling method according to claim 4, wherein The based on the logical operation relationship between the operator nodes to split and combine the operator nodes includes: Use the convolutional layer in the target neural network model as a splitting identifier; Based on the splitting identifier, partition the convolutional layer and its subsequent operator nodes into the same set of fused nodes, where there is a logical operation relationship between the convolutional layer and its subsequent operator nodes, and the subsequent operator nodes include at least one of the following: the activation function layer, the batch normalization layer, and the pooling layer in the target neural network model.
6. The heterogeneous chip model calling method according to claim 4, wherein The based on the processing speed of the operator nodes by the processors to split and combine the operator nodes includes: Determine the processing speeds when the memories in the multiple processors process the operator nodes respectively; Partition the operator nodes into the processing part of the target neural network model data corresponding to the processor with the highest processing speed.
7. The heterogeneous chip model calling method according to claim 1, wherein The method further includes: When the set of fusion nodes is a model subgraph of the target neural network model, obtain each of the model subgraphs compiled on the development machine through the board side, where the heterogeneous chip is provided on the board side; Determine the inference performance data of the processor for processing the model subgraph; Based on the inference performance data, determine whether it is necessary to re-partition and combine the model subgraph.
8. The heterogeneous chip model calling method according to claim 1, wherein The collaborative processing of the data associated with the adjusted pre-processing part, the inference part, and the post-processing part by using multiple different types of the processors includes: Process the data associated with the pre-processing part through the vision digital signal processor of the heterogeneous chip; Process the data associated with the inference part through the neural network processor of the heterogeneous chip; Process the data associated with the post-processing part through the central processor of the heterogeneous chip, where the processing times of the vision digital signal processor, the neural network processor, and the central processor at least partially overlap.
9. The heterogeneous chip model calling method according to claim 1, wherein The method further includes: During the collaborative processing of the target neural network model by using multiple processors of the heterogeneous chip, determine whether the multiple processors are load-balanced; When it is determined that the processor is not load-balanced, re-partition and combine the operator nodes.
10. A heterogeneous chip, characterized in that, Includes: A determination module configured to determine a target neural network model corresponding to a heterogeneous chip, where the target neural network model has multiple operator nodes, the target neural network model includes a pre-processing part, an inference part, and a post-processing part, the heterogeneous chip includes multiple different types of processors, and the multiple processors respectively correspond to the pre-processing part, the inference part, and the post-processing part; A partitioning module configured to use a partitioning strategy to partition and combine the operator nodes to form a corresponding set of fusion nodes; and assign the set of fusion nodes to the pre-processing part, the inference part, and the post-processing part respectively; A processing module configured to use multiple different types of the processors to collaboratively process the data associated with the adjusted pre-processing part, the inference part, and the post-processing part respectively.