Pattern recognition method and device, electronic device, computer readable medium

Abstract

The present disclosure provides a pattern recognition method and device, an electronic device, and a computer readable medium. The method comprises: generating a first sequence according to an execution order of a plurality of operators in a specified computation graph and characteristic information of the plurality of operators; labeling a fusion attribute of the operators according to the first sequence to obtain a second sequence; determining a target subgraph to be fused from a plurality of subgraphs of the specified computation graph according to the second sequence; and in a case where a target fusion mode not belonging to a fusion mode set is included in a fusion mode of the target subgraph, updating the fusion mode set based on the target fusion mode. According to the embodiments of the present disclosure, the subgraphs that can be fused can be efficiently recognized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a pattern recognition method and apparatus, electronic device, and computer-readable medium. Background Technology

[0002] A computational graph is a directed graph used to describe functions and has been widely used in various deep learning frameworks (such as TensorFlow and ONNX). Typically, a computational graph needs to be compiled to generate an instruction stream that can run on hardware. During the compilation phase, once the computational graph enters the compiler, it is divided into several subgraphs, each containing one or more operators.

[0003] In related technologies, after the compiler divides the subgraph, it can also manually merge the operators in some subgraphs and execute the corresponding instructions as an execution unit, so as to reduce on-chip storage and speed up execution.

[0004] However, due to the large number of operators, the complex and diverse ways to combine operators, and the multiple choices of operator parameters, it is impossible to fully determine the subgraphs that can be fused by operators through manual methods, and the efficiency of manual determination is relatively low. Summary of the Invention

[0005] This disclosure provides a pattern recognition method and apparatus, an electronic device, and a computer-readable medium.

[0006] In a first aspect, this disclosure provides a pattern recognition method, which includes: generating a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators, wherein each item of the first sequence corresponds to the feature vector of each operator; labeling the fusion attributes of the operators based on the first sequence to obtain a second sequence, wherein the fusion attributes are used to characterize whether the operator has the characteristic of fusion with adjacent operators; determining a target subgraph to be fused from multiple subgraphs of the specified computation graph based on the second sequence; and updating the fusion mode set based on the target fusion mode if the fusion mode of the target subgraph includes a target fusion mode that does not belong to the fusion mode set, wherein the fusion mode set includes at least one fusion mode.

[0007] Secondly, this disclosure provides a pattern recognition device, comprising: a generation module configured to generate a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators, wherein each item of the first sequence corresponds to the feature vector of each operator; a labeling module configured to label the fusion attributes of the operators based on the first sequence to obtain a second sequence, wherein the fusion attributes are used to characterize whether the operator has the characteristic of fusion with adjacent operators; a determination module configured to determine a target subgraph to be fused from multiple subgraphs of the specified computation graph based on the second sequence; and an update module configured to update the fusion mode set based on the target fusion mode if the fusion mode of the target subgraph includes a target fusion mode that does not belong to the fusion mode set, wherein the fusion mode set includes at least one fusion mode.

[0008] Thirdly, this disclosure provides a compiler comprising: at least one pattern recognition device for implementing the pattern recognition method described above.

[0009] Fourthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the pattern recognition method described above.

[0010] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described pattern recognition method.

[0011] The embodiments provided in this disclosure can obtain a first sequence composed of operator feature vectors based on the execution order and feature information of multiple operators in a computation graph, and determine the fusion attribute of the operators based on the first sequence. This allows for efficient and convenient determination of the target subgraph to be fused based on the fusion attribute. When there is a target fusion mode that does not belong to the fusion mode set among the fusion modes of the determined target subgraph, the fusion mode set is updated using the target fusion mode to obtain a more comprehensive and complete fusion mode set. This enables a more comprehensive identification of subgraphs in the computation graph that can be fused, thereby reducing the occupation of on-chip storage resources and speeding up the execution speed when performing computation tasks based on the fused subgraph.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0014] Figure 1 A flowchart of a pattern recognition method provided in an embodiment of this disclosure;

[0015] Figure 2 A flowchart of a pattern recognition method provided in an embodiment of this disclosure;

[0016] Figure 3 A schematic diagram illustrating the training process of a vector acquisition model provided in an embodiment of this disclosure;

[0017] Figure 4 A schematic diagram illustrating the training process of a labeled model provided in an embodiment of this disclosure;

[0018] Figure 5 A schematic diagram illustrating the processing procedure of a pattern recognition method provided in this embodiment of the disclosure;

[0019] Figure 6 A schematic diagram illustrating a pattern recognition method provided in an embodiment of this disclosure;

[0020] Figure 7 A block diagram of a pattern recognition device provided in an embodiment of this disclosure;

[0021] Figure 8 A block diagram of a compiler provided for embodiments of this disclosure;

[0022] Figure 9 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0025] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0028] Machine learning (ML) is a multidisciplinary field involving probability theory, statistical approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Deep learning, as a branch of machine learning's neural network algorithms, is motivated by the desire to build and simulate neural networks that mimic the human brain's analytical learning processes.

[0029] In practical applications, computation graphs can be used to represent neural networks corresponding to deep learning models. In a computation graph, computation is represented as a directed graph, defining the flow of data, the methods of data computation, and the relationships between various computations. Typically, a computation graph includes two basic elements: nodes and directed edges. Nodes include variable nodes and operator nodes, each variable node corresponding to a tensor, which can be a scalar, vector, or matrix. Operator nodes represent mathematical operations (e.g., convolution / deconvolution operators, pooling operators, activation operators, classifier operators, fully connected operators, etc.). Directed edges represent dependencies between two nodes (e.g., two operators connected sequentially, where the output of the previous operator serves as the input of the adjacent subsequent operator). Through the combined action of multiple internal operators, the computation graph as a whole can be used to implement relatively complete algorithms with specific functions, including but not limited to various deep learning algorithms.

[0030] In related technologies, a computation graph can be divided into multiple subgraphs, and each subgraph can be assigned to a computation unit for computation, thereby obtaining the computation result of the computation graph through distributed computing. Considering that a many-core system includes multiple processing cores (cores) connected in a preset manner, and each processing core is the smallest unit that can be independently scheduled and has independent computing capabilities, possessing the ability to independently execute computation tasks, a many-core system can be used to perform distributed processing on each subgraph of the computation graph, thereby improving processing efficiency.

[0031] It should be understood that when a many-core system processes a computation graph, it needs to allocate different resources to different subgraphs, or "map" each subgraph to different resources in the many-core network. That is, when the many-core system is running, each subgraph of the computation graph should use (occupy) a specific set of resources in the many-core system without duplication.

[0032] The resources of a many-core system can include runtime resources (i.e., hardware resources). For example, the resources of a many-core system include processing cores, threads, on-chip storage space, etc. After allocating target runtime resources to each subgraph of the computation graph, when a subgraph runs in a many-core system, the processing cores, threads, and on-chip storage space in its target runtime resources can only be occupied by that subgraph.

[0033] The resources of a many-core system can also include time resources. The target time resource allocated to a subgraph is equivalent to specifying when the subgraph should be processed by the many-core system. That is, multiple subgraphs can be processed in the many-core system using a "time-division multiplexing" approach. In other words, the same resource cannot be allocated to multiple different subgraphs simultaneously; that is, any runtime resource, under the same time resource (at the same time), can at most be the target resource for one subgraph. However, the same runtime resource can be allocated to different subgraphs at different times.

[0034] It should be noted that before many-core systems process the computation graph, the computation graph typically needs to be compiled by a compiler to generate an instruction stream that can run on hardware. In related technologies, after the compiler divides the graph into subgraphs, it can also manually fuse operators in some subgraphs and execute the corresponding instructions as an execution unit to reduce on-chip storage and speed up execution. However, due to the large number of operator types, the complex and diverse ways of combining operators, and the multiple choices of operator parameters, it is impossible to comprehensively determine the subgraphs that can be fused by manual methods, and the determination efficiency is relatively low.

[0035] In view of this, embodiments of this disclosure provide a pattern recognition method, apparatus, electronic device, and medium. Based on the execution order and feature information of multiple operators in a computation graph, a first sequence composed of operator feature vectors is obtained. The fusion attribute of the operators is determined based on the first sequence, so as to efficiently and conveniently determine the target subgraph to be fused based on the fusion attribute. When there is a target fusion mode that does not belong to the fusion mode set among the fusion modes of the determined target subgraph, the fusion mode set is updated using the target fusion mode to obtain a more comprehensive and complete set of target fusion modes. This allows for a more comprehensive identification of subgraphs in the computation graph that can be fused by operators. When performing computational tasks based on the fused subgraph, the occupation of on-chip storage resources is reduced, and the execution speed is accelerated.

[0036] Figure 1 A flowchart illustrating a pattern recognition method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes:

[0037] Step S101: Generate a first sequence based on the execution order of multiple operators in the specified computation graph and the feature information of the multiple operators.

[0038] The specified computation graph refers to the computation graph in the current compilation process, where the connections between nodes may include one or more subgraphs. The execution order is used to characterize the order in which operators are operated when the specified computation graph is executed.

[0039] The feature information of an operator includes information that reflects certain properties of the operator. Each term in the first sequence corresponds to the feature vector of each operator, and the feature vector is a vector obtained based on the feature information of the operator.

[0040] Step S102: Label the fusion attributes of the operators according to the first sequence to obtain the second sequence.

[0041] The fusion attribute is used to characterize whether an operator has the property of fusion with adjacent operators. Labeling the fusion attribute aims to determine its labeled value, so that the operator's fusion attribute can be determined based on the labeled value. The second sequence is a sequence composed of the operator's fusion attribute labeled values.

[0042] In some embodiments, preset annotation values ​​are used to annotate the fusion attributes of each operator in the first sequence to obtain the second sequence.

[0043] Step S103: Based on the second sequence, determine the target subgraph to be fused from multiple subgraphs of the specified computation graph.

[0044] The target subgraph is a subgraph in the specified computation graph that can be fused with operators.

[0045] Step S104: If the target subgraph's fusion patterns include target fusion patterns that do not belong to the fusion pattern set, update the fusion pattern set based on the target fusion patterns.

[0046] The set of fusion patterns includes at least one fusion pattern, which indicates the patterns of operators that can be fused, including but not limited to operator type and operator connection relationship. The target fusion pattern is a pattern that does not belong to the set of fusion patterns. In other words, the target fusion pattern is different from all existing fusion patterns in the set of fusion patterns.

[0047] It should be noted that if all the fusion patterns of the target subgraph belong to the target fusion pattern set, it means that no new fusion patterns have been found, and therefore, there is no need to update the fusion pattern set.

[0048] It should also be noted that updating the set of fusion patterns is an iterative process. Typically, multiple pattern recognition processes are required before a relatively comprehensive set of fusion patterns can be identified, and the fusion pattern set built based on the identified patterns can be considered a relatively complete set.

[0049] In this embodiment of the disclosure, a first sequence composed of operator feature vectors can be obtained based on the execution order and feature information of multiple operators in the computation graph. The fusion attribute of the operator is determined based on the first sequence, so as to efficiently and conveniently determine the target subgraph to be fused based on the fusion attribute. When there is a target fusion mode that does not belong to the fusion mode set in the fusion mode of the target subgraph, the fusion mode set is updated using the target fusion mode to obtain a more comprehensive and complete target fusion mode set. This allows for a more comprehensive identification of subgraphs in the computation graph that can be fused by operators. When performing computation tasks based on the subgraph after operator fusion, the occupation of on-chip storage resources is reduced and the execution speed is accelerated.

[0050] It should be noted that, in some embodiments, before step S101, the pattern recognition method according to the embodiments of this disclosure may further include: determining the execution order of multiple operators in a specified computation graph.

[0051] In some embodiments, the execution order of operators can be determined based on the connection relationships between operators in a specified computation graph.

[0052] It should be noted that the above methods for determining the execution order are merely illustrative examples, and the specific methods for determining the execution order in this disclosure are not limited.

[0053] Figure 2 A flowchart illustrating a pattern recognition method provided in an embodiment of this disclosure. (Refer to...) Figure 2 The method includes:

[0054] Step S201: Generate a first sequence based on the execution order of multiple operators in the specified computation graph and the feature information of the multiple operators.

[0055] In some embodiments, the feature information of an operator includes at least one of operator structure, operator execution mechanism, operator memory access characteristics, and operator computational characteristics. By extracting features from one or more of the operator's feature information, a corresponding feature vector can be obtained. Specifically, the operator structure characterizes the internal components of the operator and the relationships between these components (e.g., the dimensionality of a two-dimensional convolution operator (Conv2d)); the operator execution mechanism includes the principles followed by the operator during execution, which may include the quantitative relationship between the operator's input and output data (e.g., an activation operator corresponds to one input and one output, while a convolution operator corresponds to multiple inputs and multiple outputs); the operator memory access characteristics characterize the operator's memory access behavior, involving data reuse (locality) and the decoupling of data access (e.g., in a convolution operator, the decoupling of input data and weights allows for separate memory access); and the operator computational characteristics characterize the operator's operational features (e.g., the operator can perform matrix operations, vector operations, scalar operations, etc.).

[0056] In some embodiments, during the compilation of a specified computation graph, firstly, multiple operators in the specified computation graph are arranged according to their execution order to obtain a third sequence (i.e., execution sequence); secondly, feature vectors of operators are generated based on the feature information of the operators; finally, the feature vectors of multiple operators are arranged according to the order of the operators in the third sequence to obtain a first sequence (i.e., feature vector sequence).

[0057] In some embodiments, the feature vector of the operator can be generated by a preset vector acquisition model, that is, the vector acquisition model generates the feature vector of the operator based on the feature information of the operator. The vector acquisition model is a trained model, and its training method can be found in [reference needed]. Figure 3 The training process of the vector acquisition model is shown.

[0058] Step S202: According to the first sequence, the fusion attributes of the operator are labeled using preset label values ​​to obtain the label values ​​of the operator.

[0059] Among them, the preset annotation value is used to annotate the fusion attributes of the operator.

[0060] In some embodiments, the preset label value of the fusion attribute includes a first value and a second value. The first value indicates that the corresponding operator has the characteristic of fusion with adjacent operators, and the second value indicates that the corresponding operator does not have the characteristic of fusion with adjacent operators. For example, if the label value of the fusion attribute of an operator is the first value, it means that the operator can be fused with adjacent operators, and its fusion attribute is fusionable; conversely, if the label value of the fusion attribute of an operator is the second value, it means that the operator cannot be fused with adjacent operators, and its fusion attribute is non-fusionable. It should be noted that the above label values ​​are only illustrative examples. In practical applications, the type and number of label values ​​can be set according to requirements, and this disclosure does not impose any limitations on this.

[0061] Step S203: Obtain the second sequence based on the labeled values ​​of multiple operators.

[0062] In some embodiments, if the specified computation graph includes three operators, their execution order is third operator - first operator - second operator, and the preset label values ​​include "0" and "1". When the label value of the third operator is 1, and the label values ​​of the first and second operators are 0, the corresponding second sequence is determined to be 100.

[0063] It should be noted that in some embodiments, steps S202-S203 can be implemented using a preset annotation model. That is, the first sequence is input into the annotation model, which determines the annotation values ​​for each operator. A second sequence is obtained based on the operator annotation values, and this second sequence is output as output data. Thus, the fusion properties of the operators can be determined based on the second sequence. The annotation model is a trained model based on a Conditional Random Field (CRF).

[0064] In some possible implementations, the training process of the annotation model includes: constructing an initial annotation model based on a conditional random field; inputting a first training sequence into the initial annotation model to obtain a predicted annotation sequence, which is a sequence obtained by the initial annotation model based on the first training sequence for predicting operator fusion attributes; iteratively updating the annotation model based on the predicted annotation sequence and the second training sequence to obtain a trained annotation model, which has the function of determining a second sequence corresponding to the first sequence based on the first sequence. Here, the first training sequence is a sequence obtained by arranging operator features according to the operator execution order, and the second training sequence is a sequence composed of operator fusion attribute annotation values ​​(operators are arranged in execution order).

[0065] After obtaining the trained annotation model, the annotation model can be used to obtain the second sequence in step S102 or steps S202-S203.

[0066] In some possible implementations, after obtaining the predicted annotation sequence and before iteratively updating the annotation model based on the predicted annotation sequence and the second training sequence, the following steps are also included:

[0067] The predicted annotation sequence is reviewed to obtain the review result; based on the review result, the predicted annotation sequence is updated to obtain the updated predicted annotation sequence. The updated predicted annotation sequence and the second training sequence are used to iteratively update the annotation model to obtain the trained annotation model.

[0068] The review of predicted annotation sequences involves assessing their accuracy and reasonableness based on experience, statistical data, or one or more pre-defined conditions, and obtaining corresponding review results (i.e., identifying incorrectly fused or unfusionable predicted annotation sequences). Based on the review results, inaccurate or unreasonable predicted annotation sequences are removed or modified, thus updating the predicted annotation sequences. When the updated predicted annotation sequences and a second training sequence are used to iteratively update the annotation model, a better-performing annotation model can be obtained, thereby improving model training efficiency and effectiveness.

[0069] It should be noted that the review of the predicted labeled sequences can be performed by corresponding smart hardware devices or manually, and this disclosure does not limit this.

[0070] It should also be noted that, in practical applications, there are various variations and improvements of conditional random fields (CRFs). The annotation model in this embodiment can be established based on any type of CRF and used to output the second sequence. This embodiment does not limit the specific type of CRF.

[0071] Step S204: Based on the second sequence, determine the target subgraph to be fused from multiple subgraphs of the specified computation graph.

[0072] In some embodiments, firstly, the operator corresponding to the first value in the second sequence is determined as the target operator; secondly, the subgraphs containing the target operator in the multiple subgraphs are determined as the target subgraphs.

[0073] It should be noted that when a subgraph containing a target operator is identified as a target subgraph, two types of target subgraphs will be generated. In the first type of target subgraph, all operators are target operators to be fused. In the second type of target subgraph, in addition to the target operators to be fused, the target subgraph also includes at least one operator that cannot be fused or does not need to be fused. In practical applications, the type of target subgraph to be identified can be determined according to the specific recognition requirements. For example, when both types of target subgraphs need to be identified, the subgraph containing the target operator can be directly identified as the target subgraph. Alternatively, when only the first type of target subgraph needs to be identified, the subgraphs containing the target operator are first selected as candidate target subgraphs, and then the first type of target subgraph is further selected from the candidate target subgraphs.

[0074] Step S205: If the target subgraph's fusion patterns include target fusion patterns that do not belong to the fusion pattern set, update the fusion pattern set based on the target fusion patterns.

[0075] In some embodiments, if the target subgraph's fusion patterns include a target fusion pattern that does not belong to the fusion pattern set, it can be determined that the target fusion pattern is different from all existing fusion patterns in the fusion pattern set, i.e., the target fusion pattern belongs to a new fusion pattern. To make the fusion pattern set more complete, the target fusion pattern is included in the fusion pattern set, thereby updating the fusion pattern set.

[0076] In this embodiment, by accurately labeling the fusion attributes of the operator using preset label values, the target subgraph can be identified more conveniently and accurately. Furthermore, the existing fusion pattern set can be updated based on the fusion mode of the target subgraph, resulting in a more comprehensive and complete fusion pattern set. Moreover, in practical applications, the above labeling process can be implemented using a labeling model based on conditional random fields, making the labeling operation of fusion attributes even more convenient.

[0077] It should be noted that, in some embodiments, after step S205, the pattern recognition method according to the embodiments of this disclosure may further include: determining a target subgraph in the computational graph to be identified based on a target fusion pattern set, wherein the target fusion pattern set is obtained by updating the fusion pattern set multiple times; fusing at least two operators in the target subgraph to obtain a fusion operator; obtaining an execution strategy for the target subgraph, wherein the execution strategy includes at least an execution method for the fusion operator (when there are operators in the target subgraph that cannot be fused or are not fused, the execution strategy also includes an execution method for operators that cannot be fused or are not fused); generating instructions corresponding to the target subgraph based on the execution strategy of the target subgraph; and sending the instructions corresponding to the target subgraph to a designated processing core so that the designated processing core can execute the computational task corresponding to the target subgraph according to the instructions.

[0078] In formulating the execution strategy, the target subgraph after operator fusion can be used as an execution unit. Based on this, some intermediate data no longer needs to be stored, which can effectively reduce the number of "store-retrieval" operations, speed up the execution of the target subgraph, and improve the processing power of many-core systems or many-core chips.

[0079] Figure 3 This is a schematic diagram illustrating the training process of a vector acquisition model provided in an embodiment of this disclosure. (Refer to...) Figure 3 The training process for the vector acquisition model includes the following steps:

[0080] Step S301: Construct the initial vector acquisition model.

[0081] Among them, the vector acquisition model is used to obtain the corresponding feature vectors based on the feature information of the operator.

[0082] In some embodiments, an initial vector acquisition model is established based on neural network technology, including but not limited to one or more of the following: Recursive Neural Network (RNN) model, Convolutional Neural Network (CNN) model, and Deep Belief Neural Network (DBNN) model (for example, an RNN model can be superimposed on a CNN model to construct a vector acquisition model).

[0083] It should be noted that the above examples of vector acquisition models are merely illustrative, and the embodiments disclosed herein do not limit the construction method of the vector acquisition model.

[0084] Since the initial vector acquisition model is an untrained, raw model, the feature vectors obtained using the initial vector acquisition model are not very accurate. Therefore, it is necessary to obtain good model parameters through training and to iteratively update the model based on the model parameters in order to obtain a vector acquisition model with better performance.

[0085] Step S302: Obtain the first training set data.

[0086] In some embodiments, the first training set data includes, but is not limited to, the feature information of the operator and the corresponding feature vector.

[0087] Step S303: Input the first training set data into the vector to obtain the model and obtain the predicted feature vector.

[0088] Among them, the predicted feature vector is the prediction result obtained by the vector acquisition model through processing the input data.

[0089] In some embodiments, the predicted feature vector includes a first vector for characterizing independent features of the operator and a second vector for characterizing associated features of the operator with neighboring operators.

[0090] Step S304: Update the vector acquisition model based on the predicted feature vector and the actual feature vector.

[0091] Step S305: If the preset first iteration stopping condition is met, stop training and obtain the final vector acquisition model.

[0092] The first iteration stopping condition is used to indicate the stopping vector to obtain the training process of the model.

[0093] In some embodiments, the first iteration stopping condition includes the number of iterations reaching a preset iteration threshold, and / or the feature vector accuracy reaching a preset accuracy requirement, and / or the feature vector accuracy no longer improving.

[0094] It should be noted that the above-described stopping conditions for the first iteration are merely illustrative examples, and the embodiments disclosed herein do not impose any limitations on them.

[0095] Figure 4 This is a schematic diagram illustrating the training process of a labeled model provided in an embodiment of this disclosure. (Refer to...) Figure 4 The training process of the labeled model includes the following steps:

[0096] Step S401: Construct an initial labeled model based on the conditional random field.

[0097] Conditional random fields (CRFs) are conditional probability distribution models of another set of output random variables given a set of input random variables. Their key characteristic is the assumption that the output random variables constitute a Markov random field. CRFs are commonly used in sequence labeling problems, such as part-of-speech tagging and named entity recognition.

[0098] In this embodiment of the disclosure, the annotation model based on CRF is used to annotate or identify the fusion attributes of operators. For example, the input random variable of the annotation model is a sequence of feature vectors formed by arranging the feature vectors of each operator of the specified computation graph in their execution order (i.e., the first sequence), and the output random variable is a sequence formed by the fusion attribute annotation values ​​of each operator (i.e., the second sequence).

[0099] It should be noted that since the initial annotation model uses pre-configured model parameters and is untrained, the annotation results obtained using the initial annotation model are not very accurate. Good model parameters need to be obtained through training, and the model needs to be iteratively updated based on these parameters to obtain a highly accurate annotation model. These model parameters include, but are not limited to, the weight coefficients of the feature vectors, which characterize the degree of influence of the feature vectors on the annotation results.

[0100] Step S402: Obtain the second training set data.

[0101] In some embodiments, the second training set data includes, but is not limited to, the feature vector sequence of the training computation graph (i.e., the first training sequence) and the corresponding actual annotation sequence (i.e., the second training sequence). The actual annotation sequence is a sequence obtained by annotating according to the actual fusion properties of the operators.

[0102] The training computation graph can be obtained based on different neural networks. Generally, the more types and the larger the number of neural networks, the more abundant and diverse the training set data becomes, resulting in better training performance.

[0103] Step S403: Input the second training set data into the annotation model to obtain the predicted annotation sequence.

[0104] Among them, the predicted label sequence is the label sequence obtained by the labeling model based on the input data.

[0105] Step S404: Update the annotation model based on the actual annotation sequence and the predicted annotation sequence.

[0106] When the predicted labeled sequence differs from the actual labeled sequence, it indicates that there is still room for improvement in the accuracy of the labeling model. Therefore, the model parameters are adjusted based on the difference between the actual labeled sequence and the preset labeled sequence, and the labeling model is updated using the model parameters to start a new round of training based on the updated labeling model.

[0107] Step S405: If the preset second iteration stopping condition is met, stop training and obtain the final labeled model.

[0108] The second iteration stopping condition is used to indicate the cessation of the training process of the labeled model.

[0109] In some embodiments, the second iteration stopping conditions include the number of iterations reaching a preset iteration threshold, and / or the model's annotation accuracy reaching a preset accuracy requirement, and / or the model's standard accuracy no longer improving.

[0110] It should be noted that the above-described stopping conditions for the second iteration are merely illustrative examples, and the embodiments disclosed herein do not impose any limitations on them.

[0111] It should also be noted that the training of the labeled model can be implemented through electronic devices such as servers and virtual machines, and this disclosure does not limit this. Moreover, in some embodiments, considering the limited data processing capacity of the aforementioned electronic devices, and the fact that the computation graph of a neural network is usually large in scale (in terms of the large number of operators, etc.), when acquiring the second training set data, the computation graph obtained based on the neural network can be pre-divided to obtain multiple smaller computation graphs, and then these smaller computation graphs can be used to train the labeled model according to the above method.

[0112] It should also be noted that in some embodiments, the annotation model and the vector acquisition model can be trained in a single training process. For example, firstly, the second training set data is input into the initial vector acquisition model to obtain predicted feature vectors, and these predicted feature vectors are then input into the initial annotation model. The initial annotation model annotates the fusion attributes of each operator based on the predicted feature vectors, obtaining a predicted annotation sequence. Secondly, the vector acquisition model updates itself based on the predicted and actual feature vectors; similarly, the annotation model also updates itself based on the predicted and actual annotation sequences. Finally, a new round of training is conducted based on the updated vector acquisition and annotation models until a preset first and / or second iteration stopping condition is met, at which point training stops, resulting in a trained vector acquisition and annotation model.

[0113] It should be understood that training the annotation model and the vector acquisition model in a training process is equivalent to training these two models as different layers of a neural network. Therefore, the actual feature vector used to update the vector acquisition model in step S304 is not the actual feature vector of a single layer of the network, but rather the overall actual feature vector of the entire neural network, in order to obtain good training results.

[0114] Figure 5 This is a schematic diagram illustrating the processing procedure of a pattern recognition method provided in an embodiment of this disclosure. (Refer to...) Figure 5 The processing steps of pattern recognition methods include the following:

[0115] Step S501: Generate a specified computation graph corresponding to the specified neural network model based on the specified neural network model.

[0116] Step S502: Generate the execution sequence of the specified computation graph according to the execution order of each operator in the specified computation graph.

[0117] Step S503: Input the feature information of each operator in the execution sequence into the vector acquisition model to obtain the feature vector of each operator.

[0118] Step S504: Arrange the feature vectors of each operator according to the execution sequence to obtain the first sequence.

[0119] Step S505: Input the first sequence into the annotation model to obtain the second sequence.

[0120] The second sequence is the fusion attribute annotation sequence.

[0121] Step S506: Determine the operator in the second sequence that corresponds to the first value as the target operator.

[0122] The first value is the fusion attribute label value, which indicates that the corresponding operator has the characteristic of fusion with adjacent operators. Correspondingly, the fusion attribute label value also includes a second value, which indicates that the corresponding operator does not have the characteristic of fusion with adjacent operators.

[0123] Step S507: The subgraph containing the target operator is identified as the target subgraph.

[0124] Step S508: Determine whether the fusion patterns of the target subgraph include target fusion patterns that do not belong to the fusion pattern set.

[0125] Step S509: If the target subgraph's fusion patterns include target fusion patterns that do not belong to the fusion pattern set, update the fusion pattern set based on the target fusion patterns.

[0126] It should be noted that if all the fusion patterns of the target subgraph belong to the fusion pattern set, there is no need to update the fusion pattern set.

[0127] Figure 6 This is a schematic diagram illustrating a pattern recognition method provided in an embodiment of this disclosure. The following is in conjunction with... Figure 6 Describe the processing procedure of pattern recognition methods.

[0128] Reference Figure 6 The specified computation graph includes 9 operators, of which operators 1-3 are sub-operators. Figure 1 Operators 4 and 5 are sub-operators in the table. Figure 2 The operators, operators 6 to 8 are sub-operators. Figure 3 The operator, operator 9 constitutes the sub. Figure 4Assuming the execution order of the above 9 operators is from operator 1 to operator 9, the execution sequence of the specified computation graph is operator 1-operator 2-...-operator 8-operator 9. For each operator, the feature vector of the operator is determined based on its feature information, and the feature vectors of each operator are arranged according to the execution order to obtain a first sequence with the same length as the execution sequence and a corresponding relationship (i.e., the same position in the execution sequence corresponds to the same operator). After the first sequence is input into the annotation model, the annotation model outputs a second sequence. The annotation value of the operator's fusion attribute includes two cases: "0" and "1". "0" indicates that the operator does not have the characteristic of fusion with adjacent operators, and "1" indicates that the operator has the characteristic of fusion with adjacent operators.

[0129] like Figure 6 As shown, the second sequence is 000110000. Therefore, the label value corresponding to operators 4 and 5 is "1". Thus, operators 4 and 5 are determined to be target operators. Furthermore, operators 4 and 5 are child operators. Figure 2 The operator in, therefore, the determinant Figure 2 The target subgraph is defined. Based on the operator characteristics and connectivity characteristics of operators 4 and 5, the subgraph is obtained. Figure 2 Corresponding fusion features. Comparators Figure 2 Check if the corresponding fusion features are the same as the fusion features of each fusion mode in the fusion mode set. If they are not the same, then the sub-features will be... Figure 2 The corresponding fusion pattern is added to the fusion pattern set. Conversely, if the child... Figure 2 If the corresponding fusion feature is the same as the fusion feature of a certain fusion pattern in the fusion pattern set, then the fusion pattern set is not updated.

[0130] Figure 7 This is a block diagram of a pattern recognition device provided in an embodiment of the present disclosure.

[0131] Reference Figure 7 This disclosure provides a pattern recognition device, which includes:

[0132] The generation module 701 is configured to generate a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators.

[0133] In this sequence, each term corresponds to the eigenvector of each operator.

[0134] The annotation module 702 is configured to annotate the fusion attributes of the operators based on the first sequence to obtain the second sequence.

[0135] Among them, the fusion attribute is used to characterize whether an operator has the property of fusion with neighboring operators.

[0136] The determination module 703 is configured to determine the target subgraph to be fused from multiple subgraphs of a specified computation graph according to a second sequence.

[0137] Update module 704 is configured to update the fusion pattern set based on the target fusion pattern when the fusion patterns of the target subgraph include target fusion patterns that do not belong to the fusion pattern set.

[0138] The set of fusion modes includes at least one fusion mode.

[0139] In some embodiments, the generation module 701 includes: a first sorting unit, a vector generation unit, and a second sorting unit. The first sorting unit is used to arrange multiple operators according to their execution order to obtain a third sequence; the vector generation unit is used to generate feature vectors of the operators based on their feature information; and the second sorting unit is used to arrange the feature vectors of the multiple operators according to the order of the operators in the third sequence to obtain a first sequence.

[0140] In some embodiments, the feature information includes at least one of operator structure, operator execution mechanism, operator memory access features, and operator computation features.

[0141] In some embodiments, the feature vector of an operator includes a first vector for characterizing the independent features of the operator and a second vector for characterizing the associated features of the operator with neighboring operators.

[0142] It should be noted that, in some embodiments, the feature vector of the operator can be generated by a preset vector acquisition model based on the feature information of the operator. The vector acquisition model is a trained model capable of generating feature vectors. The training process of the vector acquisition model can be found in this disclosure. Figure 4 The embodiments shown are not described again here.

[0143] In some embodiments, the annotation module 702 includes an annotation unit and an annotation sequence acquisition unit. The annotation unit is used to annotate the fusion attributes of the operator using preset annotation values ​​according to a first sequence, thereby obtaining the annotation values ​​of the operator; the annotation sequence acquisition unit is used to obtain a second sequence based on the annotation values ​​of multiple operators.

[0144] It should be noted that, in some embodiments, the annotation module obtains the second sequence by annotating the fusion attributes of the operators based on the first sequence using a preset annotation model. The annotation model is a trained model established based on a conditional random field. The training process of the annotation model can be found in this disclosure. Figure 3 The embodiments shown are not described again here.

[0145] In some embodiments, the determining module 703 includes a target operator determining unit and a target subgraph determining unit. The target operator determining unit is used to determine the operator in the second sequence corresponding to the first value as the target operator; the target subgraph determining unit is used to determine the subgraph containing the target operator as the target subgraph. The first value is a preset label value of the fusion attribute, which is used to characterize that the corresponding operator has the characteristic of fusion with adjacent operators. Correspondingly, the preset label value of the fusion attribute also includes a second value, and the second value is used to characterize that the corresponding operator does not have the characteristic of fusion with adjacent operators.

[0146] In some embodiments, the pattern recognition device according to this disclosure may further include: a target subgraph recognition unit, an operator fusion unit, an execution strategy acquisition unit, an instruction generation unit, and an instruction sending unit. Specifically, the target subgraph recognition unit is used to determine a target subgraph in the computational graph to be recognized based on a target fusion pattern set, wherein the target fusion pattern set is obtained by updating the fusion pattern set multiple times; the operator fusion unit is used to fuse at least two operators in the target subgraph to obtain a fusion operator; the execution strategy acquisition unit is used to acquire an execution strategy for the target subgraph, wherein the execution strategy includes at least the execution method of the fusion operator (when there are operators in the target subgraph that cannot be fused or are not fused, the execution strategy also includes the execution method of the operators that cannot be fused or are not fused); the instruction generation unit is used to generate an instruction corresponding to the target subgraph based on the execution strategy of the target subgraph; and the instruction sending unit is used to send the instruction corresponding to the target subgraph to a designated processing core, so that the designated processing core can execute the computational task corresponding to the target subgraph according to the instruction.

[0147] Figure 8 A block diagram of a compiler provided for embodiments of this disclosure.

[0148] Reference Figure 8 This disclosure provides a compiler 800, which includes at least one pattern recognition device 801.

[0149] The pattern recognition device 801 employs any one of the operator fusion devices in the embodiments of this disclosure and is used to implement any one of the pattern recognition methods in the embodiments of this disclosure.

[0150] In some embodiments, the compiler 800 includes a pattern recognition device 801. Specifically, the pattern recognition device 801 includes a generation module, an annotation module, a determination module, and an update module. Specifically, the generation module is used to generate a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators; the annotation module is used to annotate the fusion attributes of the operators based on the first sequence to obtain a second sequence; the determination module is used to determine the target subgraph to be fused from multiple subgraphs of the specified computation graph based on the second sequence; and the update module is used to update the fusion pattern set based on the target fusion pattern if the fusion pattern of the target subgraph includes a target fusion pattern that does not belong to the fusion pattern set.

[0151] It should be clarified that this disclosure is not limited to the specific configurations and processes described in the foregoing embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0152] Figure 9 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0153] Reference Figure 9 This disclosure provides an electronic device, which includes: at least one processor 901; and a memory 902 communicatively connected to at least one processor 901; wherein the memory 902 stores one or more computer programs that can be executed by at least one processor 901, and the one or more computer programs are executed by at least one processor 901 to enable at least one processor 901 to perform the above-described pattern recognition method.

[0154] In some embodiments, the electronic device can be a neuromorphic chip. Since neuromorphic chips can employ vectorized computation methods and require external memory, such as Double Data Rate (DDR) synchronous dynamic random access memory, to load parameters like weight information of the neural network model, the batch processing method used in this disclosure provides higher computational efficiency.

[0155] Furthermore, this disclosure also provides a computer-readable medium having a computer program stored thereon, wherein the computer program implements the above-described pattern recognition method when executed by a processor / processing core.

[0156] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0157] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A pattern recognition method, characterized in that, include: Based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators, a first sequence is generated, wherein each item of the first sequence corresponds to the feature vector of each operator. The fusion attributes of the operators are labeled according to the first sequence to obtain a second sequence. The fusion attributes are used to characterize whether the operator has the characteristic of fusion with adjacent operators. Based on the second sequence, a target subgraph to be fused with operators is determined from multiple subgraphs of the specified computation graph; If the target subgraph's fusion patterns include target fusion patterns that do not belong to the fusion pattern set, the fusion pattern set is updated based on the target fusion patterns, and the fusion pattern set includes at least one fusion pattern. After updating the fusion pattern set based on the target fusion pattern, the method further includes: Based on the target fusion pattern set, the target subgraph in the computation graph to be identified is determined, wherein the target fusion pattern set is obtained by updating the fusion pattern set multiple times; At least two operators in the target subgraph are fused to obtain a fusion operator; Obtain the execution strategy for the target subgraph, wherein the execution strategy includes at least the execution method of the fusion operator; Based on the execution strategy of the target subgraph, generate the instructions corresponding to the target subgraph; The instructions corresponding to the target subgraph are sent to the designated processing core, so that the designated processing core can execute the computational task corresponding to the target subgraph according to the instructions.

2. The pattern recognition method according to claim 1, characterized in that, The step of labeling the fusion attributes of the operator based on the first sequence to obtain the second sequence includes: Based on the first sequence, the fusion attributes of the operator are labeled using preset labeling values ​​to obtain the labeling values ​​of the operator; The second sequence is obtained based on the labeled values ​​of the multiple operators.

3. The pattern recognition method according to claim 1 or 2, characterized in that, The second sequence is obtained by labeling the fusion attributes of the operator based on the first sequence using a preset labeling model; The labeled model is a model obtained through training based on a conditional random field.

4. The pattern recognition method according to claim 3, characterized in that, The preset training set data includes a first training sequence for characterizing operator features and a second training sequence for characterizing operator fusion properties; The training process of the labeled model includes: An initial labeling model is constructed based on conditional random fields; The first training sequence is input into the initial annotation model to obtain a predicted annotation sequence, which is a sequence obtained by the initial annotation model based on the first training sequence for predicting operator fusion attributes. The annotation model is iteratively updated based on the predicted annotation sequence and the second training sequence to obtain a trained annotation model. The trained annotation model has the function of determining the second sequence corresponding to the first sequence based on the first sequence.

5. The pattern recognition method according to claim 4, characterized in that, After obtaining the predicted annotation sequence, and before iteratively updating the annotation model based on the predicted annotation sequence and the second training sequence, the method further includes: The predicted labeled sequence is reviewed to obtain the review result; Based on the review results, the predicted annotation sequence is updated to obtain an updated predicted annotation sequence. The updated predicted annotation sequence and the second training sequence are then used to iteratively update the annotation model to obtain a trained annotation model.

6. The pattern recognition method according to claim 1, characterized in that, The step of generating a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators includes: Arrange the multiple operators according to their execution order to obtain a third sequence; Based on the feature information of the operator, generate the feature vector of the operator; The feature vectors of the multiple operators are arranged according to the order of the operators in the third sequence to obtain the first sequence.

7. The pattern recognition method according to claim 6, characterized in that, The feature vector of the operator is generated by a preset vector acquisition model based on the feature information of the operator. The feature vector of the operator includes a first vector for characterizing the independent features of the operator and a second vector for characterizing the associated features of the operator with neighboring operators. The vector acquisition model is obtained through training.

8. The pattern recognition method according to claim 6 or 7, characterized in that, The feature information includes at least one of the following: operator structure, operator execution mechanism, operator memory access characteristics, and operator computation characteristics.

9. The pattern recognition method according to claim 1, characterized in that, The preset label value of the fusion attribute includes a first value and a second value, wherein the first value is used to characterize that the corresponding operator has the characteristic of fusion with adjacent operators, and the second value is used to characterize that the corresponding operator does not have the characteristic of fusion with adjacent operators. The step of determining the target subgraph to be fused from multiple subgraphs of the specified computation graph according to the second sequence includes: The operator in the second sequence corresponding to the first value is determined as the target operator; The subgraph containing the target operator in the multiple subgraphs is determined as the target subgraph.

10. A pattern recognition device, characterized in that, The apparatus for performing the pattern recognition method as described in any one of claims 1-9 includes: The generation module is configured to generate a first sequence based on the execution order of multiple operators in a specified computation graph and the feature information of the multiple operators, wherein each item of the first sequence corresponds to the feature vector of each operator. The annotation module is configured to annotate the fusion attributes of the operator according to the first sequence to obtain a second sequence, wherein the fusion attributes are used to characterize whether the operator has the characteristic of fusion with adjacent operators; The determination module is configured to determine, based on the second sequence, a target subgraph from multiple subgraphs of the specified computation graph to be fused with operators; The update module is configured to update the fusion pattern set based on the target fusion pattern when the fusion patterns of the target subgraph include a target fusion pattern that does not belong to the fusion pattern set, wherein the fusion pattern set includes at least one fusion pattern.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the pattern recognition method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the pattern recognition method as described in any one of claims 1-9.

Images