Image recognition method, device, storage medium and electronic device

Abstract

This specification discloses a method, device, storage medium, and electronic device for image recognition. The method includes: constructing an initial recognition model to identify each model operator contained in the initial recognition model, the operator type corresponding to each model operator, and the data transfer type between each model operator; for each candidate operator, and based on the data transfer type corresponding to the candidate operator, determining whether the candidate operator and the adjacent operator meet a preset fusion condition; if so, fusing the candidate operator with the adjacent operator to obtain a fused operator; constructing a target recognition model based on the fused operator and each unfused model operator and deploying it; obtaining an image to be recognized, inputting the image to be recognized into the deployed target recognition model, and outputting an image recognition result corresponding to the image to be recognized through the target recognition model.

Description

[0001] This application claims priority from patent application number 2022111480736, the contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to the field of computer technology, and in particular to an image recognition method, device, storage medium, and electronic device. Background Art

[0003] With the development of artificial intelligence, deep learning models have been widely used in fields such as machine vision and natural language processing. For models with large parameter counts, efficient inference on a single machine is difficult to achieve. Therefore, distributed deep learning methods are often used. Based on the computation time of each operator on each device, the data transmission between operators, and the network bandwidth between devices, a partitioning scheme is generated. The model is then divided into multiple parts according to this partitioning scheme, and these parts are then placed on different devices for collaborative training or inference.

[0004] However, deep learning models with large parameter counts, such as image recognition models, are usually composed of hundreds to tens of thousands of operators. A large number of operators will result in a longer model segmentation time, seriously reducing the overall efficiency of image recognition. In addition, too many operators will make it difficult to search for the optimal segmentation scheme to segment the model, and the current segmentation method is likely to destroy the optimal computing effect between the original operators, thereby reducing the recognition accuracy of the recognition model.

[0005] Therefore, how to reduce the number of operators of the recognition model, improve the segmentation efficiency and recognition efficiency of the recognition model, and ensure the accuracy of the recognition model is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides an image recognition method, device, storage medium, and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides an image recognition method, including:

[0009] Obtaining model parameters input by the user, and constructing an initial recognition model based on the model parameters;

[0010] Identifying the initial recognition model to identify each model operator included in the initial recognition model, an operator type corresponding to each model operator, and a data transfer type between each model operator;

[0011] Determine a model operator of a specified type from the model operators as a candidate operator, determine, for each candidate operator, a model operator with a data transfer type with the candidate operator as an adjacent operator, and determine, based on the data transfer type corresponding to the candidate operator, whether the candidate operator and the adjacent operator meet a preset fusion condition; if so, fuse the candidate operator with the adjacent operator to obtain a fused operator;

[0012] Constructing and deploying a target recognition model based on the fusion operator and the unfused model operators;

[0013] An image to be recognized is obtained, and the image to be recognized is input into the deployed target recognition model, so that an image recognition result corresponding to the image to be recognized is output through the target recognition model.

[0014] Optionally, judging whether the candidate operator and the adjacent operator meet a preset fusion condition according to the data transfer type corresponding to the candidate operator specifically includes:

[0015] According to the data transfer type corresponding to the candidate operator, the operator type corresponding to the candidate operator, and the operator type corresponding to the adjacent operator, it is determined whether the candidate operator and the adjacent operator meet a preset fusion condition.

[0016] Optionally, before determining whether the candidate operator and the adjacent operator meet a preset fusion condition, the method further includes:

[0017] For each model operator, connect the node corresponding to the model operator and the node corresponding to the adjacent operator with the data transfer type of the model operator through edges to construct an operator connection graph containing each model operator;

[0018] The type of edge between nodes is determined based on the data transfer type between operators in each model.

[0019] Optionally, for each candidate operator, a model operator having a data transfer type with the candidate operator is determined as a neighboring operator, and based on the data transfer type corresponding to the candidate operator, whether the candidate operator and the neighboring operator meet a preset fusion condition is determined. If so, the candidate operator is fused with the neighboring operator to obtain a fused operator, specifically including:

[0020] Determine the node corresponding to the candidate operator in the operator connection graph as the target node;

[0021] According to the operator type corresponding to the target node, the operator type corresponding to the child node of the target node, and the type of the edge between the target node and the child node, it is determined whether the candidate operator and the adjacent operator meet the preset fusion condition.

[0022] Optionally, judging whether the candidate operator and the adjacent operator meet a preset fusion condition according to the data transfer type corresponding to the candidate operator specifically includes:

[0023] If the candidate operator outputs data to at least two adjacent operators, it is determined that the candidate operator and each adjacent operator corresponding to the candidate operator do not meet the preset fusion condition.

[0024] Optionally, for each candidate operator, a model operator having a data transfer type with the candidate operator is determined as a neighboring operator, and based on the data transfer type corresponding to the candidate operator, whether the candidate operator and the neighboring operator meet a preset fusion condition is determined. If so, the candidate operator is fused with the neighboring operator to obtain a fused operator, specifically including:

[0025] If the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited from fusion by the preset fusion condition, then it is determined that the candidate operator and the adjacent operator meet the preset fusion condition.

[0026] Optionally, for each candidate operator, a model operator having a data transfer type with the candidate operator is determined as a neighboring operator, and based on the data transfer type corresponding to the candidate operator, whether the candidate operator and the neighboring operator meet a preset fusion condition is determined. If so, the candidate operator is fused with the neighboring operator to obtain a fused operator, specifically including:

[0027] If the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator match at least some of the operator types specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited by the preset fusion condition, then temporarily binding the model operator to the adjacent operator;

[0028] If the operator type of each bound candidate operator and the operator type of the next adjacent operator still match at least part of the operator types specified by the preset fusion condition, then the next adjacent operator is bound to each bound candidate operator;

[0029] Each subsequent model operator is traversed until the operator type corresponding to each bound candidate operator completely matches the operator type specified by the preset fusion condition, and then the bound candidate operators are fused.

[0030] Optionally, the method further includes:

[0031] If the operator type of each bound candidate operator and the operator type corresponding to the next adjacent operator do not match the operator type specified by the preset fusion condition, it is determined that the bound candidate operators and the next adjacent operator do not meet the preset fusion condition, and the binding of each bound candidate operator is released.

[0032] This specification provides an image recognition device, including:

[0033] An acquisition module obtains model parameters input by the user and constructs an initial recognition model based on the model parameters;

[0034] A first recognition module recognizes the initial recognition model to identify each model operator included in the initial recognition model, an operator type corresponding to each model operator, and a data transfer type between each model operator;

[0035] A fusion module determines a model operator of a specified type from the model operators as a candidate operator, and for each candidate operator, determines a model operator with a data transfer type with the candidate operator as an adjacent operator, and determines whether the candidate operator and the adjacent operator meet a preset fusion condition based on the data transfer type corresponding to the candidate operator. If so, the candidate operator is fused with the adjacent operator to obtain a fused operator;

[0036] A deployment module constructs and deploys a target recognition model based on the fusion operator and the unfused model operators;

[0037] The second recognition module obtains an image to be recognized, and inputs the image to be recognized into the deployed target recognition model, so as to output an image recognition result corresponding to the image to be recognized through the target recognition model.

[0038] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image recognition method is implemented.

[0039] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image recognition method when executing the program.

[0040] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0041] In the image recognition method provided in this specification, an initial recognition model is constructed to identify the model operators contained in the initial recognition model, the operator type corresponding to each model operator, and the data transfer type between each model operator. For each candidate operator, and based on the data transfer type corresponding to the candidate operator, it is determined whether the candidate operator and the adjacent operator meet the preset fusion conditions. If so, the candidate operator is fused with the adjacent operator to obtain a fused operator; a target recognition model is constructed and deployed based on the fused operator and the unfused model operators; an image to be recognized is obtained, and the image to be recognized is input into the deployed target recognition model to output the image recognition result corresponding to the image to be recognized through the target recognition model.

[0042] It can be seen from the above method that this specification will determine whether the preset fusion conditions are met based on the data transfer type between the candidate operator and its adjacent operators in the initial model, thereby avoiding destroying the optimal computing effect between some operators in the initial model and improving the accuracy of the recognition model. In addition, the image recognition method provided in this specification can fuse model operators that meet the preset fusion conditions, significantly reducing the number of model operators and improving the image recognition efficiency of the recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0044] Figure 1 A flowchart of an image recognition method provided in this specification;

[0045] Figure 2 This is a schematic diagram of a data transmission type provided in this specification;

[0046] Figure 3 A schematic diagram of an operator connection graph provided in this specification;

[0047] Figure 4 A schematic diagram of a fusion rule provided in this specification;

[0048] Figure 5 A schematic diagram of a target recognition model corresponding to an operator connection diagram provided in this specification;

[0049] Figure 6 This is a schematic diagram of an image recognition process provided in this manual;

[0050] Figure 7 A schematic diagram of an image recognition device provided in this specification;

[0051] Figure 8 This manual provides a Figure 1 Schematic diagram of the electronic device. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0053] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a flow chart of an image recognition method provided in this specification, comprising the following steps:

[0055] S101: Obtain model parameters input by the user, and construct an initial recognition model based on the model parameters.

[0056] In fields such as image recognition and natural language processing, the number of parameters of the mathematical models used (such as deep learning models) is often too large. Therefore, in order to improve the computational efficiency of the model, a distributed deep learning method is usually adopted. According to the computing time of each operator in the model on each device, the data transmission speed between operators, and the network bandwidth between devices, a corresponding segmentation scheme is generated, and the model is divided into multiple parts according to the segmentation scheme, so that these multiple parts are placed on different devices for collaborative reasoning and calculation.

[0057] However, the current model segmentation method cannot accurately determine the optimal segmentation scheme when facing models with a large number of parameters such as image recognition models. It will also destroy the optimal calculation effect between the original some operators, seriously reducing the accuracy of the recognition model. In addition, the large number of parameters also requires a long time to determine the segmentation scheme, which increases the overall time of image recognition and seriously reduces the efficiency of image recognition.

[0058] Based on this, this specification provides an image recognition method to improve image recognition efficiency while ensuring the accuracy of the recognition model.

[0059] The server needs to obtain the model parameters input by the user and build an initial recognition model based on the model parameters. In actual applications, the above model parameters can be the model parameters of a trained recognition model. After the server obtains the corresponding model parameters, it can directly build the initial recognition model.

[0060] It should be noted that the initial recognition model can be a trained model with certain recognition capabilities. In practical applications, the initial recognition model can also be a trained open source model provided by other users or organizations. After the server obtains the initial model, it only needs to perform the corresponding inference on it. Once the inference is completed, it can be deployed for image recognition.

[0061] In this specification, the execution subject for implementing the image recognition method can be a designated device such as a server. For the sake of convenience, this specification only takes the server as the execution subject as an example to illustrate an image recognition method provided in this specification.

[0062] S102: Identify the initial recognition model to identify each model operator included in the initial recognition model, the operator type corresponding to each model operator, and the data transfer type between each model operator.

[0063] After obtaining the initial recognition model, the server can identify the initial recognition model to identify the model operators contained in the initial recognition model and the operator types corresponding to each model operator. In practical applications, deep learning algorithms are composed of individual computing units, which are model operators (OPs). In the initial recognition model, the operator types of the model operators can correspond to the computing logic in the network layer of the initial recognition model, such as: convolution calculation (conv) in the convolution layer, activation calculation (relu) in the activation layer, normalization calculation (bn) in the batch normalization (bn) layer, summation calculation (add), matrix multiplication calculation (matmul), etc.

[0064] It should be noted that the above descriptions are only based on several representative operator types. In actual applications, other operator types may also be included, which are not listed one by one in this specification.

[0065] In addition, the server can also determine the data transfer type between each model operator. Generally, the data transfer type between operators includes a unique input type, a unique output type, a multiple input type, and a multiple output type. For ease of understanding, this specification provides a schematic diagram of the data transfer type between model operators, such as Figure 2 shown.

[0066] Figure 2 This is a diagram of a data transmission type provided in this specification.

[0067] like Figure 2As shown, operator op1 only outputs to operator op2, and operator op2 only receives input from operator op1. Therefore, the data transfer type corresponding to operator op1 is the only output type, and the data transfer type corresponding to operator op2 is the only input type. In other words, op1 is the only input object of op2, and op2 is the only output object of op1.

[0068] Operator op3 outputs to operators op4 and op5 respectively, that is, one-to-many output, so the data transfer type corresponding to operator op3 is a multiple output type.

[0069] Operator op8 receives inputs from operators op6 and op7 respectively, that is, many-to-one input, so the data transfer type corresponding to operators op6 and op7 is a multiple-input type.

[0070] In addition, for each model operator, the server can also connect the node corresponding to the model operator and the node corresponding to the adjacent operator that has a data transfer relationship with the model operator through edges, build an operator connection graph containing each model operator, and determine the type of edge between each node based on the data transfer type between each model operator. In this specification, the operator connection graph can be a directed acyclic graph. For ease of understanding, this specification provides a schematic diagram of the operator connection graph, such as Figure 3 shown.

[0071] Figure 3 This is a schematic diagram of an operator connection graph provided in this specification.

[0072] Among them, the nodes corresponding to each model operator are connected with the data transfer type between the model operators as edges, thus forming a directed acyclic graph.

[0073] It should be noted that there may be multiple different model operators in this specification, but these different model operators belong to the same operator type, such as Figure 3 The model operators corresponding to the nodes conv(1) and conv(2) in the example can be different conv model operators. These two model operators differ only in their operation logic, but they belong to the same operator type (conv). However, the model operators corresponding to nodes such as add1, add2, and add3 can be model operators with the same operation logic.

[0074] S103: Determine a model operator of a specified type from the model operators as a candidate operator. For each candidate operator, determine a model operator with a data transfer type with the candidate operator as an adjacent operator. Based on the data transfer type corresponding to the candidate operator, determine whether the candidate operator and the adjacent operator meet a preset fusion condition. If so, fuse the candidate operator with the adjacent operator to obtain a fused operator.

[0075] In practice, not all types of model operators can be fused. Therefore, the server can identify model operators that can be fused from among the various model operators as candidate operators. In this specification, the operator types of the candidate operators can include at least one of conv, bn, add, relu, etc. Of course, other types of candidate operators may also be included, and this specification does not specifically limit this.

[0076] Specifically, the server can determine the node corresponding to the candidate operator in the above-mentioned operator connection graph, and judge whether the candidate operator corresponding to the node and the adjacent operator corresponding to the subnode can be fused based on the operator type corresponding to the node and the operator type corresponding to the subnode connected to the node, as well as the type of the edge between the node and the subnode.

[0077] The edge type in the operator connection graph corresponds to the data transfer type between operators. The server can set the edge corresponding to the data transfer relationship of the unique input / output type to the first type edge (e.g. Figure 3 The edge between node conv1 and node bn1 in the middle), sets the edge corresponding to the multi-input type data transfer relationship to the second type edge (such as Figure 3 The edge between node relu2 and node add3, and the edge between node bn2 and node add3), the edge corresponding to the multi-output type data transfer relationship is set to the third type edge (such as Figure 3 The edge between nodes add1 and relu1, and the edge between nodes add1 and conv1).

[0078] In practical applications, there are some specific model operators whose cooperation often achieves the best operational results (such as operational efficiency and accuracy). For example, when candidate operators of the operator type conv and bn are executed sequentially on the same device, the operational results are better than deploying conv and bn on different devices for collaborative operation. In practical applications, the algorithms of the two candidate operators can be adjusted to merge the candidate operator conv and the operator bn into a single model operator without changing the accuracy, thereby reducing the number of model operators.

[0079] For another example, when the model operators conv, bn, and relu are executed sequentially on the same device, and when the model operators conv, bn, add, and relu are executed sequentially on the same device, the best operation effect can be achieved, and the above model operators can be fused by adjusting the algorithm. Therefore, the server can determine different fusion rules based on the above candidate operators. The preset fusion rules are as follows: Figure 4 shown.

[0080] Figure 4 This is a schematic diagram of a preset fusion rule provided in this manual.

[0081] Among them, when candidate operators of operator type conv and operator type bn are connected in sequence in the operator connection graph, the above two types of candidate operators can be fused into conv+bn; when candidate operators of operator types conv, bn, and relu are connected in sequence in the operator connection graph, the three candidate operators can be fused into conv+bn+relu; similarly, when candidate operators of operator types conv, bn, add, and relu are connected in sequence in the operator connection graph, the four candidate operators can be fused into conv+bn+add+relu.

[0082] If necessary, this specification only uses the above three specific representative fusion rules for illustration. In actual applications, other fusion rules may be set, which are not listed one by one in this specification.

[0083] The server can construct additional input nodes as root nodes in the operator connection graph, and then traverse each node in the operator connection graph through a depth-first search algorithm.

[0084] In the specification, the above-mentioned preset fusion conditions can be of two types, namely, the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited from fusion by the preset fusion condition.

[0085] The server may fuse the candidate operator with the adjacent operator only when the candidate operator and the adjacent operator meet both of the above-mentioned preset fusion conditions. Of course, the server may also fuse the candidate operator with the adjacent operator when the candidate operator and the adjacent operator meet only one of the preset fusion conditions.

[0086] It should be noted that, when the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator are exactly the same as the operator type corresponding to at least one of the fusion rules, and the order of the candidate operator and the adjacent operator is also exactly the same as the order of the two operator types specified by the above fusion rules, the server can consider that the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator are completely matched with the operator type specified by the preset fusion conditions.

[0087] Therefore, when the candidate operator and the adjacent operator completely match the operator type corresponding to any of the above fusion rules, it can be considered that the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition. At this time, it can be further determined based on the type of the edge between the node and the node corresponding to the adjacent operator whether the data transfer type corresponding to the candidate operator is a data transfer type prohibited from fusion by the preset fusion condition. If not, it means that the candidate operator and the adjacent operator meet the preset fusion condition. The data transfer type prohibited from fusion by the preset fusion condition will be described in detail below, and this manual will not go into too much detail here.

[0088] In practical applications, if a candidate operator outputs to two or more adjacent operators at the same time, once the candidate operator is fused with one of the adjacent operators, it will inevitably cause the other adjacent operator that has not been fused to be unable to obtain the required input normally. For example, if Figure 2 If operators op3 and op4 in the above are fused, operators op3 and op4 will become a fused operator. At this time, operator op5 needs to spend a lot of time to extract the output of operator op3 from the above fused operator separately as the input of operator op5, which will reduce the recognition efficiency of the recognition model.

[0089] Therefore, the server can use the multi-output type as the data transfer type prohibited by the preset fusion condition. When the candidate operator outputs to at least two adjacent operators, it can be determined that the data transfer type between the candidate operator and each adjacent operator is the data transfer type (multi-output type) prohibited by the preset fusion condition. At this time, it can be determined that the candidate operator and each adjacent operator do not meet the preset fusion condition.

[0090] Of course, the server can also directly judge whether the candidate operator and the adjacent operator can be fused based on the type of edge between the node corresponding to the candidate operator and its child nodes (nodes corresponding to adjacent operators) in the operator connection graph. If the type of the edge between the node corresponding to the candidate operator and its child nodes is the third type of edge (that is, the edge corresponding to the multi-output type of data transmission relationship), it can be determined that the candidate operator and its adjacent operator do not meet the preset fusion conditions.

[0091] If the data transfer type between the candidate operator and the adjacent operator is not a data transfer type prohibited by the preset fusion condition (such as the only input type, the only output type, and the multiple input type), the candidate operator can be fused with the adjacent operator, such as Figure 3 The operator types corresponding to the nodes conv(1) and bn1 in the example completely match the operator types and sequence corresponding to fusion rule 1, and the data transfer type is not a multi-output type. Therefore, the candidate operator corresponding to the node conv(1) and the candidate operator corresponding to the node bn1 can be fused.

[0092] If the model operator and the adjacent operators do not match the operator type corresponding to each preset fusion rule, the server can determine that the candidate operator and its adjacent operators do not meet the preset fusion conditions, such as Figure 3 If the nodes add1 and relu1 in do not meet any fusion rule, the candidate operators corresponding to these two nodes do not meet the preset fusion rule.

[0093] Furthermore, for each candidate operator, if the operator types corresponding to the candidate operator and its adjacent operators match at least some of the operator types corresponding to the fusion rule, it can be determined whether the data transfer type between the candidate operator and its adjacent operators is a data transfer type prohibited by the preset fusion condition.

[0094] For example, when the candidate operator is Figure 3 When the candidate operator corresponding to the node conv(2) in the adjacent operator is bn2, the operator types of the two candidate operators match some of the operator types in the preset fusion rule 2 and the preset fusion rule 3. The server can further determine the data transfer type between the two candidate operators.

[0095] If the data transfer type between the two model operators is not a multi-output type, the server can temporarily bind the candidate operator and its adjacent operators to further determine the next adjacent node.

[0096] It should be noted that the binding and fusion mentioned in this specification are two completely different concepts. Fusion refers to the adjustment and fusion of the algorithms of two candidate operators, so as to merge them into a model operator, and the fused model operator (fusion operator) cannot be split. Binding is a temporary combination of two candidate operators. When it is determined that the preset fusion conditions are not met, the two candidate operators can be split out. When it is determined that the preset fusion conditions are met, the bound candidate operators can be fused at a deeper level.

[0097] In addition, in this specification, candidate operators and their adjacent operators will be bound only when they meet some fusion rules. On this basis, the operator type corresponding to the adjacent operator must also be the operator type corresponding to one of the candidate operators. Therefore, all bound model operators are candidate operators.

[0098] If the operator types of each bound candidate operator and the next adjacent operator still match at least part of the operator types corresponding to one of the fusion rules, then each bound candidate operator is further bound to the next adjacent operator.

[0099] For example, when determining Figure 3 When the operator types corresponding to the nodes conv(2) and bn2 in the fusion rules match some of the operator types in fusion rules 2 and 3, the model operators corresponding to the nodes conv(2) and bn2 can be temporarily bound, and then it is determined whether the next adjacent operator add3 and the candidate operators bound above match some of the operator types in one of the fusion rules. Figure 3 The next adjacent operator of bn2 is add3, so the nodes conv(2), bn2, and add3 still match some operator types in the preset fusion rule 3. At this time, we can continue to bind the candidate operators corresponding to the node add3 to obtain the bound candidate operators corresponding to the conv(2), bn2, and add3 nodes.

[0100] Each subsequent model operator is traversed until the operator type corresponding to the bound candidate operator completely matches the operator type corresponding to the preset fusion rule, and then the bound candidate operators are fused.

[0101] For example, in Figure 2 In the example, the child node of node add3 is relu3. Therefore, after binding the previously bound candidate operators again, we can obtain the candidate operators corresponding to the four bound nodes conv, bn, add, and relu, and match all the operator types corresponding to fusion rule 3. The edges between these nodes are not third-type edges, so the candidate operators corresponding to the four nodes conv, bn, add, and relu can be fused.

[0102] In this specification, the order between the above-mentioned nodes can correspond to the input and output order corresponding to each node, or the order during actual execution, and these model operators can only be fused when the order of each node is the same as the order of each operator type specified in the fusion rule, and when these candidate operators are completely matched with the operator types corresponding to the preset fusion rule.

[0103] If the next adjacent operator is not a candidate operator, it means that the server cannot fuse the adjacent operator with the already bound candidate operator. At this time, it can be determined that these model operators do not meet the preset fusion conditions and the binding of the already bound candidate operators is released.

[0104] If the operator type of each bound candidate operator and the operator type corresponding to the next adjacent operator do not match the operator type specified by the preset fusion condition (that is, they do not meet the operator type corresponding to any fusion rule), the server can determine that each bound candidate operator and the next adjacent operator do not meet the preset fusion condition, and unbind the bound candidate operators.

[0105] by Figure 3 Taking the operator link graph shown in the figure as an example, when data is input to node add1, node add1 will output to node relu1 and node conv(1) respectively. Since node add1 and node relu1, and node add1 and node conv(1) do not meet any fusion rules, and the edge between these two nodes is a third-type edge, the server can prohibit the fusion of candidate operators corresponding to the above nodes.

[0106] However, the nodes relu1, add2, and relu2 following the node add1 do not conform to any fusion rule, so the server may prohibit the fusion of the candidate operators corresponding to the nodes add1, relu1, add2, and relu2.

[0107] Node conv(1) and node bn1 meet fusion rule 1, and the edge between node conv(1) and node bn is a first-type edge. Therefore, the server can fuse the candidate operators corresponding to node conv(1) and node bn1 to obtain the fused operator conv(1)+bn1.

[0108] Since the node conv(2) after node bn1 and node bn2 meet some of the operator types in fusion rule 2 and fusion rule 3, the server can first temporarily bind the candidate operators corresponding to conv(2) and node bn2. The bound nodes conv(2) and bn2 and the subsequent node add3 still meet some of the fusion rule 3. The server can continue to temporarily bind the candidate operators corresponding to nodes conv(2), bn2, and add3. Since the node after add3 is relu3, the candidate operators corresponding to the previous nodes conv(2), bn2, and add3 exactly match the operator types specified in fusion rule 3, and the edges between these nodes are not third-type edges, indicating that the data transfer type between these nodes is not a multi-output relationship. Therefore, the candidate operators corresponding to nodes conv(2), bn2, add3, and relu3 can be fused to obtain the fused operator conv(2)+bn2+add3+relu3.

[0109] S104: Constructing and deploying a target recognition model based on the fusion operator and the unfused model operators.

[0110] S105: Acquire an image to be recognized, and input the image to be recognized into the deployed target recognition model, so that the target recognition model outputs an image recognition result corresponding to the image to be recognized.

[0111] After obtaining the fusion operator, the server can reconstruct the target recognition model based on each fusion operator and each unfused model operator. For ease of understanding, this manual provides a target recognition model corresponding operator connection diagram, such as Figure 5 shown.

[0112] Figure 5 A schematic diagram of an operator connection graph corresponding to a target recognition model provided in this specification.

[0113] Among them, the candidate operators corresponding to nodes conv(1) and bn1 completely match the operator types corresponding to fusion rule 1, so the candidate operators corresponding to nodes conv(1) and bn1 are fused. The candidate operators corresponding to nodes conv(2), bn2, add3, and relu3 completely match all operator types corresponding to fusion rule 3, so the server can fuse the candidate operators corresponding to nodes conv(2), bn2, add3, and relu3.

[0114] After determining the target model, the server can determine the segmentation scheme corresponding to the target recognition model and deploy the target recognition model to different devices according to the segmentation scheme.

[0115] After the server obtains the image to be recognized, the image to be recognized can be input into the target recognition model deployed on different devices, and the target recognition model deployed on different devices can recognize the image to be recognized, thereby outputting the image recognition result corresponding to the image to be recognized.

[0116] The above-mentioned devices may be different Raspberry Pi computing devices, and of course, may also be other computing devices such as laptop computers, desktop computers, etc., which is not specifically limited in this specification.

[0117] For ease of understanding, this manual also provides a schematic diagram of the image recognition process, such as Figure 6 shown.

[0118] Figure 6 This is a schematic diagram of an image recognition process provided in this manual.

[0119] Among them, the server can build an operator connection graph based on each model operator in the initial recognition model, and add an input node to the operator connection graph, and traverse each node in the operator connection graph after the input node. For each node accessed by the server, if the operator type corresponding to the node and its child nodes completely matches one of the fusion rules, and the edge between the two nodes is a fusionable edge (first type edge or second type edge), the candidate operators corresponding to the two nodes can be fused to obtain a fusion operator, and then continue to access the next node.

[0120] If the operator types corresponding to the current node and its child nodes match some of the operator types in one of the fusion rules, and the edge between the two nodes is the aforementioned fusionable edge, then the candidate operators corresponding to the current node and its child nodes can be bound to obtain the bound operator, and the next node can be accessed until the operator types of all bound candidate operators completely match the operator types corresponding to any fusion rule, and then the bound nodes that match the fusion rule will be fused. If the node types corresponding to each fusion rule do not match, or the edge between the nodes is an unfusionable edge (third type edge), the bound nodes are unbound, and finally the target recognition model is obtained based on the fusion operators and the operators that have not been fused.

[0121] It can be seen from the above method that this specification will determine whether the preset fusion conditions are met based on the data transfer type between the candidate operator and its adjacent operators in the initial model, thereby avoiding destroying the optimal computing effect between some operators in the initial model and improving the accuracy of the recognition model. In addition, the image recognition method provided in this specification can fuse model operators that meet the preset fusion conditions, significantly reducing the number of model operators and improving the image recognition efficiency of the recognition model.

[0122] Moreover, under the condition that the input image size is 3*224*224, after the image is input into the fused target recognition model, the image is subjected to feature extraction and classification calculation based on each model operator and the fusion operator to obtain the image recognition result. On the Raspberry Pi computing device, the calculation time of conv(1)+bn1 is 348ms, and the calculation time of conv(2)+bn2+add3+relu3 is 235ms. Through calculation, it can be seen that Figure 3 In , the end-to-end delay of the computation graph before locating the operator fusion rules and performing fusion is 878ms; Figure 5 In the example, after locating the operator fusion rule and performing operator fusion, the end-to-end latency of the computational graph was 595ms, and the end-to-end inference latency was reduced to 67.8% of the original. This shows that the image recognition method provided in this specification can greatly improve the recognition efficiency of the recognition model.

[0123] The above are one or more implementation methods of this specification. Based on the same idea, this specification also provides corresponding devices, such as Figure 7 shown.

[0124] Figure 7 A schematic diagram of an image recognition device provided in this specification, including:

[0125] The acquisition module 701 is used to obtain the model parameters input by the user and construct an initial recognition model based on the model parameters;

[0126] A first recognition module 702 is configured to recognize the initial recognition model to identify each model operator included in the initial recognition model, an operator type corresponding to each model operator, and a data transfer type between each model operator;

[0127] Fusion module 703 is configured to determine a model operator of a specified type from the model operators as a candidate operator, determine, for each candidate operator, a model operator with a data transfer type with the candidate operator as a neighboring operator, and determine, based on the data transfer type corresponding to the candidate operator, whether the candidate operator and the neighboring operator meet a preset fusion condition. If so, fuse the candidate operator with the neighboring operator to obtain a fused operator.

[0128] A deployment module 704 is used to construct and deploy a target recognition model based on the fusion operator and the unfused model operators;

[0129] The second recognition module 705 is used to obtain an image to be recognized, and input the image to be recognized into the deployed target recognition model, so as to output an image recognition result corresponding to the image to be recognized through the target recognition model.

[0130] Optionally, the fusion module 703 is specifically configured to determine whether the candidate operator and the adjacent operator meet a preset fusion condition based on the data transfer type corresponding to the candidate operator, the operator type corresponding to the candidate operator, and the operator type corresponding to the adjacent operator.

[0131] Optionally, before determining whether the candidate operator and the adjacent operator meet the preset fusion conditions, the fusion module 703 is further used to, for each model operator, connect the node corresponding to the model operator and the node corresponding to the adjacent operator having the data transfer type with the model operator through an edge, to construct an operator connection graph containing each model operator; and determine the type of edge between each node according to the data transfer type between each model operator.

[0132] Optionally, the fusion module 703 is specifically used to determine the node corresponding to the candidate operator in the operator connection graph as the target node; based on the operator type corresponding to the target node, the operator type corresponding to the child node of the target node, and the type of the edge between the target node and the child node, determine whether the candidate operator and the adjacent operator meet the preset fusion condition.

[0133] Optionally, the fusion module 703 is specifically configured to, if the candidate operator outputs data to at least two adjacent operators, determine that the candidate operator and each adjacent operator corresponding to the candidate operator do not satisfy a preset fusion condition.

[0134] Optionally, the fusion module 703 is specifically used to determine that the candidate operator and the adjacent operator meet the preset fusion condition if the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited from fusion by the preset fusion condition.

[0135] Optionally, the fusion module 703 is specifically used to temporarily bind the model operator to the adjacent operator if the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator match at least part of the operator types specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited by the preset fusion condition; if the operator type of each bound candidate operator and the operator type of the next adjacent operator still match at least part of the operator type specified by the preset fusion condition, then bind the next adjacent operator to each already bound candidate operator; traverse each subsequent model operator until the operator type corresponding to each bound candidate operator completely matches the operator type specified by the preset fusion condition, and then fuse the bound candidate operators.

[0136] Optionally, the fusion module 703 is also used to determine that the bound candidate operators and the next adjacent operator do not meet the preset fusion condition, and to unbind the bound candidate operators if the operator type of the bound candidate operators and the operator type corresponding to the next adjacent operator do not match the operator type specified by the preset fusion condition.

[0137] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 A method for image recognition is provided.

[0138] This specification also provides a corresponding Figure 1 Schematic diagram of the electronic equipment. Figure 8 shown.

[0139] Figure 8 This manual provides a Figure 1 Schematic diagram of the electronic device.

[0140] As shown in the figure, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The image recognition method.

[0141] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0142] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0143] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0144] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0145] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0146] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0152] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0154] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0157] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for image recognition, characterized in that: include: Obtaining model parameters input by the user, and constructing an initial recognition model based on the model parameters; Identifying the initial recognition model to identify each model operator included in the initial recognition model, an operator type corresponding to each model operator, and a data transfer type between each model operator; Determine a model operator of a specified type from the model operators as a candidate operator, and for each candidate operator, determine a model operator with a data transfer type between the candidate operator and the candidate operator as an adjacent operator, and judge whether the candidate operator and the adjacent operator meet a preset fusion condition based on the data transfer type corresponding to the candidate operator. If so, fuse the candidate operator with the adjacent operator to obtain a fused operator; wherein, if the candidate operator outputs data to at least two adjacent operators, it is determined that the candidate operator and each adjacent operator corresponding to the candidate operator do not meet the preset fusion condition; if the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited by the preset fusion condition, then it is determined that the candidate operator and the adjacent operator meet the preset fusion condition; Constructing and deploying a target recognition model based on the fusion operator and the unfused model operators; An image to be recognized is obtained, and the image to be recognized is input into the deployed target recognition model, so that an image recognition result corresponding to the image to be recognized is output through the target recognition model.

2. The method according to claim 1, wherein According to the data transfer type corresponding to the candidate operator, determining whether the candidate operator and the adjacent operator meet the preset fusion conditions specifically includes: According to the data transfer type corresponding to the candidate operator, the operator type corresponding to the candidate operator, and the operator type corresponding to the adjacent operator, it is determined whether the candidate operator and the adjacent operator meet a preset fusion condition.

3. The method according to claim 1, wherein Before determining whether the candidate operator and the adjacent operator meet a preset fusion condition, the method further includes: For each model operator, connect the node corresponding to the model operator and the node corresponding to the adjacent operator with the data transfer type of the model operator through edges to construct an operator connection graph containing each model operator; The type of edge between nodes is determined based on the data transfer type between operators in each model.

4. The method according to claim 3, wherein For each candidate operator, determine a model operator with a data transfer type between the candidate operator and the candidate operator as a neighboring operator, and determine whether the candidate operator and the neighboring operator meet the preset fusion condition based on the data transfer type corresponding to the candidate operator. If so, fuse the candidate operator with the neighboring operator to obtain a fused operator, which specifically includes: Determine the node corresponding to the candidate operator in the operator connection graph as the target node; According to the operator type corresponding to the target node, the operator type corresponding to the child node of the target node, and the type of the edge between the target node and the child node, it is determined whether the candidate operator and the adjacent operator meet the preset fusion condition.

5. The method according to claim 1, wherein For each candidate operator, determine a model operator with a data transfer type between the candidate operator and the candidate operator as a neighboring operator, and determine whether the candidate operator and the neighboring operator meet the preset fusion condition based on the data transfer type corresponding to the candidate operator. If so, fuse the candidate operator with the neighboring operator to obtain a fused operator, which specifically includes: If the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator match at least some of the operator types specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited by the preset fusion condition, then temporarily binding the model operator to the adjacent operator; If the operator type of each bound candidate operator and the operator type of the next adjacent operator still match at least part of the operator types specified by the preset fusion condition, then the next adjacent operator is bound to each bound candidate operator; Each subsequent model operator is traversed until the operator type corresponding to each bound candidate operator completely matches the operator type specified by the preset fusion condition, and then the bound candidate operators are fused.

6. The method according to claim 5, wherein The method further comprises: If the operator type of each bound candidate operator and the operator type corresponding to the next adjacent operator do not match the operator type specified by the preset fusion condition, it is determined that the bound candidate operators and the next adjacent operator do not meet the preset fusion condition, and the binding of each bound candidate operator is released.

7. An image recognition device, characterized in that: include: An acquisition module obtains model parameters input by the user and constructs an initial recognition model based on the model parameters; A first recognition module recognizes the initial recognition model to identify each model operator included in the initial recognition model, an operator type corresponding to each model operator, and a data transfer type between each model operator; A fusion module determines a model operator of a specified type from the model operators as a candidate operator, and for each candidate operator, determines a model operator with a data transfer type between the candidate operator and the candidate operator as an adjacent operator, and judges whether the candidate operator and the adjacent operator meet a preset fusion condition based on the data transfer type corresponding to the candidate operator. If so, the candidate operator is fused with the adjacent operator to obtain a fused operator; wherein, if the candidate operator outputs data to at least two adjacent operators, it is determined that the candidate operator and each adjacent operator corresponding to the candidate operator do not meet the preset fusion condition; if the operator type corresponding to the candidate operator and the operator type corresponding to the adjacent operator completely match the operator type specified by the preset fusion condition, and the data transfer type corresponding to the candidate operator is not a data transfer type prohibited by the preset fusion condition, then it is determined that the candidate operator and the adjacent operator meet the preset fusion condition; A deployment module constructs and deploys a target recognition model based on the fusion operator and the unfused model operators; The second recognition module obtains an image to be recognized, and inputs the image to be recognized into the deployed target recognition model, so as to output an image recognition result corresponding to the image to be recognized through the target recognition model.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images