Neural network optimization method and related device for image processing

Abstract

The present application provides a neural network optimization method and related device for image processing; the method includes: obtaining multiple pre-selected neural networks, performing multiple rounds of iterations on all the pre-selected neural networks, in each round of iterations, executing a branch increase and decrease strategy on the candidate neural networks in the previous round to obtain multiple second neural networks, using the embedding layer, encoding layer and sorting layer in the structure controller to sort the relative positions of the second neural networks, determine multiple third neural networks, obtain multiple candidate neural networks of the round by performing bidirectional distillation migration on the third neural network, and determine a target neural network of the round among the multiple candidate neural networks, and output the target neural network obtained in the last round in response to determining that the preset first iteration index has been reached. It can be seen that the method selects the optimal neural network by relative sorting during the iteration process, which simplifies the process of training and testing each neural network.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a neural network optimization method and related devices for image processing. Background Art

[0002] Neural networks are often used when processing image data. However, whether it is manual design or automated design, its carrier platform must be considered. That is to say, a platform with multiple GPU (graphics processing unit) devices can provide rich computing and storage resources, so it can carry more complex neural networks in pursuit of higher accuracy. On other platforms, such as most mobile devices, only the CPU (central processing unit) can be used to support the operation of neural networks. Therefore, only neural networks with lighter structures are allowed to be carried, and their accuracy is often difficult to guarantee.

[0003] Due to the differences in computing resources and memory between different platforms or devices, a solution is needed to quickly obtain a suitable neural network. In other words, the obtained neural network needs to be able to strike a balance between accuracy and structural complexity. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a neural network optimization method and related devices for image processing.

[0005] Based on the above objectives, the present application provides a neural network optimization method for image processing, including:

[0006] Acquire multiple pre-selected neural networks, perform multiple rounds of iterations on all of the pre-selected neural networks, obtain multiple candidate neural networks and a target neural network in each round, and output the target neural network obtained in the last round in response to determining that a preset first iteration indicator is reached;

[0007] The following operations are performed for each of the multiple rounds of iterations:

[0008] In response to the current round being the first round, the plurality of pre-selected neural networks are used as a plurality of first neural networks of the current round,

[0009] In response to the current round not being the first round, using the plurality of candidate neural networks of the previous round as the plurality of first neural networks of the current round;

[0010] By adding neural nodes and reducing neural nodes in each of the plurality of first neural networks, a plurality of second neural networks of the current round are obtained;

[0011] Using the accuracy and memory consumption of each of the second neural networks in the current round when running, construct an evaluation index function, using the evaluation index function to construct a ranking loss function, and using the ranking loss function to determine multiple third neural networks in the current round;

[0012] In response to the current round being the first round, an auxiliary neural network is determined from the multiple pre-selected neural networks, and the multiple third neural networks of the current round are bidirectionally distilled and migrated using the auxiliary neural network; in response to the current round being not the first round, the multiple third neural networks of the current round are bidirectionally distilled and migrated using the target neural network of the previous round, and the results of the bidirectional distillation and migration are used as the multiple candidate neural networks of the current round, and the target neural network of the current round is determined from the multiple candidate neural networks.

[0013] Further, the obtaining of a plurality of pre-selected neural networks comprises:

[0014] constructing a first predetermined number of building blocks;

[0015] The following operation is performed for each of the structural blocks:

[0016] Using each operation type in the preset operation type set as the operation type of the neural layer to obtain a plurality of neural nodes, each of the neural nodes includes two neural layers, each of the neural layers corresponds to one operation type, and any two neural nodes in the plurality of neural nodes include different operation types;

[0017] Select any one neural node from the plurality of neural nodes and place it in the structural block;

[0018] Cascading all the structural blocks to obtain an initial backbone neural network;

[0019] Perform multiple rounds of iterative addition and subtraction of branches on the initial backbone neural network, and obtain multiple initial neural networks of the round in each round;

[0020] In response to determining that a preset second iteration indicator is reached, outputting a plurality of initial neural networks of a final round as the plurality of pre-selected neural networks;

[0021] The following operations are performed for each of the multiple rounds of adding and reducing branches iterations:

[0022] In response to the current round being the first round, the initial trunk neural network is used as the input neural network of the current round; in response to the current round being not the first round, multiple initial neural networks of the previous round are used as the input neural network of the current round;

[0023] Based on all the neural nodes in each of the structural blocks in the input neural network, obtaining a plurality of first candidate structural blocks of the structural block by adding neural nodes and reducing neural nodes;

[0024] Based on the first predetermined number and the multiple first candidate structural blocks of each structural block, multiple initial neural networks of the current round are obtained by combination.

[0025] Furthermore, the step of obtaining multiple second neural networks of the current round by adding neural nodes and reducing neural nodes in each of the multiple first neural networks includes:

[0026] The following operations are performed for each of the structural blocks in each of the first neural networks:

[0027] Adding a new neural node to the structural block, and determining a plurality of newly added structural blocks of the structural block based on the adding position of the new neural node, all the operation types contained in the new neural node, and the connection mode with the neural nodes in the structural block;

[0028] Select one neural node from all neural nodes in the structural block and delete it to obtain multiple branch-reduced structural blocks of the structural block;

[0029] Merging the multiple newly added structural blocks and the multiple branch-reduced structural blocks as multiple second candidate structural blocks of the structural block;

[0030] Based on the first predetermined number and all of the plurality of second candidate structural blocks, the plurality of second neural networks of the current round are determined.

[0031] Furthermore, before constructing the evaluation index function by using the accuracy and memory consumption of each second neural network during operation in the current round, the method includes:

[0032] Input each of the second neural networks in the current round into a preset structure controller;

[0033] Using the encoding layer in the structure controller to convert each structural feature of the second neural network into a corresponding network feature;

[0034] inputting the network features into a sorting layer in the fabric controller;

[0035] The step of constructing an evaluation index function using the accuracy and memory consumption of each second neural network during operation in the current round includes:

[0036] Determine the accuracy and memory consumption of each of the second neural networks during execution in this round;

[0037] Determine the accuracy and memory consumption of the target neural network runtime in the previous round;

[0038] For each of the second neural networks, an accuracy evaluation item is determined using the accuracy of the second neural network and the accuracy of the target neural network in the previous round, and a memory consumption evaluation item is determined using the memory consumption of the second neural network and the memory consumption of the target neural network in the previous round;

[0039] The accuracy evaluation item and the memory consumption evaluation item are fused according to a preset weight coefficient to obtain an evaluation index function for the second neural network.

[0040] Furthermore, converting the structural features of each of the second neural networks into corresponding network features includes:

[0041] For each structural block in each of the second neural networks, the position, connection mode and all the operation types contained in each neural node in the structural block are respectively uniquely encoded to obtain the block structure network features of each structural block;

[0042] All of the block structure network features of the second neural network are input into the encoding layer to obtain the network features of the second neural network.

[0043] Furthermore, the use of the evaluation index function to construct a ranking loss function, and the use of the ranking loss function to determine multiple third neural networks in the current round, includes:

[0044] For each of the second neural networks, the following operations are performed:

[0045] determining a true positional ranking of the second neural network;

[0046] Determine a relative position ranking between the second neural network and each of all other second neural networks;

[0047] Using the relative position ranking and the real position ranking, constructing a ranking loss function;

[0048] Based on the ranking loss function, a second predetermined number of the plurality of third neural networks is determined from all of the second neural networks.

[0049] Furthermore, the bidirectional distillation migration comprises:

[0050] Determining a target probability distribution of the auxiliary neural network and / or the target neural network of the previous round;

[0051] determining a candidate probability distribution for each of the third neural networks;

[0052] Calculating relative entropy and cross entropy between the target probability distribution and the candidate probability distribution;

[0053] A training loss function is constructed according to the relative entropy and the cross entropy, and the third neural network of the current round is trained using the training loss function.

[0054] Based on the same inventive concept, the present application also provides a neural network optimization device for image processing, comprising:

[0055] An iteration module, wherein the iteration module includes a branch adding and reducing submodule, a sorting submodule and a bidirectional distillation submodule;

[0056] The iteration module is configured to obtain a plurality of pre-selected neural networks, perform multiple rounds of iterations on all of the pre-selected neural networks, obtain a plurality of candidate neural networks and a target neural network in each round, and output the target neural network obtained in the last round in response to determining that a preset first iteration indicator is reached;

[0057] The adding and reducing branch submodule is configured to, in response to the current round being the first round, use the plurality of pre-selected neural networks as the plurality of first neural networks of the current round,

[0058] In response to the current round not being the first round, using the plurality of candidate neural networks of the previous round as the plurality of first neural networks of the current round;

[0059] By adding neural nodes and reducing neural nodes in each of the plurality of first neural networks, a plurality of second neural networks of the current round are obtained;

[0060] The sorting submodule is configured to construct an evaluation index function using the accuracy and memory consumption of each second neural network in the current round, construct a sorting loss function using the evaluation index function, and determine multiple third neural networks in the current round using the sorting loss function;

[0061] The bidirectional distillation submodule is configured to, in response to the current round being the first round, determine an auxiliary neural network from the multiple pre-selected neural networks, use the auxiliary neural network to perform bidirectional distillation migration on the multiple third neural networks of the current round, and in response to the current round not being the first round, use the target neural network of the previous round to perform bidirectional distillation migration on the multiple third neural networks of the current round, use the result of the bidirectional distillation migration as the multiple candidate neural networks of the current round, and determine the target neural network of the current round from the multiple candidate neural networks.

[0062] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the neural network optimization method for image processing as described in any one of the above items is implemented.

[0063] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the neural network optimization method for image processing as described above.

[0064] From the above, it can be seen that the neural network optimization method and related devices for image processing provided by the present application optimize the performance of the target neural network based on multiple rounds of iterations of the pre-selected neural network. The structure and accuracy of the target neural network are comprehensively considered in each round of iteration. The structure of the target neural network is changed by adding and reducing neural nodes in the iteration, so that it has both more complex structures and more lightweight structures in terms of structural possibilities. The sorting loss function is constructed through the evaluation index function, so that the sorting process integrates the structure and accuracy, and the relative position sorting is used instead of the real absolute sorting in the sorting process, so that the amount of calculation is greatly reduced, and the third neural network selected by sorting is assisted in learning based on the target neural network in the previous round, so as to realize bidirectional distillation migration, thereby realizing the selection of the target neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1A It is a schematic diagram of the iteration principle of an embodiment of the present application;

[0067] Figure 1B It is a flowchart of the iterative process of the neural network optimization method for image processing according to an embodiment of the present application;

[0068] Figure 2 This is a schematic diagram of the effect of the branch adding and reducing strategy of the embodiment of the present application;

[0069] Figure 3 A schematic diagram of the execution of bidirectional distillation migration according to an embodiment of the present application;

[0070] Figure 4A schematic diagram of an iterative module of a neural network optimization device for image processing according to an embodiment of the present application;

[0071] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0073] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0074] As described in the background technology section, the related neural network optimization methods for image processing are still difficult to meet the needs of screening image data in actual production.

[0075] In the process of implementing this application, the applicant discovered that the main problem with the related neural network optimization methods for image processing is that although neural networks are currently used in various fields of computer vision and have gradually evolved into more complex structures, while their accuracy has been improved, their operation also requires greater computing power and resources.

[0076] However, the computing resources of the platforms that carry the neural networks for related image processing vary. Some platforms have multiple GPU devices that can provide rich computing and storage resources, but some other platforms, such as most mobile device platforms, can only provide limited memory and computing resources, and can often only use the CPU or limited on-chip memory resources. When such platforms carry large network models composed of complex neural networks, their complex structures lead to huge resource usage, and it is often difficult to quickly process related image data. When the structure of the neural network becomes more lightweight, it is difficult to achieve the expected accuracy.

[0077] The applicant also found in the research that when it is necessary to obtain a neural network that meets the requirements of low forward operation memory usage and good network representation ability, it is necessary to test and fine-tune a large number of neural networks, and the workload is enormous. However, in the actual selection, the balance degree of each neural network can also be determined according to the relative position relationship between each neural network, instead of training the neural networks one by one.

[0078] It can be understood that the method can be executed by any device, equipment, platform, or device cluster having computing and processing capabilities.

[0079] The technical method of the present application is described in detail below through specific embodiments.

[0080] In one embodiment of the present application, a neural network optimization method for image processing includes: obtaining multiple pre-selected neural networks, performing multiple rounds of iterations on all of the pre-selected neural networks, obtaining multiple candidate neural networks and a target neural network in each round, and in response to determining that a preset first iteration indicator has been reached, outputting the target neural network obtained in the last round.

[0081] In this embodiment, the multiple pre-selected neural networks are specifically neural networks for processing image data. In the framework of communication between the cloud-side platform and the terminal-side platform, the pre-selected neural network can be set on the cloud testing platform or on the terminal-side platform. In other words, the method is not limited to which platform or processor in the platform. The pre-selected neural network can be prepared in advance or generated by adding or subtracting branches iteratively or in other ways.

[0082] Furthermore, based on the obtained multiple pre-selected neural networks, multiple rounds of iterations are performed on them, such as Figure 1A As shown in the figure, in each round of iteration, multiple candidate neural networks and a target neural network are generated by executing branch addition and reduction strategies, sorting, and bidirectional distillation migration.

[0083] Among them, in any round, the multiple candidate neural networks obtained from the previous round of iteration are used as the input of the iterative process of this round, and the target neural network obtained from the previous round is used as an auxiliary to perform branch addition and reduction strategies, sorting and bidirectional distillation migration on the candidate neural networks of the previous round.

[0084] Furthermore, when the preset first iteration index is reached, the iteration can be stopped, and the target neural network obtained in the last round is output as the result and used as the final selected neural network for image processing.

[0085] The preset first iteration indicator may be reaching a preset number of iterations, or that the accuracy of image data processing of the target neural network obtained in this round meets a preset accuracy threshold, or that the structure of the target neural network obtained in this round meets the expected lightweight structure in terms of memory consumption, or that the accuracy and memory consumption of the target neural network obtained in this round reach an expected balance.

[0086] In another embodiment of the present application, the obtaining of a plurality of pre-selected neural networks comprises:

[0087] constructing a first predetermined number of building blocks;

[0088] The following operation is performed for each of the structural blocks:

[0089] Using each operation type in the preset operation type set as the operation type of the neural layer to obtain a plurality of neural nodes, each of the neural nodes includes two neural layers, each of the neural layers corresponds to one operation type, and any two neural nodes in the plurality of neural nodes include different operation types;

[0090] Select any one neural node from the plurality of neural nodes and place it in the structural block;

[0091] Cascading all the structural blocks to obtain an initial backbone neural network;

[0092] Perform multiple rounds of iterative addition and subtraction of branches on the initial backbone neural network, and obtain multiple initial neural networks of the round in each round;

[0093] In response to determining that a preset second iteration indicator is reached, outputting a plurality of initial neural networks of a final round as the plurality of pre-selected neural networks;

[0094] The following operations are performed for each of the multiple rounds of adding and reducing branches iterations:

[0095] In response to the current round being the first round, the initial trunk neural network is used as the input neural network of the current round; in response to the current round being not the first round, multiple initial neural networks of the previous round are used as the input neural network of the current round;

[0096] Based on all the neural nodes in each of the structural blocks in the input neural network, obtaining a plurality of first candidate structural blocks of the structural block by adding neural nodes and reducing neural nodes;

[0097] Based on the first predetermined number and the multiple first candidate structural blocks of each structural block, multiple initial neural networks of the current round are obtained by combination.

[0098] In this embodiment, the structure of the preselected neural network to be constructed is first determined, and each preselected neural network may have: multiple structural blocks, each block structure is cascaded, and the predetermined number of structural blocks is used as the first predetermined number in this embodiment; each structural block may have: at least one initial neural node in the structure, and any two initial neural nodes may be in a cascade relationship or a parallel relationship; each initial neural node may have: two neural layers, and one operation type is configured for each neural layer to perform one neural operation, wherein there may be multiple operation types for neural operations, and all operation types are used as the operation type set in this embodiment.

[0099] Furthermore, based on the structure of the pre-determined neural network, a specific construction process can be performed: constructing an initial backbone neural network, and iteratively adding and removing branches from the initial backbone neural network to obtain an initial neural network.

[0100] Specifically, first, the first predetermined number of the structural blocks can be set to 5 in this embodiment, and the 5 structural blocks are connected in a cascade manner.

[0101] For each of these blocks, do the following:

[0102] One neural node is set in each structural block, and two neural layers are set in the neural node.

[0103] Specifically, for each neural node, 2 operation types are selected from a preset set of operation types, and 1 operation type is configured for each neural layer. In a specific example, the preset set of operation types may include 7 operation types, such as 3x3 convolution operation layer, 3x3 maximum pooling layer, 3x3 average pooling layer, 3x3 hole convolution operation layer, 3x3 depth-wise separable convolution operation layer, 5x5 depth-wise separable convolution operation layer, and 1x7 cascaded 7x1 convolution operation layer.

[0104] In this embodiment, there are multiple possible combinations of the operation types possessed by the two neural layers. Therefore, based on all possible combinations, multiple neural nodes can be obtained, and the combinations of operation types included in each neural node are different, and the two operation types included in each neural node are also different.

[0105] Furthermore, among the multiple neural nodes obtained by combining the operation types, one is randomly selected and placed into a structural block. Therefore, the neural nodes included in the structural block have multiple possibilities.

[0106] Furthermore, the possibilities of each structural block are fully combined to obtain all the possibilities of combination between the structural blocks, and each possibility of combination between the structural blocks is used as an initial backbone neural network.

[0107] In a specific example, for one neural node, by selecting any two of the seven operation types, there are a total of 42 possible combinations, which means that 42 different neural nodes can be obtained based on the combination of their operation types.

[0108] Furthermore, at this stage, since each structural block contains only one neural node, there are 42 ways to place one neural node in each structural block, that is, there are 42 possibilities for structural blocks.

[0109] Furthermore, taking the five structural blocks set above as an example, the 42 possibilities of each structural block are combined according to the number of five structural blocks to obtain all the possibilities of combining the five structural blocks, where each combination is used as an initial backbone neural network. Based on all the possibilities of combining the structural blocks, multiple initial backbone neural networks can be obtained.

[0110] Furthermore, taking the initial backbone neural network as a basis, multiple rounds of branch addition and subtraction iterations are performed by adding neural nodes and reducing neural nodes in each structural block, and multiple initial neural networks of that round are obtained in each round.

[0111] Specifically, in each round of branch addition and subtraction iteration, it is first determined whether the current round is the first round. If this round is the first round, the multiple initial trunk neural networks obtained above are used as the multiple input neural networks in the first round of branch addition and subtraction iteration. If this round is not the first round, the multiple initial neural networks obtained in the previous round are used as the multiple input neural networks in the current round.

[0112] Furthermore, a branch increase and decrease strategy is executed for each structural block in all input neural networks. Specifically, all possibilities of newly added structural blocks that can be obtained by adding one neural node to the structural block, and all possibilities of reduced branch structural blocks that can be obtained by reducing one neural node are judged, and all possibilities of newly added structural blocks are added to all possibilities of reduced branch structural blocks as all first candidate structural blocks of the structural block.

[0113] Specifically, the process of obtaining all possibilities of adding new structural blocks includes:

[0114] First, in each structural block, the connection positions where neural nodes can be added are determined. The newly added neural nodes can be connected to any existing neural nodes in the structural block. Based on this, all possible connection positions can be determined according to the existing neural nodes in the structural block, and the number of possible connection positions can be obtained.

[0115] Furthermore, in this embodiment, the newly added neural nodes and the existing neural nodes in the structural block may be connected in parallel or in cascade, with a total of two possible connection modes.

[0116] Furthermore, all possibilities of combining the above two neural operations, all possibilities of connection positions, and two possible connection methods are combined together to obtain all possibilities of newly added structural blocks that can be obtained after adding one neural node to the structural block.

[0117] In this embodiment, the process of obtaining all possibilities of the branch-reducing structural block includes:

[0118] First, based on the existing neural nodes in each structural block, eliminating any one of the neural nodes can obtain a possibility of generating a reduced-branch structural block. Therefore, all possibilities of the reduced-branch structural blocks that can be obtained after eliminating one neural node in the structural block can be determined based on the number of existing neural nodes.

[0119] Furthermore, all possibilities of the above-mentioned newly added structural blocks are added to all possibilities of the branch-reduced structural blocks to obtain all possibilities of new structural blocks that can be generated after executing the branch-adding and branch-reducing strategies on the structural block, and each possibility is used as a first candidate structural block to obtain all first candidate structural blocks of the structural block.

[0120] Furthermore, after executing the above-mentioned branch addition and reduction strategy for each structural block in all input neural networks, the first candidate structural block of all structural blocks can be obtained.

[0121] Furthermore, according to the above-mentioned first predetermined number, all first candidate structural blocks are fully combined and cascaded to obtain multiple initial neural networks.

[0122] In the above specific example, all first candidate structural blocks need to be grouped and cascaded in groups of five to obtain multiple initial neural networks of the specific example.

[0123] Furthermore, when a predetermined second iteration index is reached, the iteration may be stopped, and the multiple initial neural networks obtained in the last round may be used as multiple pre-selected neural networks.

[0124] Among them, the second iteration indicator can be a preset number of iterations. In the above specific example, two rounds are used as the preset number of iterations for adding and reducing branches, so that a local neural network search space with a certain scale can be obtained, that is, all the pre-selected neural networks.

[0125] refer to Figure 1B In another embodiment of the neural network optimization method for image processing of the present application, the following steps are performed for each of the multiple iterations:

[0126] Step S101: In response to the current round being the first round, the multiple pre-selected neural networks are used as the multiple first neural networks of the current round; in response to the current round not being the first round, the multiple candidate neural networks of the previous round are used as the multiple first neural networks of the current round; and multiple second neural networks of the current round are obtained by adding neural nodes and reducing neural nodes in each of the multiple first neural networks.

[0127] In this step, it is first determined whether the current round is the first round. If this round is the first round, the multiple pre-selected neural networks in the aforementioned embodiment are used as the multiple first neural networks in the current round. If this round is not the first round, the multiple candidate neural networks obtained in the previous round are used as the multiple first neural networks in the current round.

[0128] Furthermore, a branch increase and decrease strategy is executed for each structural block in all first neural networks of the current round. Specifically, all possibilities of newly added structural blocks that can be obtained by adding one neural node to the structural block, and all possibilities of reduced branch structural blocks that can be obtained by reducing one neural node are judged, and all possibilities of newly added structural blocks are added to all possibilities of reduced branch structural blocks as all first candidate structural blocks of the structural block.

[0129] Specifically, the process of obtaining all possibilities of adding new structural blocks includes:

[0130] First, in each structural block, the connection positions where neural nodes can be added are determined. The newly added neural nodes can be connected to any existing neural nodes in the structural block. Based on this, all possible connection positions can be determined according to the existing neural nodes in the structural block, and the number of possible connection positions can be obtained.

[0131] Furthermore, in this embodiment, the newly added neural nodes and the existing neural nodes in the structural block may be connected in parallel or in cascade, with a total of two possible connection modes.

[0132] Furthermore, all possibilities of combining the two neural operations in the aforementioned embodiment, all connection position possibilities, and the two possible connection methods are combined together to obtain all possibilities of newly added structural blocks that can be obtained after adding one neural node to the structural block.

[0133] In this embodiment, the process of obtaining all possibilities of the branch-reducing structural block includes:

[0134] First, based on the existing neural nodes in each structural block, eliminating any one of the neural nodes can obtain a possibility of generating a reduced-branch structural block. Therefore, all possibilities of the reduced-branch structural blocks that can be obtained after eliminating one neural node in the structural block can be determined based on the number of existing neural nodes.

[0135] Furthermore, all possibilities of newly added structural blocks in this embodiment are added to all possibilities of branch-reduced structural blocks to obtain all possibilities of new structural blocks that can be generated after executing the branch-adding and branch-reducing strategies on the structural block, and each possibility is used as a second candidate structural block to obtain all second candidate structural blocks of the structural block.

[0136] Furthermore, after executing the branch adding and reducing strategy in this embodiment for each structural block in all the first neural networks, the second candidate structural blocks of all the structural blocks can be obtained.

[0137] Furthermore, according to the first predetermined number in the aforementioned embodiment, all the second candidate structural blocks are fully combined and cascaded to obtain multiple second neural networks.

[0138] In the specific example of the aforementioned embodiment, all the second candidate structural blocks need to be grouped and cascaded in groups of 5 to obtain multiple second neural networks of the specific example.

[0139] like Figure 2 As shown, the newly added structural blocks obtained after adding new neural nodes can improve the representation ability of the second neural network, that is, improve its prediction accuracy; and the reduced branch structural blocks obtained after eliminating existing neural nodes can reduce the memory consumption of the second neural network and reduce the computing resources it occupies.

[0140] Step S102: construct an evaluation index function using the accuracy and memory consumption of each of the second neural networks in the current round, construct a ranking loss function using the evaluation index function, and determine multiple third neural networks in the current round using the ranking loss function.

[0141] In this step, a structure controller is first pre-set. In this embodiment, the pre-set structure controller includes: an embedding layer, a coding layer and a sorting layer.

[0142] Among them, the embedding layer is configured to convert the structural features of each structural block of the second neural network input therein into corresponding network features, and the embedding layer can also be called a block structure encoder.

[0143] The encoding layer is configured to encode the network features of each structural block into the network features of each corresponding second neural network. The encoding layer is composed of a layer of recurrent neural network, which can also be called a network feature extractor.

[0144] The sorting layer is configured to use the network features to sort the relative positions of the second neural network input therein. The sorting layer is also composed of a layer of recurrent neural network, which can also be called a predictor.

[0145] Furthermore, based on the structural controller set up as above, the second neural network input therein is sorted.

[0146] In this embodiment, the second neural network input to the structure controller will first enter the embedding layer, in which the vector form of the structural features of each structural block in the second neural network is converted into bit form of block structure network features by using one-hot encoding so that the structure controller can recognize it.

[0147] Among them, the structural characteristics of each structural block include the structural characteristics of each neural node in the structural block, and the structural characteristics of each neural node include: connection position, connection method and all operation types contained in the neural node. That is to say, to obtain the network characteristics of each structural block, it is necessary to first obtain the network characteristics of each neural node in the structural block. It should be noted that when the structural characteristics are mapped to network characteristics using the one-hot encoding method, the mapping relationship can be customized according to specific circumstances and needs.

[0148] Specifically, in the specific example of this embodiment, for each neural node in each structural block, the position of its input data represents the source of the input data of the two neurons in the neural node, and then the connection position of the neural node can be indicated according to the source of its input data, and after one-hot encoding, it is represented as I_{i} and I_{j} in this embodiment, where I_{i} and I_{j} can be multiple bits such as 0010 and 0001, and the length and category value of its one-hot encoding are defined by the output of the neural nodes already existing in the structural block and the output of the previous adjacent structural block to the structural block.

[0149] Furthermore, the connection mode of each neural node in the structural block includes the above-mentioned cascade and parallel, and is represented as O_{i} in the present embodiment, wherein O_{i} can be four bits such as 1010. Specifically, among the above-mentioned 4 bits, the first two bits represent the connection mode of the operations of the two neural layers in the neural node at the output, wherein "01" represents the addition of the operations of the two neural layers in the neural node at the output, that is, parallel connection, and "10" represents the cascading of the operations of the two neural layers at the output; further, the last two bits represent whether the output of the neural node is cascaded in the output stage of the structural block, for example, "01" represents that the output of the neural node will not be cascaded in the output data of the structural block, and "10" represents that the output of the neural node will be cascaded in the output data of the structural block.

[0150] Further, as described in the aforementioned embodiment, each neural node in the structural block may include two operation types in the operation type set. In this embodiment, the two operation types after one-hot encoding are represented as L_{i} and L_{j}.

[0151] Among them, the mapping relationship between the operation type and each operation type in the set during one-hot encoding is: 3x3 convolution operation layer is represented as 0000001, 3x3 maximum pooling layer is represented as 0100000, 3x3 average pooling layer is represented as 0010000, 3x3 void convolution operation layer is represented as 1000000, 3x3 depthwise separable convolution operation layer is represented as 0000010, 5x5 depthwise separable convolution operation layer is represented as 0000100 and 1x7 cascaded 7x1 convolution operation layer is represented as 0001000.

[0152] Further, in the embedding layer, after each neural node in each structural block is one-hot encoded based on the above-mentioned mapping relationship, a network feature representing all the neural nodes in the structural block is obtained, and is expressed in this embodiment as: "\{B_{1}, B_{2}, ..., B_{5}\}_{i}", wherein B_{1} to B_{5} respectively represent the one-hot encoding of all the neural nodes in each structural block from the 1st structural block to the 5th structural block in the second neural network, that is, the block structure network feature of the second neural network, and i indicates that the second neural network is the i-th among all second neural networks.

[0153] Furthermore, the block structure network features of the second neural network are input into the encoding layer one by one. For each second neural network input therein, its five block structure network features are encoded into one network feature representing the second neural network, and represented as F_{i} in this embodiment.

[0154] Furthermore, the above encoding operation is repeated for each second neural network of the input encoding layer until the network characteristics of all second neural networks are obtained, which are expressed as "F_{i}\in \{F_{1},F_{2},...,F_{n}\}". Wherein, i indicates that the second neural network is the i-th one among all second neural networks, and F_{1} to F_{n} respectively represent the network characteristics of the first second neural network to the n-th second neural network.

[0155] Furthermore, the network features of each second neural network are input into the sorting layer, and all the second neural networks are sorted.

[0156] Specifically, first, the preselected neural network obtained in the aforementioned embodiment is used to perform image processing, and the sorting layer of the structure controller is trained based on the actual results of the image processing, such as accuracy and memory consumption. When training the structure controller, since the number and structure types of the second neural networks in this embodiment are constantly increasing in large quantities, it will be very difficult to use the trained controller to predict the results of each second neural network one by one, and then sort them according to their predicted results.

[0157] Therefore, in the sorting layer, the relative position of the second neural network is predicted, and the relative position ranking of each second neural network in the current round is directly output, and the sorting process is converted into a sequential selection process to solve the above difficulties.

[0158] Specifically, a list-based sorting loss function ListMLE is constructed and expressed as the following formula:

[0159]

[0160] in, represents the true position order of all second neural networks from the mth second neural network to the nth second neural network, Indicates that the i-th second neural network f i The probability of being selected as the target neural network from all the second neural networks from the mth to the nth, y i This represents the second neural network f i evaluation indicators.

[0161] in, It can be expressed as the following formula:

[0162]

[0163] Among them, S i and S j Together they represent the i-th second neural network f iand the jth second neural network f j The relative position between them.

[0164] Furthermore, for the evaluation index of the second neural network, each second neural network can be used to process the image data according to the performance of the second neural network when processing the image data.

[0165] In this embodiment, the evaluation index is calculated by an evaluation index function, which consists of two parts: the accuracy of the second neural network in processing image data, and the memory consumption representing the lightweight degree of the second neural network structure. Specifically, the evaluation index function is shown in the following formula:

[0166]

[0167] Among them, a represents the accuracy of the second neural network when processing image data; r represents the intermediate value memory size occupied by the second neural network during forward operation; t represents the memory size occupied by each parameter in the second neural network; the subscript i of a, r and t represents the i-th second neural network in the current round, and the subscript pre represents the best candidate neural network in the previous round, that is, the target neural network.

[0168] In this embodiment, the first term related to a in the above evaluation index function expression is used as the accuracy evaluation term in the evaluation index function, the second term related to r and t is used as the memory consumption evaluation term in the evaluation index function, and As a weight coefficient between the accuracy evaluation item and the memory consumption evaluation item, in this embodiment, the weight coefficient is customized and is used to balance the emphasis on the accuracy and memory consumption of the second neural network in the evaluation. When the weight coefficient is 0, it means that only a single optimization selection is made with memory consumption as the target, and when the weight coefficient is 1, it means that only a single optimization selection is made with accuracy as the target.

[0169] Furthermore, by optimizing the above-mentioned sorting loss function, the closer the relative position sorting obtained is to the actual position sorting, the smaller the loss of the sorting loss function is.

[0170] Furthermore, for each second neural network, the loss is calculated by a ranking loss function, and the first multiple neural networks with the smallest losses are determined as the third neural network.

[0171] Among them, the first multiple can be used as the second predetermined number in this embodiment, and can be determined according to specific circumstances. In a specific example, it can be the first 50.

[0172] Step S103: In response to the current round being the first round, an auxiliary neural network is determined from the multiple pre-selected neural networks, and the multiple third neural networks of the current round are bidirectionally distilled and migrated using the auxiliary neural network; in response to the current round being not the first round, the multiple third neural networks of the current round are bidirectionally distilled and migrated using the target neural network of the previous round, and the results of the bidirectional distillation and migration are used as the multiple candidate neural networks of the current round, and the target neural network of the current round is determined from the multiple candidate neural networks.

[0173] In this step, based on the structural correlation between the third neural network and the local optimal neural network, a bidirectional distillation migration method is adopted to train the multiple third neural networks obtained above, and determine the target neural network, wherein the bidirectional distillation migration method adopted requires the use of a local optimal neural network as an auxiliary to the third neural network.

[0174] Further, determining the local optimal neural network includes:

[0175] First, determine whether the current round is the first round. If it is the first round, select one auxiliary neural network from the multiple pre-selected neural networks in the aforementioned embodiments as the local optimal neural network in the bidirectional distillation migration for auxiliary training; if the current round is not the first round, extract the target neural network obtained in the previous round as the local optimal neural network in the bidirectional distillation migration for auxiliary training.

[0176] Among them, the process of selecting one auxiliary neural network from the multiple pre-selected neural networks in the aforementioned embodiments can be determined based on the specific performance and resource consumption of the pre-selected neural network when processing image data.

[0177] Furthermore, if Figure 3 As shown, the determined local optimal neural network and all third neural networks are input into the model training framework of bidirectional distillation transfer, and the transfer loss function is constructed in the framework.

[0178] In this embodiment, first, a transfer loss function is constructed to perform unidirectional knowledge transfer training from the local optimal neural network to the third neural network. Specifically, for each third neural network, relative entropy, that is, KL (Kullback Leibler) divergence, is used as a transfer loss function to match the probability distribution of the third neural network and the local optimal neural network, and is expressed as the following formula:

[0179]

[0180] Among them, p represents the third neural network, p irepresents the probability distribution of the third neural network in the current round; q represents the local optimal neural network, q i It represents the probability distribution of the local optimal neural network.

[0181] Furthermore, in the one-way knowledge transfer process, if Figure 3 As shown, each structural block can be used as the minimum unit of knowledge transfer. Therefore, in the above transfer loss function, p(x i ) and q(x i ) represent the i-th structural block in the third neural network and the local optimal neural network respectively.

[0182] Furthermore, based on the above one-way knowledge transfer, the cross entropy loss function L between the third neural network and the local optimal neural network can be CE as a supervised loss function.

[0183] Furthermore, by combining the transfer loss function and the cross entropy loss function, we can get the training loss function, which is expressed as L P , to achieve bidirectional knowledge transfer in the model training framework of bidirectional distillation transfer.

[0184] Among them, the training loss function is specifically expressed as:

[0185] L P =L CE +D KL (q i ||p i )

[0186] Furthermore, based on the application of the above-mentioned training loss function, the representation information contained in the local optimal neural network can be migrated to the third neural network of the current round in the model training framework of bidirectional distillation migration, which effectively accelerates the training process of the third neural network of the current round.

[0187] Furthermore, when the convergence degree of the training loss function reaches a custom-set relevant threshold, the training of the bidirectional distillation migration can be terminated, and the trained third neural network is output, and the third neural network output at this time is used as the candidate neural network for this round.

[0188] Furthermore, among the candidate neural networks of the current round, the target neural network of the current round is determined according to the accuracy and memory consumption when processing image data and the preset relevant thresholds, and the candidate neural networks and target neural networks of the current round are input into the next round.

[0189] It can be seen that the neural network optimization method for image processing in the embodiment of the present application optimizes the performance of the target neural network based on multiple rounds of iterations of the pre-selected neural network. In each round of iteration, the structure and accuracy of the target neural network are comprehensively considered. In the iteration, the structure of the target neural network is changed by adding and reducing neural nodes, so that the structure may have both a more complex structure and a lighter structure. The sorting loss function is constructed through the evaluation index function, so that the sorting process combines the structure and accuracy. In the sorting process, the relative position sorting is used instead of the real absolute sorting, which greatly reduces the amount of calculation. The third neural network selected by the sorting is assisted in learning based on the target neural network in the previous round, so as to realize bidirectional distillation migration, thereby realizing the selection of the target neural network.

[0190] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0191] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0192] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides a device for optimizing a neural network for image processing.

[0193] refer to Figure 4 , the neural network optimization device for image processing comprises: an iteration module 401, wherein the iteration module comprises a branch increase and decrease submodule 401-1, a sorting submodule 401-2 and a bidirectional distillation submodule 401-3;

[0194] The iteration module 401 is configured to obtain a plurality of pre-selected neural networks, perform multiple rounds of iterations on all of the pre-selected neural networks, obtain a plurality of candidate neural networks and a target neural network in each round, and output the target neural network obtained in the last round in response to determining that a preset first iteration indicator has been reached.

[0195] The adding and reducing branch module 401-1 is configured to, in response to the current round being the first round, use the multiple pre-selected neural networks as multiple first neural networks of the current round;

[0196] In response to the current round not being the first round, using the plurality of candidate neural networks of the previous round as the plurality of first neural networks of the current round;

[0197] By adding neural nodes and reducing neural nodes in each of the multiple first neural networks, multiple second neural networks of the current round are obtained.

[0198] The sorting submodule 401-2 is configured to construct an evaluation index function using the accuracy and memory consumption of each of the second neural networks in the current round, construct a sorting loss function using the evaluation index function, and determine multiple third neural networks in the current round using the sorting loss function.

[0199] The bidirectional distillation submodule 401-3 is configured to, in response to the current round being the first round, determine an auxiliary neural network from the multiple pre-selected neural networks, and use the auxiliary neural network to perform bidirectional distillation migration on the multiple third neural networks of the current round; in response to the current round not being the first round, use the target neural network of the previous round to perform bidirectional distillation migration on the multiple third neural networks of the current round, use the result of the bidirectional distillation migration as the multiple candidate neural networks of the current round, and determine the target neural network of the current round from the multiple candidate neural networks.

[0200] As an optional embodiment, the branch adding and reducing module 401-1 is specifically configured as follows:

[0201] constructing a first predetermined number of building blocks;

[0202] The following operation is performed for each of the structural blocks:

[0203] Using each operation type in the preset operation type set as the operation type of the neural layer to obtain a plurality of neural nodes, each of the neural nodes includes two neural layers, each of the neural layers corresponds to one operation type, and any two neural nodes in the plurality of neural nodes include different operation types;

[0204] Select any one neural node from the plurality of neural nodes and place it in the structural block;

[0205] Cascading all the structural blocks to obtain an initial backbone neural network;

[0206] Perform multiple rounds of iterative addition and subtraction of branches on the initial backbone neural network, and obtain multiple initial neural networks of the round in each round;

[0207] In response to determining that a preset second iteration indicator is reached, outputting a plurality of initial neural networks of a final round as the plurality of pre-selected neural networks;

[0208] The following operations are performed for each of the multiple rounds of adding and reducing branches iterations:

[0209] In response to the current round being the first round, the initial trunk neural network is used as the input neural network of the current round; in response to the current round being not the first round, multiple initial neural networks of the previous round are used as the input neural network of the current round;

[0210] Based on all the neural nodes in each of the structural blocks in the input neural network, obtaining a plurality of first candidate structural blocks of the structural block by adding neural nodes and reducing neural nodes;

[0211] Based on the first predetermined number and the multiple first candidate structural blocks of each structural block, multiple initial neural networks of the current round are obtained by combination.

[0212] The following operations are performed for each of the structural blocks in each of the first neural networks:

[0213] Adding a new neural node to the structural block, and determining a plurality of newly added structural blocks of the structural block based on the adding position of the new neural node, all the operation types contained in the new neural node, and the connection mode with the neural nodes in the structural block;

[0214] Select one neural node from all neural nodes in the structural block and delete it to obtain multiple branch-reduced structural blocks of the structural block;

[0215] Merging the multiple newly added structural blocks and the multiple branch-reduced structural blocks as multiple second candidate structural blocks of the structural block;

[0216] Based on the first predetermined number and all of the plurality of second candidate structural blocks, the plurality of second neural networks of the current round are determined.

[0217] As an optional embodiment, the sorting submodule 401-2 is specifically configured as follows:

[0218] Input each of the second neural networks in the current round into a preset structure controller;

[0219] Using the encoding layer in the structure controller to convert each structural feature of the second neural network into a corresponding network feature;

[0220] inputting the network features into a sorting layer in the fabric controller;

[0221] The step of constructing an evaluation index function using the accuracy and memory consumption of each second neural network during operation in the current round includes:

[0222] Determine the accuracy and memory consumption of each of the second neural networks during execution in this round;

[0223] Determine the accuracy and memory consumption of the target neural network runtime in the previous round;

[0224] For each of the second neural networks, an accuracy evaluation item is determined using the accuracy of the second neural network and the accuracy of the target neural network in the previous round, and a memory consumption evaluation item is determined using the memory consumption of the second neural network and the memory consumption of the target neural network in the previous round;

[0225] The accuracy evaluation item and the memory consumption evaluation item are fused according to a preset weight coefficient to obtain an evaluation index function for the second neural network.

[0226] The step of converting each structural feature of the second neural network into a corresponding network feature includes:

[0227] For each structural block in each of the second neural networks, the position, connection mode and all the operation types contained in each neural node in the structural block are respectively uniquely encoded to obtain the block structure network features of each structural block;

[0228] All of the block structure network features of the second neural network are input into the encoding layer to obtain the network features of the second neural network.

[0229] Furthermore, the use of the evaluation index function to construct a ranking loss function, and the use of the ranking loss function to determine multiple third neural networks in the current round, includes:

[0230] For each of the second neural networks, the following operations are performed:

[0231] determining a true positional ranking of the second neural network;

[0232] Determine a relative position ranking between the second neural network and each of all other second neural networks;

[0233] Using the relative position ranking and the real position ranking, constructing a ranking loss function;

[0234] Based on the ranking loss function, a second predetermined number of the plurality of third neural networks is determined from all of the second neural networks.

[0235] As an optional embodiment, the bidirectional distillation submodule 401-3 is specifically configured as follows:

[0236] Determining a target probability distribution of the auxiliary neural network and / or the target neural network of the previous round;

[0237] determining a candidate probability distribution for each of the third neural networks;

[0238] Calculating relative entropy and cross entropy between the target probability distribution and the candidate probability distribution;

[0239] A training loss function is constructed according to the relative entropy and the cross entropy, and the third neural network of the current round is trained using the training loss function.

[0240] For the convenience of description, the above devices are described in terms of functions and are divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0241] The device of the above embodiment is used to implement the corresponding neural network optimization method for image processing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0242] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the neural network optimization method for image processing as described in any of the above embodiments is implemented.

[0243] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0244] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0245] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0246] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0247] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0248] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0249] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0250] The device of the above embodiment is used to implement the corresponding neural network optimization method for image processing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0251] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the neural network optimization method for image processing as described in any of the above embodiments.

[0252] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0253] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the neural network optimization method for image processing as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0254] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0255] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram so as to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0256] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0257] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A neural network optimization method for image processing, comprising: Acquire multiple pre-selected neural networks, perform multiple rounds of iterations on all of the pre-selected neural networks, obtain multiple candidate neural networks and a target neural network in each round, and output the target neural network obtained in the last round in response to determining that a preset first iteration indicator is reached; The following operations are performed for each of the multiple rounds of iterations: In response to the current round being the first round, using the plurality of pre-selected neural networks as a plurality of first neural networks of the current round; In response to the current round not being the first round, using the plurality of candidate neural networks of the previous round as the plurality of first neural networks of the current round; By adding neural nodes and reducing neural nodes in each of the plurality of first neural networks, a plurality of second neural networks of the current round are obtained; Using the accuracy and memory consumption of each of the second neural networks in the current round when running, construct an evaluation index function, using the evaluation index function to construct a ranking loss function, and using the ranking loss function to determine multiple third neural networks in the current round; In response to the current round being the first round, an auxiliary neural network is determined from the multiple pre-selected neural networks, and the multiple third neural networks of the current round are bidirectionally distilled and migrated using the auxiliary neural network; in response to the current round being not the first round, the multiple third neural networks of the current round are bidirectionally distilled and migrated using the target neural network of the previous round, and the result of the bidirectional distillation migration is used as the multiple candidate neural networks of the current round, and the target neural network of the current round is determined from the multiple candidate neural networks; The obtaining of a plurality of pre-selected neural networks comprises: constructing a first predetermined number of building blocks; The following operation is performed for each of the structural blocks: Using each operation type in the preset operation type set as the operation type of the neural layer to obtain a plurality of neural nodes, each of the neural nodes includes two neural layers, each of the neural layers corresponds to one operation type, and any two neural nodes in the plurality of neural nodes include different operation types; Select any one neural node from the plurality of neural nodes and place it in the structural block; Cascading all the structural blocks to obtain an initial backbone neural network; Perform multiple rounds of iterative addition and subtraction of branches on the initial backbone neural network, and obtain multiple initial neural networks of the round in each round; In response to determining that a preset second iteration indicator is reached, outputting a plurality of initial neural networks of a final round as the plurality of pre-selected neural networks; The following operations are performed for each of the multiple rounds of adding and reducing branches iterations: In response to the current round being the first round, the initial trunk neural network is used as the input neural network of the current round; in response to the current round being not the first round, multiple initial neural networks of the previous round are used as the input neural network of the current round; Based on all the neural nodes in each of the structural blocks in the input neural network, obtaining a plurality of first candidate structural blocks of the structural block by adding neural nodes and reducing neural nodes; Based on the first predetermined number and the multiple first candidate structural blocks of each structural block, multiple initial neural networks of the current round are obtained by combination.

2. The method according to claim 1, characterized in that The method of obtaining multiple second neural networks of the current round by adding neural nodes and reducing neural nodes in each of the multiple first neural networks includes: The following operations are performed for each of the structural blocks in each of the first neural networks: Adding a new neural node to the structural block, and determining a plurality of newly added structural blocks of the structural block based on the adding position of the new neural node, all the operation types contained in the new neural node, and the connection mode with the neural nodes in the structural block; Select one neural node from all neural nodes in the structural block and delete it to obtain multiple branch-reduced structural blocks of the structural block; Merging the multiple newly added structural blocks and the multiple branch-reduced structural blocks as multiple second candidate structural blocks of the structural block; Based on the first predetermined number and all of the plurality of second candidate structural blocks, the plurality of second neural networks of the current round are determined.

3. The method according to claim 1, characterized in that Before constructing the evaluation index function by using the accuracy and memory consumption of each second neural network during operation in the current round, the method includes: Input each of the second neural networks in the current round into a preset structure controller; Using the encoding layer in the structure controller to convert each structural feature of the second neural network into a corresponding network feature; inputting the network features into a sorting layer in the fabric controller; The step of constructing an evaluation index function using the accuracy and memory consumption of each second neural network during operation in the current round includes: Determine the accuracy and memory consumption of each second neural network during execution in the current round; Determine the accuracy and memory consumption of the target neural network runtime in the previous round; For each of the second neural networks, an accuracy evaluation item is determined using the accuracy of the second neural network and the accuracy of the target neural network in the previous round, and a memory consumption evaluation item is determined using the memory consumption of the second neural network and the memory consumption of the target neural network in the previous round; The accuracy evaluation item and the memory consumption evaluation item are fused according to a preset weight coefficient to obtain an evaluation index function for the second neural network.

4. The method according to claim 3, characterized in that The converting the structural features of each of the second neural networks into corresponding network features comprises: For each structural block in each of the second neural networks, the position, connection mode and all the operation types contained in each neural node in the structural block are respectively uniquely encoded to obtain the block structure network features of each structural block; All of the block structure network features of the second neural network are input into the encoding layer to obtain the network features of the second neural network.

5. The method according to claim 1, characterized in that The step of constructing a ranking loss function using the evaluation index function and determining multiple third neural networks of the current round using the ranking loss function includes: For each of the second neural networks, the following operations are performed: determining a true positional ranking of the second neural network; Determine a relative position ranking between the second neural network and each of all other second neural networks; Using the relative position ranking and the real position ranking, constructing a ranking loss function; Based on the ranking loss function, a second predetermined number of the plurality of third neural networks is determined from all of the second neural networks.

6. The method according to claim 1, characterized in that The bidirectional distillation migration comprises: Determining a target probability distribution of the auxiliary neural network and / or the target neural network of the previous round; determining a candidate probability distribution for each of the third neural networks; Calculating relative entropy and cross entropy between the target probability distribution and the candidate probability distribution; A training loss function is constructed according to the relative entropy and the cross entropy, and the third neural network of the current round is trained using the training loss function.

7. A neural network optimization device for image processing, comprising: An iteration module, wherein the iteration module includes a branch adding and reducing submodule, a sorting submodule and a bidirectional distillation submodule; The iteration module is configured to obtain a plurality of pre-selected neural networks, perform multiple rounds of iterations on all of the pre-selected neural networks, obtain a plurality of candidate neural networks and a target neural network in each round, and output the target neural network obtained in the last round in response to determining that a preset first iteration indicator is reached; The adding and reducing branch module is configured to, in response to the current round being the first round, use the plurality of pre-selected neural networks as the plurality of first neural networks of the current round; In response to the current round not being the first round, using the plurality of candidate neural networks of the previous round as the plurality of first neural networks of the current round; By adding neural nodes and reducing neural nodes in each of the plurality of first neural networks, a plurality of second neural networks of the current round are obtained; The sorting submodule is configured to construct an evaluation index function using the accuracy and memory consumption of each second neural network in the current round, construct a sorting loss function using the evaluation index function, and determine multiple third neural networks in the current round using the sorting loss function; The bidirectional distillation submodule is configured to, in response to the current round being the first round, determine an auxiliary neural network from the multiple pre-selected neural networks, use the auxiliary neural network to perform bidirectional distillation migration on the multiple third neural networks of the current round, and in response to the current round not being the first round, use the target neural network of the previous round to perform bidirectional distillation migration on the multiple third neural networks of the current round, use the result of the bidirectional distillation migration as the multiple candidate neural networks of the current round, and determine the target neural network of the current round from the multiple candidate neural networks; The branch adding and reducing module is specifically configured as follows: constructing a first predetermined number of building blocks; The following operation is performed for each of the structural blocks: Using each operation type in the preset operation type set as the operation type of the neural layer to obtain a plurality of neural nodes, each of the neural nodes includes two neural layers, each of the neural layers corresponds to one operation type, and any two neural nodes in the plurality of neural nodes include different operation types; Select any one neural node from the plurality of neural nodes and place it in the structural block; Cascading all the structural blocks to obtain an initial backbone neural network; Perform multiple rounds of iterative addition and subtraction of branches on the initial backbone neural network, and obtain multiple initial neural networks of the round in each round; In response to determining that a preset second iteration indicator is reached, outputting a plurality of initial neural networks of a final round as the plurality of pre-selected neural networks; The following operations are performed for each of the multiple rounds of adding and reducing branches iterations: In response to the current round being the first round, the initial trunk neural network is used as the input neural network of the current round; in response to the current round being not the first round, multiple initial neural networks of the previous round are used as the input neural network of the current round; Based on all the neural nodes in each of the structural blocks in the input neural network, multiple first candidate structural blocks of the structural block are obtained by adding and reducing neural nodes; based on the first predetermined number and the multiple first candidate structural blocks of each of the structural blocks, multiple initial neural networks of the current round are obtained by combination.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

Images