Image processing network generation, image classification method and related device

Abstract

The present application discloses an image processing network generation, image classification method and related devices, and relates to the field of artificial intelligence technology. The present application can be applied to the fields of blockchain, cloud technology and map vehicle networking, and the method includes: obtaining an initial network structure unit, which includes at least one node, and there are connecting edges between the nodes; obtaining a candidate operation set, and the candidate operation set includes at least one candidate operation; according to the structural parameters of the candidate operation corresponding to each connecting edge, generating a probability distribution of each connecting edge selecting the corresponding candidate operation; according to the probability distribution corresponding to each connecting edge, only one candidate operation is selected for each connecting edge and added to the initial network structure unit to obtain the current network to be optimized; the probability distribution is processed into a discrete distribution continuous process, and the current network to be optimized is gradient optimized to obtain the target image processing network. The present application can effectively reduce the computing resources consumed by the generation of the image processing network, and can realize flexible deployment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an image processing network generation, image classification method and related devices. Background Art

[0002] An image processing network is a neural network that processes images, such as a neural network used for image detection or image classification.

[0003] At present, in related schemes, image processing networks are searched and generated through differentiable network structure search methods. Differentiable network structure search methods will inevitably connect all candidate operations between nodes in the supernet to be searched, and then search and generate the target image processing network. Under related schemes, the search and generation process has high requirements for computing resources and is difficult to deploy on devices with limited video memory.

[0004] Therefore, there is currently a problem that the generation of image processing networks consumes high computing resources and is difficult to deploy flexibly. Summary of the invention

[0005] The embodiments of the present application provide an image processing network generation, image classification method and related devices, which can effectively reduce the computing resources consumed in image processing network generation and achieve flexible deployment.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] According to one embodiment of the present application, a method for generating an image processing network includes: obtaining an initial network structure unit, wherein the initial network structure unit includes at least one node representing a feature graph, and the nodes have connecting edges; obtaining a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; generating a probability distribution of selecting corresponding candidate operations for each connecting edge in the current forward propagation according to the structural parameters of the candidate operations corresponding to each connecting edge; according to the probability distribution corresponding to each connecting edge, selecting only one candidate operation from the corresponding candidate operations for each connecting edge and adding it to the initial network structure unit to obtain the current network to be optimized; performing discrete distribution continuation processing on the probability distribution, and performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result until the predetermined optimization conditions are met to obtain the target image processing network.

[0008] According to one embodiment of the present application, an image processing network generation device includes: a structural unit acquisition module, which is used to obtain an initial network structural unit, wherein the initial network structural unit includes at least one node representing a feature graph, and there are connecting edges between the nodes; an operation set acquisition module, which is used to obtain a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each of the connecting edges, and the candidate operation is an operation for processing the feature graph; a probability calculation module, which is used to generate a probability distribution of selecting corresponding candidate operations for each of the connecting edges in the current forward propagation according to the structural parameters of the candidate operations corresponding to each of the connecting edges; a single operation sampling module, which is used to select only one candidate operation from the corresponding candidate operations for each of the connecting edges according to the probability distribution corresponding to each of the connecting edges and add it to the initial network structural unit to obtain the current network to be optimized; an optimization module, which is used to perform discrete distribution continuous processing on the probability distribution, and based on the continuous processing result, perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized until the predetermined optimization conditions are met to obtain the target image processing network.

[0009] In some embodiments of the present application, the optimization module includes: a sample input unit, which is used to use the current network to be optimized and the preset image processing network to process the sample image respectively to obtain an image processing result; a network optimization unit, which is used to perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results according to the image processing results and the network intermediate layer of the preset image processing network.

[0010] In some embodiments of the present application, the image processing result includes a first result corresponding to the current network to be optimized and a second result corresponding to the preset image processing network, and the sample image is calibrated with a predetermined result; the network optimization unit is used to: calculate a first loss based on the first result and the predetermined result; calculate a second loss based on the first result and the second result; calculate a third loss based on the preset image processing network and the network intermediate layer corresponding to the same level in the current network to be optimized; and perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results according to the first loss, the second loss and the third loss.

[0011] In some embodiments of the present application, the probability calculation module is used to: for each of the connecting edges, perform an exponential operation on the structural parameters of each candidate operation corresponding to the connecting edge to obtain the parameter operation result corresponding to each of the candidate operations; sum the parameter operation results corresponding to each candidate operation corresponding to the connecting edge to obtain the summed result corresponding to the connecting edge; divide the sum of the parameter operation results corresponding to each candidate operation corresponding to the connecting edge by the summed result corresponding to the connecting edge to obtain the probability of each of the candidate operations being selected by the corresponding connecting edge; sum the probability of each of the candidate operations being selected by the corresponding connecting edge with the random sampling value to obtain the target probability value corresponding to each of the candidate operations; based on the target probability value corresponding to each of the candidate operations corresponding to each of the connecting edges, obtain the probability distribution of each of the connecting edges selecting the corresponding candidate operation in the current forward propagation.

[0012] In some embodiments of the present application, the single operation sampling module is used to: determine the candidate operation corresponding to the largest target probability value in the probability distribution corresponding to each of the connection edges, and obtain the target candidate operation corresponding to each of the connection edges; add the target candidate operation corresponding to each of the connection edges to the corresponding position of each of the connection edges in the initial network structure unit.

[0013] In some embodiments of the present application, the network optimization unit is used to: perform a mean pooling operation on two network feature maps corresponding to the same level of the network middle layer in the preset image processing network and the current network to be optimized to obtain two target feature maps with a uniform number of channels; convert the two target feature maps into a first weighted feature map and a second weighted feature map, respectively; and calculate the third loss based on the first weighted feature map and the second weighted feature map.

[0014] In some embodiments of the present application, the network optimization unit is used to: use a mean square error loss function to calculate the first weighted feature map and the second weighted feature map to obtain the third loss.

[0015] In some embodiments of the present application, the network optimization unit is used to: use a cross entropy loss function to calculate and process the first result and the predetermined result to obtain the first loss; use a relative entropy loss function to calculate and process the first result and the second result to obtain the second loss.

[0016] In some embodiments of the present application, the network optimization unit is used to: perform second-order approximate estimation processing on the network parameters in the current network to be optimized and the continuous processing results to obtain second-order approximate estimation results; and perform alternating gradient optimization on the network parameters and structural parameters in the current network to be optimized based on the second-order approximate estimation results.

[0017] According to one embodiment of the present application, an image classification method includes: obtaining an image to be classified; using a target image processing network for image classification to classify the image to be classified to obtain a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the image processing network generation method described in any embodiment of the present application.

[0018] According to one embodiment of the present application, an image classification device includes: an image acquisition module, used to acquire an image to be classified; a classification module, used to classify the image to be classified using a target image processing network for image classification to obtain a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the image processing network generation method described in any embodiment of the present application.

[0019] According to another embodiment of the present application, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of a computer, the computer executes the method described in the embodiment of the present application.

[0020] According to another embodiment of the present application, an electronic device includes: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the method described in the embodiment of the present application.

[0021] According to another embodiment of the present application, a computer program product or a computer program includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0022] In an embodiment of the present application, an initial network structure unit is obtained, wherein the initial network structure unit includes at least one node representing a feature graph, and the nodes have connecting edges; a candidate operation set is obtained, wherein the candidate operation set includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; according to the structural parameters of the candidate operations corresponding to each connecting edge, a probability distribution of selecting the corresponding candidate operation for each connecting edge in the current forward propagation is generated; according to the probability distribution corresponding to each connecting edge, only one candidate operation is selected from the corresponding candidate operations for each connecting edge and added to the initial network structure unit to obtain the current network to be optimized; the probability distribution is subjected to discrete distribution continuation processing, and based on the continuous processing result, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized until the predetermined optimization conditions are met, thereby obtaining the target image processing network.

[0023] In this way, during the generation of the image processing network, by calculating the probability distribution of selecting the corresponding candidate operation for each connection edge, only one candidate operation is selected from the corresponding candidate operations for each connection edge and added to the network structure unit, an image processing network generation method based on single-path sampling network structure search is implemented, avoiding the need for all candidate operations for the connection edges between nodes, effectively reducing the consumption of computing resources in the search and generation process of the image processing network, improving the efficiency of the network structure search, saving video memory occupancy, and allowing for flexible deployment on devices with limited video memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of a system to which an embodiment of the present application can be applied is shown.

[0026] Figure 2 A flowchart of an image processing network generation method according to an embodiment of the present application is shown.

[0027] Figure 3 A schematic diagram of an initial network structure unit according to an embodiment of the present application is shown.

[0028] Figure 4 A schematic diagram of a network structure unit for searching for candidate operations according to an embodiment of the present application is shown.

[0029] Figure 5 A schematic diagram of network optimization according to an embodiment of the present application is shown.

[0030] Figure 6 A flowchart of an image classification method according to an embodiment of the present application is shown.

[0031] Figure 7 A block diagram of an image processing network generating device according to an embodiment of the present application is shown.

[0032] Figure 8 A block diagram of an image classification device according to an embodiment of the present application is shown.

[0033] Fig. 9 A block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0035] It is understandable that in the specific implementation of the present application, data related to object images, user information, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0036] Figure 1 FIG. 1 is a schematic diagram showing a system 100 to which an embodiment of the present application can be applied. Figure 1 As shown, the system 100 may include a server 101 and a terminal 102 .

[0037] Server 101 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In one implementation of this example, server 101 is a cloud server, and server 101 can provide artificial intelligence cloud services, such as artificial intelligence cloud services for large-scale image classification.

[0038] The terminal 102 may be any device, including but not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, VR / AR devices, smart watches, computers, etc. In one embodiment, the server 101 or the terminal 102 may be a node device in a blockchain network or a map vehicle networking platform.

[0039] In one implementation of this example, the server 101 or the terminal 102 may: obtain an initial network structure unit, wherein the initial network structure unit includes at least one node representing a feature graph, and the nodes have connecting edges; obtain a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; calculate the probability distribution of selecting the corresponding candidate operation for each connecting edge in the current forward propagation according to the structural parameters of the candidate operation corresponding to each connecting edge; according to the probability distribution corresponding to each connecting edge, select only one candidate operation from the corresponding candidate operations for each connecting edge and add it to the network structure unit to obtain the current network to be optimized; perform discrete distribution continuation processing on the probability distribution, and perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result until the predetermined optimization conditions are met to obtain the target image processing network.

[0040] In one implementation of this example, the server 101 or the terminal 102 can: obtain an image to be classified; use a target image processing network for image classification to classify the image to be classified to obtain a classification result corresponding to the image to be classified, and the target image processing network is generated according to the image processing network generation method described in any embodiment of the present application.

[0041] Figure 2 The flowchart of the image processing network generation method according to an embodiment of the present application is schematically shown. The execution subject of the image processing network generation method can be any device, such as Figure 1 The server 101 or the terminal 102 is shown.

[0042] like Figure 2 As shown, the image processing network generation method may include steps S210 to S250.

[0043] Step S210, obtaining an initial network structure unit, wherein the initial network structure unit includes at least one node representing a feature graph, and there are connecting edges between the nodes;

[0044] Step S220, obtaining a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each connection edge, and the candidate operation is an operation for processing the feature graph;

[0045] Step S230, generating a probability distribution of selecting a corresponding candidate operation for each connection edge in the current forward propagation according to the structural parameters of the candidate operation corresponding to each connection edge;

[0046] Step S240, according to the probability distribution corresponding to each connection edge, only one candidate operation is selected from the corresponding candidate operations for each connection edge and added to the initial network structure unit to obtain the current network to be optimized;

[0047] Step S250, the probability distribution is processed into a discrete distribution continuation, and based on the continuation processing result, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized until the predetermined optimization conditions are met, thereby obtaining the target image processing network.

[0048] The initial network structure unit is a neural network structure unit for the candidate operation to be searched. The neural network structure unit can be called a cell unit, and the entire neural network can be formed by stacking the cell units. The initial network structure unit can include at least one node, each node represents a feature graph; the feature graph is a feature matrix, such as the feature matrix obtained by the convolution operation; the nodes can be connected by directed connecting edges, the connecting edges are the unknown candidate operations in the initial network structure, and the candidate operations at the connecting edges are to be searched. The initial network structure unit can be obtained from a predetermined position.

[0049] The candidate operation set includes at least one candidate operation corresponding to each connection edge, and the candidate operation is an operation for processing the feature graph, such as a convolution operation, a pooling operation, a skip connection operation, etc. The candidate operation set may be a search space composed of candidate operations, and the candidate operation set may be obtained from a predetermined position.

[0050] Each initial network structure unit can be abstracted as containing N nodes {x (0) ,x (1) ,…,x (N-1)}, for example, Figure 3 The initial network structure unit shown includes 7 nodes (for example, 0 and 1), where each node x (i) Represents a feature graph in the network. A candidate operation (such as a convolution operation) can be connected between two nodes connected by a connection edge. The purpose of generating an image processing network is to select a candidate operation that best suits each connection edge from the candidate operation set through network structure search. The feature graph represented by the previous node can be processed through the candidate operation to obtain the feature graph represented by the next node.

[0051] Each time forward propagation is performed (before performing gradient optimization until the predetermined optimization conditions are met, multiple forward propagations can be performed, for example, each time the gradient optimization of the current network to be optimized is completed, the next forward propagation can be performed), the probability distribution of each connection edge selecting the corresponding candidate operation in the current forward propagation can be calculated based on the structural parameters of the candidate operation corresponding to the connection edge, and the probability distribution is generated by the probability of selecting each corresponding candidate operation at each connection edge. For example, connection edge A corresponds to 9 predetermined candidate operations, and the probability of the connection edge selecting each of the 9 predetermined candidate operations can be calculated, thereby generating the probability distribution corresponding to connection edge A.

[0052] According to the probability distribution corresponding to each connection edge, only one candidate operation is selected from the corresponding candidate operations for each connection edge and added to the initial network structure unit. For example, connection edge A corresponds to 9 predetermined candidate operations. According to the probability distribution corresponding to connection edge A, only one candidate operation is selected from the 9 predetermined candidate operations, and the selected single candidate operation is added to the connection edge A in the initial network structure unit. A candidate operation is searched for each connection edge in the initial network structure unit, and a complete network containing nodes and candidate operations is obtained, that is, the current network to be optimized.

[0053] The probability distribution generated based on the structural parameters of the candidate operations is discretized, and the probability distribution is processed into a discrete distribution. Based on the continuous processing result, the structural parameters of the candidate operations in the current network to be optimized can be gradient optimized. In this way, when backpropagation is performed through gradient optimization, the structural parameters of each candidate operation have the gradient from the deep network, and then the structural parameters that meet the requirements can be optimized. When the optimization meets the predetermined optimization conditions, the target image processing network is obtained.

[0054] The target image processing network is a network composed of network structural units (i.e., searched network structural units) of candidate operations that meet the required structural parameters, for example, Figure 4 Shown Figure 3 The searched network structure unit is obtained after searching for a candidate operation (such as skip-connect) between the seven nodes (for example, 0 and 1) in the initial network structure unit. The image to be processed can be processed based on the target image processing network, and the target image processing network can be a neural network for image classification or image target detection.

[0055] In this way, based on steps S210 to S250, during the image processing network generation process, by calculating the probability distribution of selecting the corresponding candidate operation for each connection edge, and then selecting only one candidate operation from the corresponding candidate operations for each connection edge and adding it to the network structure unit, an image processing network generation method based on single-path sampling network structure search is implemented, thereby avoiding the need for all candidate operations for the connection edges between nodes, effectively reducing the consumption of computing resources in the search and generation process of the image processing network, improving the efficiency of the network structure search, saving video memory occupancy, and allowing for flexible deployment on devices with limited video memory.

[0056] Described below Figure 2 When the image processing network is generated in the embodiment, other specific optional embodiments are provided for each step performed.

[0057] In one embodiment, step S230 generates a probability distribution of selecting a corresponding candidate operation for each connection edge in the current forward propagation according to the structural parameters of the candidate operation corresponding to each connection edge, including:

[0058] For each connecting edge, perform exponential operation on the structural parameters of each candidate operation corresponding to the connecting edge to obtain the parameter operation result corresponding to each candidate operation; sum the parameter operation results corresponding to each candidate operation corresponding to the connecting edge to obtain the summed result corresponding to the connecting edge; divide the sum of the parameter operation results corresponding to each candidate operation corresponding to the connecting edge by the summed result corresponding to the connecting edge to obtain the probability of each candidate operation being selected by the corresponding connecting edge; sum the probability of each candidate operation being selected by the corresponding connecting edge with the random sampling value to obtain the target probability value corresponding to each candidate operation; based on the target probability value corresponding to each candidate operation corresponding to each connecting edge, obtain the probability distribution of each connecting edge selecting the corresponding candidate operation in the current forward propagation.

[0059] Specifically, the probability distribution of the candidate operation corresponding to each connection edge selection can be calculated based on the following Gumbel-Max method formula:

[0060]

[0061]

[0062] in, is the structural parameter of each candidate operation o corresponding to the connection edge (i, j); represents the probability of selecting each candidate operation o by the connection edge (i, j), that is, the probability of each candidate operation o being selected by the corresponding connection edge (i, j), where the connection edge (i, j) is the connection edge between nodes i and j. The candidate operation o belongs to the set of at least one candidate operation corresponding to the connection edge That is, the parameter operation result corresponding to the candidate operation o; That is, the sum of the corresponding connection edges (i, j); g is the random sampling value of the Gumbel distribution. Due to randomness, the candidate operations sampled in any forward propagation may be different; That is, the target probability value corresponding to the candidate operation o; the set of V is the probability distribution of the candidate operations corresponding to the connection edge selection.

[0063] In one embodiment, based on the probability distribution corresponding to each connection edge, only one candidate operation is selected from the corresponding candidate operations for each connection edge and added to the initial network structure unit, including: determining the candidate operation corresponding to the largest target probability value in the probability distribution corresponding to each connection edge, and obtaining the target candidate operation corresponding to each connection edge; adding the target candidate operation corresponding to each connection edge to the corresponding position of each connection edge in the initial network structure unit.

[0064] Among them, the formula Determine the maximum target probability value in the probability distribution corresponding to each connection edge (i, j). Through this formula, the set of at least one candidate operation corresponding to the connection edge can be The target probability value corresponding to each candidate operation o in Normalized to the normalized value A i,j Then, the largest normalized value is processed as 1 through one-hot encoding, and the rest of the normalized values ​​are processed as 0. Then, the target probability value corresponding to the normalized value corresponding to 1 is the largest.

[0065] The determined target candidate operation is added to the corresponding position of each connection edge in the initial network structure unit, and the network structure unit to be optimized in which the candidate operation is preliminarily searched is obtained.

[0066] In one embodiment, in step S250, the probability distribution is processed into a discrete distribution continuous distribution, which can be the following formula of the Gumbel-Softmax method. The formed probability distribution is discretely distributed and the continuous processing result B is obtained. i,j :

[0067]

[0068]

[0069] Where τ is the temperature coefficient.

[0070] In one embodiment, step S250, performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result, includes: directly performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result.

[0071] In one embodiment, step S250, based on the continuous processing results, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized, including: using the current network to be optimized and the preset image processing network to process the sample image respectively to obtain the image processing results; according to the image processing results and the network middle layer of the preset image processing network, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results.

[0072] The preset image processing network is a pre-designed and trained image processing network. The search process of directly using candidate operations for gradient optimization is unstable. When the number of search rounds becomes longer, the performance of the network obtained by the search will decrease instead, because there is an imbalance in the inter-layer gradient in the current network to be optimized, and candidate operations without parameters such as jump connections can provide additional pathways for gradient conduction. Therefore, as the search proceeds, the network is more inclined to select meaningless jump connections as candidate operations between nodes. By jointly performing gradient optimization with the preset image processing network, the current network to be optimized can learn the inter-layer gradient distribution of the preset image processing network, and then smooth its own gradient distribution, thereby improving the stability of the search process. At the same time, since the current network to be optimized can receive information supervision from the preset image processing network, it can also further improve the performance of the target image processing network and the image processing effect of the target image processing network.

[0073] In one embodiment, the image processing result includes a first result corresponding to the current network to be optimized and a second result corresponding to the preset image processing network, and the sample image is calibrated with a predetermined result; according to the processing result and the network intermediate layer of the preset image processing network, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing result, including: calculating the first loss according to the first result and the predetermined result; calculating the second loss according to the first result and the second result; calculating the third loss according to the preset image processing network and the network intermediate layer of the same level in the current network to be optimized; based on the first loss, the second loss and the third loss, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing result.

[0074] The first loss, the second loss and the third loss respectively form learning guidance for the current network to be optimized relative to the preset image processing network from different angles. According to the first loss, the second loss and the third loss, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing results. In this way, the current network to be optimized can be used as a student network and the preset image processing network as a teacher network in a knowledge distillation manner, so that the current network to be optimized can further effectively learn the inter-layer gradient distribution of the preset image processing network, and then smooth its own gradient distribution, further improving the stability of the search process. The current network to be optimized can further effectively receive information supervision from the preset image processing network, further improving the performance of the target image processing network and improving network stability.

[0075] Among them, see Figure 5 , the preset image processing network is divided into 3 network blocks, each network block is a network middle layer of a level, the current network to be optimized includes 3 cell units, each cell unit corresponds to an initial network structure unit, and each cell unit is a network middle layer of a level. For the network blocks and cell units of the same level, the third loss can be calculated The second loss is calculated for the second result of the last network block output (i.e., the teacher output) and the first result of the last cell unit output (i.e., the student output). Calculate the first loss for the predetermined result of the sample image calibration (ie, the true label) and the first result of the last cell unit output (ie, the student output)

[0076] In one embodiment, a third loss is calculated based on a preset image processing network and a network intermediate layer corresponding to the same level in the current network to be optimized, including: performing a mean pooling operation on two network feature maps corresponding to the network intermediate layer of the same level in the preset image processing network and the current network to be optimized to obtain two target feature maps with a uniform number of channels; converting the two target feature maps into a first weighted feature map and a second weighted feature map, respectively; and calculating the third loss based on the first weighted feature map and the second weighted feature map.

[0077] Specifically, the two network feature maps corresponding to the same level of the network middle layer (such as the first cell unit and the first network block) are averaged and pooled, and the number of channels of the two network feature maps is uniformly reduced to the smaller value between them through the average pooling operation. Two target feature maps with the same number of channels are obtained.

[0078] The target feature map corresponding to the network feature map output by the cell unit at the i-th level can be expressed by F i It is indicated by the formula The target feature map Fi Convert to the first weighted feature map I kd is the total number of cell units, j is the target feature map F i The jth feature in the network; the network feature map output by the network block at the i-th level can be used It is indicated by the formula The target feature map can be Converted to the second weighted feature map A i , according to the first weighted feature map and the second weighted feature map, the calculated third loss can effectively guide the learning of the current network to be optimized.

[0079] In one embodiment, a third loss is calculated based on the first weighted feature map and the second weighted feature map, including: using a mean square error loss function to calculate and process the first weighted feature map and the second weighted feature map to obtain the third loss.

[0080] Specifically, the first weighted feature map and the second weighted feature map can be calculated based on the following mean square error loss function to obtain the third loss

[0081]

[0082] in, is the first weighted feature map, A i is the second weighted feature map.

[0083] In one embodiment, a first loss is calculated based on a first result and a predetermined result, including: using a cross entropy loss function to calculate and process the first result and the predetermined result to obtain a first loss; and a second loss is calculated based on the first result and the second result, including: using a relative entropy loss function to calculate and process the first result and the second result to obtain a second loss.

[0084] Specifically, the first result and the predetermined result can be calculated based on the following cross entropy loss function to obtain the first loss

[0085]

[0086] Where N is the total number of sub-results (such as classification probabilities) in the first result and the predetermined result, and y k is the kth sub-result in the predetermined result, is the kth sub-result in the first result.

[0087] Specifically, the first result and the second result can be calculated based on the following relative entropy loss function (i.e., KL divergence loss function) to obtain the second loss function:

[0088]

[0089] Wherein, N is the total number of sub-results (such as classification probabilities) in the first result and the second result, is the kth sub-result in the second result, p L, is the kth sub-result in the first result.

[0090] In one embodiment, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results, including: performing second-order approximate estimation processing on the network parameters in the current network to be optimized and the continuous processing results to obtain second-order approximate estimation results; and performing alternating gradient optimization on the network parameters and structural parameters in the current network to be optimized based on the second-order approximate estimation results.

[0091] The network parameters in the current network to be optimized and the continuous processing results are subjected to second-order approximate estimation processing to obtain second-order approximate estimation results. Based on the second-order approximate estimation results, the loss function can be differentiable with respect to both the network parameter ω and the structural parameter α. Based on the second-order approximate estimation results, the network parameters and structural parameters in the current network to be optimized are subjected to alternating gradient optimization until the predetermined optimization conditions are met, and the target image processing network can be obtained.

[0092] For example, the goal of the search process corresponding to gradient optimization is the following bi-objective optimization problem:

[0093]

[0094]

[0095] However, the loss function under this objective is not differentiable with respect to the structural parameter α, and the first-order approximation can directly use the current network parameter ω as ω * (α) is approximated, and the second-order approximation estimates the optimal network parameters as the network parameters after one step of gradient descent. After the second-order approximation, the loss function is differentiable for both the network parameter ω and the structural parameter α. Next, the alternating gradient optimization of training the structural parameter one step and then training the network parameter one step is used to train the current network to be optimized until the predetermined optimization conditions are met, and the target image processing network can be obtained. Among them, the alternating gradient optimization method can be to fix the value of the structural parameter α matrix on the training set, and then gradient descent the value of the network parameter ω matrix, and then gradient descent the value of the structural parameter α matrix on the validation set.

[0096] Figure 6 The flowchart of the image classification method according to an embodiment of the present application is schematically shown. The execution subject of the image classification method can be any device, such as Figure 1The server 101 or the terminal 102 is shown.

[0097] like Figure 6 As shown, the image classification method may include steps S310 to S320.

[0098] Step S310, obtaining an image to be classified; Step S320, using a target image processing network for image classification to classify the image to be classified, and obtaining a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the image processing network generation method of any of the aforementioned embodiments of the present application.

[0099] In this way, in some embodiments, the target image processing network used for image classification has the beneficial effects of low consumption of computing resources in the search and generation process, high generation efficiency, low video memory occupancy, and flexible deployment on devices with limited video memory. Using the target image processing network for image classification can further reduce the overall resource consumption of image classification tasks and improve the flexibility of image classification task deployment; in some embodiments, the target image processing network used for image classification has excellent stability and can further effectively improve the classification accuracy of image classification as a whole.

[0100] The foregoing embodiment is further described below in conjunction with a scenario in which images to be classified are classified. In this scenario, images to be classified are classified by applying the foregoing embodiment of the present application.

[0101] In this scenario, classifying the image to be classified may include steps (1) to (2).

[0102] Step (1): Use the network structure search method to search for convolutional neural networks in image classification tasks.

[0103] Specifically, an initial network structure unit is obtained, which includes at least one node representing a feature graph, and there are connecting edges between the nodes; a candidate operation set is obtained, which includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; according to the structural parameters of the candidate operation corresponding to each connecting edge, the probability distribution of each connecting edge selecting the corresponding candidate operation in the current forward propagation is calculated; according to the probability distribution corresponding to each connecting edge, only one candidate operation is selected from the corresponding candidate operation for each connecting edge and added to the initial network structure unit to obtain the current network to be optimized; the probability distribution is processed into a discrete distribution continuation, and the structural parameters of the candidate operation in the current network to be optimized are gradient optimized based on the continuous processing result until the predetermined optimization conditions are met, so as to obtain the target image processing network. The target image processing network is the convolutional neural network for image classification that is searched.

[0104] Furthermore, based on the structural parameters of the candidate operations corresponding to each connecting edge, a probability distribution of each connecting edge selecting the corresponding candidate operation in the current forward propagation is generated, including: for each connecting edge, performing an exponential operation on the structural parameters of each candidate operation corresponding to the connecting edge to obtain the parameter operation result corresponding to each candidate operation; summing the parameter operation results corresponding to each candidate operation corresponding to the connecting edge to obtain the summed result corresponding to the connecting edge; dividing the sum of the parameter operation results corresponding to each candidate operation corresponding to the connecting edge by the summed result corresponding to the connecting edge to obtain the probability of each candidate operation being selected by the corresponding connecting edge; summing the probability of each candidate operation being selected by the corresponding connecting edge and the random sampling value to obtain the target probability value corresponding to each candidate operation; based on the target probability value corresponding to each candidate operation corresponding to each connecting edge, a probability distribution of each connecting edge selecting the corresponding candidate operation in the current forward propagation is obtained.

[0105] Specifically, the probability distribution of the candidate operation corresponding to each connection edge selection can be calculated based on the following Gumbel-Max method formula:

[0106]

[0107]

[0108] in, is the structural parameter of each candidate operation o corresponding to the connection edge (i, j); represents the probability of selecting each candidate operation o by the connection edge (i, j), that is, the probability of each candidate operation o being selected by the corresponding connection edge (i, j), where the connection edge (i, j) is the connection edge between nodes i and j. The candidate operation o belongs to the set of at least one candidate operation corresponding to the connection edge That is, the parameter operation result corresponding to the candidate operation o; That is, the sum of the corresponding connection edges (i, j); g is the random sampling value of the Gumbel distribution. Due to randomness, the candidate operations sampled in any forward propagation may be different; That is, the target probability value corresponding to the candidate operation o; the set of V is the probability distribution of the candidate operations corresponding to the connection edge selection.

[0109] Furthermore, according to the probability distribution corresponding to each connecting edge, only one candidate operation is selected from the corresponding candidate operations for each connecting edge and added to the initial network structure unit, including: determining the candidate operation corresponding to the largest target probability value in the probability distribution corresponding to each connecting edge, and obtaining the target candidate operation corresponding to each connecting edge; adding the target candidate operation corresponding to each connecting edge to the corresponding position of each connecting edge in the initial network structure unit.

[0110] Among them, according to the formula Determine the maximum target probability value in the probability distribution corresponding to each connection edge (i, j). Through this formula, the set of at least one candidate operation corresponding to the connection edge can be The target probability value corresponding to each candidate operation o in Normalized to the normalized value A i,j Then, the largest normalized value is processed as 1 through one-hot encoding, and the rest of the normalized values ​​are processed as 0. Then, the target probability value corresponding to the normalized value corresponding to 1 is the largest.

[0111] The determined target candidate operation is added to the corresponding position of each connection edge in the initial network structure unit, and the network structure unit to be optimized in which the candidate operation is preliminarily searched is obtained.

[0112] Furthermore, the probability distribution can be processed into a continuous distribution by converting the discrete distribution into the continuous distribution. The following formula of the Gumbel-Softmax method can be used to convert the probability The formed probability distribution is discretely distributed and the continuous processing result B is obtained. i,j :

[0113]

[0114]

[0115] Where τ is the temperature coefficient.

[0116] Furthermore, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result, including: directly performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result.

[0117] Furthermore, based on the continuous processing results, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized, including: using the current network to be optimized and the preset image processing network to process the sample image respectively to obtain the image processing result; according to the image processing result and the network middle layer of the preset image processing network, based on the continuous processing results, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized.

[0118] The preset image processing network is a pre-designed and trained image processing network. The search process of directly using candidate operations for gradient optimization is unstable. When the number of search rounds becomes longer, the performance of the network obtained by the search will decrease instead, because there is an imbalance in the inter-layer gradient in the current network to be optimized, and candidate operations without parameters such as jump connections can provide additional pathways for gradient conduction. Therefore, as the search proceeds, the network is more inclined to select meaningless jump connections as candidate operations between nodes. By jointly performing gradient optimization with the preset image processing network, the current network to be optimized can learn the inter-layer gradient distribution of the preset image processing network, and then smooth its own gradient distribution, thereby improving the stability of the search process. At the same time, since the current network to be optimized can receive information supervision from the preset image processing network, it can also further improve the performance of the target image processing network and the image processing effect of the target image processing network.

[0119] Furthermore, the image processing result includes a first result corresponding to the current network to be optimized and a second result corresponding to the preset image processing network, and the sample image is calibrated with a predetermined result; according to the processing result and the network intermediate layer of the preset image processing network, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing result, including: calculating the first loss according to the first result and the predetermined result; calculating the second loss according to the first result and the second result; calculating the third loss according to the preset image processing network and the network intermediate layer of the same level in the current network to be optimized; based on the first loss, the second loss and the third loss, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing result.

[0120] The first loss, the second loss and the third loss respectively form learning guidance for the current network to be optimized relative to the preset image processing network from different angles. According to the first loss, the second loss and the third loss, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuous processing results. In this way, the current network to be optimized can be used as a student network and the preset image processing network as a teacher network in a knowledge distillation manner, so that the current network to be optimized can further effectively learn the inter-layer gradient distribution of the preset image processing network, and then smooth its own gradient distribution, further improving the stability of the search process. The current network to be optimized can further effectively receive information supervision from the preset image processing network, further improving the performance of the target image processing network and improving network stability.

[0121] Among them, see Figure 5, the preset image processing network is divided into 3 network blocks, each network block is a network middle layer of a level, the current network to be optimized includes 3 cell units, each cell unit corresponds to an initial network structure unit, and each cell unit is a network middle layer of a level. For the network blocks and cell units of the same level, the third loss can be calculated The second loss is calculated for the second result of the last network block output (i.e., the teacher output) and the first result of the last cell unit output (i.e., the student output). Calculate the first loss for the predetermined result of the sample image calibration (ie, the true label) and the first result of the last cell unit output (ie, the student output)

[0122] Furthermore, according to the preset image processing network and the network intermediate layer corresponding to the same level in the current network to be optimized, a third loss is calculated, including: performing a mean pooling operation on two network feature maps corresponding to the network intermediate layer of the same level in the preset image processing network and the current network to be optimized to obtain two target feature maps with a uniform number of channels; converting the two target feature maps into a first weighted feature map and a second weighted feature map, respectively; and calculating the third loss based on the first weighted feature map and the second weighted feature map.

[0123] Specifically, the two network feature maps corresponding to the same level of the network middle layer (such as the first cell unit and the first network block) are averaged and pooled, and the number of channels of the two network feature maps is uniformly reduced to the smaller value between them through the average pooling operation. Two target feature maps with the same number of channels are obtained.

[0124] The target feature map corresponding to the network feature map output by the cell unit at the i-th level can be expressed by F i It is indicated by the formula The target feature map F i Convert to the first weighted feature map I kd is the total number of cell units, j is the target feature map F i The jth feature in the network; the network feature map output by the network block at the i-th level can be used It is indicated by the formula The target feature map can be Converted to the second weighted feature map A i , according to the first weighted feature map and the second weighted feature map, the calculated third loss can effectively guide the learning of the current network to be optimized.

[0125] Further, according to the first weighted feature map and the second weighted feature map, a third loss is calculated, including: using a mean square error loss function to calculate the first weighted feature map and the second weighted feature map to obtain the third loss.

[0126] Specifically, the first weighted feature map and the second weighted feature map can be calculated based on the following mean square error loss function to obtain the third loss

[0127]

[0128] in, is the first weighted feature map, A i is the second weighted feature map.

[0129] Furthermore, based on the first result and the predetermined result, a first loss is calculated, including: using a cross entropy loss function to calculate and process the first result and the predetermined result to obtain the first loss; based on the first result and the second result, a second loss is calculated, including: using a relative entropy loss function to calculate and process the first result and the second result to obtain the second loss.

[0130] Specifically, the first result and the predetermined result can be calculated based on the following cross entropy loss function to obtain the first loss

[0131]

[0132] Where N is the total number of sub-results (such as classification probabilities) in the first result and the predetermined result, and y k is the kth sub-result in the predetermined result, is the kth sub-result in the first result.

[0133] Specifically, the first result and the second result can be calculated based on the following relative entropy loss function (i.e., KL divergence loss function) to obtain the second loss function:

[0134]

[0135] Wherein, N is the total number of sub-results (such as classification probabilities) in the first result and the second result, is the kth sub-result in the second result, p L,k is the kth sub-result in the first result.

[0136] Furthermore, based on the continuous processing results, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized, including: performing second-order approximate estimation processing on the network parameters in the current network to be optimized and the continuous processing results to obtain second-order approximate estimation results; based on the second-order approximate estimation results, alternating gradient optimization is performed on the network parameters and structural parameters in the current network to be optimized.

[0137] The network parameters in the current network to be optimized and the continuous processing results are subjected to second-order approximate estimation processing to obtain second-order approximate estimation results. Based on the second-order approximate estimation results, the loss function can be differentiable with respect to both the network parameter ω and the structural parameter α. Based on the second-order approximate estimation results, the network parameters and structural parameters in the current network to be optimized are subjected to alternating gradient optimization until the predetermined optimization conditions are met, and the target image processing network can be obtained.

[0138] For example, the goal of the search process corresponding to gradient optimization is the following bi-objective optimization problem:

[0139]

[0140]

[0141] However, the loss function under this objective is not differentiable with respect to the structural parameter α, and the first-order approximation can directly use the current network parameter ω as ω * (α) is approximated, and the second-order approximation estimates the optimal network parameters as the network parameters after one step of gradient descent. After the second-order approximation, the loss function is differentiable for both the network parameter ω and the structural parameter α. Next, the alternating gradient optimization of training the structural parameter one step and then training the network parameter one step is used to train the current network to be optimized until the predetermined optimization conditions are met, and the target image processing network can be obtained. Among them, the alternating gradient optimization method can be to fix the value of the structural parameter α matrix on the training set, and then gradient descent the value of the network parameter ω matrix, and then gradient descent the value of the structural parameter α matrix on the validation set.

[0142] Step (2): Use the searched convolutional neural network to perform image classification.

[0143] Specifically, an image to be classified is obtained; and the image to be classified is classified using the searched convolutional neural network to obtain a classification result corresponding to the image to be classified.

[0144] In this scenario, by applying the aforementioned embodiments of the present application to perform image classification, at least the following beneficial effects are achieved: the search convolutional neural network process has the beneficial effects of low consumption of computing resources, high generation efficiency, low video memory occupancy, and can be flexibly deployed on devices with limited video memory. Using this convolutional neural network for image classification can further reduce the overall resource consumption of image classification tasks and improve the flexibility of image classification task deployment; the convolutional neural network used for image classification has excellent stability and can further effectively improve the classification accuracy of image classification as a whole.

[0145] In order to facilitate better implementation of the image processing network generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing network generation device based on the above-mentioned image processing network generation method. The meanings of the terms are the same as those in the above-mentioned image processing network generation method, and the specific implementation details can refer to the description in the method embodiment. Figure 7 A block diagram of an image processing network generating device according to an embodiment of the present application is shown.

[0146] like Figure 7 As shown, the image processing network generation device 400 may include a structure unit acquisition module 410, an operation set acquisition module 420, a probability calculation module 430, a single operation sampling module 440 and an optimization module 450.

[0147] A structural unit acquisition module is used to acquire an initial network structural unit, wherein the initial network structural unit includes at least one node representing a feature graph, and there are connecting edges between the nodes; an operation set acquisition module is used to acquire a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; a probability calculation module is used to generate a probability distribution of selecting corresponding candidate operations for each connecting edge in the current forward propagation according to the structural parameters of the candidate operations corresponding to each connecting edge; a single operation sampling module is used to select only one candidate operation from the corresponding candidate operations for each connecting edge according to the probability distribution corresponding to each connecting edge, and add it to the initial network structural unit to obtain the current network to be optimized; an optimization module is used to perform discrete distribution continuous processing on the probability distribution, and based on the continuous processing result, perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized until the predetermined optimization conditions are met, so as to obtain the target image processing network.

[0148] In some embodiments of the present application, the optimization module includes: a sample input unit, which is used to use the current network to be optimized and the preset image processing network to process the sample image respectively to obtain an image processing result; a network optimization unit, which is used to perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results according to the image processing results and the network intermediate layer of the preset image processing network.

[0149] In some embodiments of the present application, the image processing result includes a first result corresponding to the current network to be optimized and a second result corresponding to the preset image processing network, and the sample image is calibrated with a predetermined result; the network optimization unit is used to: calculate a first loss based on the first result and the predetermined result; calculate a second loss based on the first result and the second result; calculate a third loss based on the preset image processing network and the network intermediate layer corresponding to the same level in the current network to be optimized; and perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing results according to the first loss, the second loss and the third loss.

[0150] In some embodiments of the present application, the probability calculation module is used to: for each of the connecting edges, perform an exponential operation on the structural parameters of each candidate operation corresponding to the connecting edge to obtain the parameter operation result corresponding to each of the candidate operations; sum the parameter operation results corresponding to each candidate operation corresponding to the connecting edge to obtain the summed result corresponding to the connecting edge; divide the sum of the parameter operation results corresponding to each candidate operation corresponding to the connecting edge by the summed result corresponding to the connecting edge to obtain the probability of each of the candidate operations being selected by the corresponding connecting edge; sum the probability of each of the candidate operations being selected by the corresponding connecting edge with the random sampling value to obtain the target probability value corresponding to each of the candidate operations; based on the target probability value corresponding to each of the candidate operations corresponding to each of the connecting edges, obtain the probability distribution of each of the connecting edges selecting the corresponding candidate operation in the current forward propagation.

[0151] In some embodiments of the present application, the single operation sampling module is used to: determine the candidate operation corresponding to the largest target probability value in the probability distribution corresponding to each of the connection edges, and obtain the target candidate operation corresponding to each of the connection edges; add the target candidate operation corresponding to each of the connection edges to the corresponding position of each of the connection edges in the initial network structure unit.

[0152] In some embodiments of the present application, the network optimization unit is used to: perform a mean pooling operation on two network feature maps corresponding to the same level of the network middle layer in the preset image processing network and the current network to be optimized to obtain two target feature maps with a uniform number of channels; convert the two target feature maps into a first weighted feature map and a second weighted feature map, respectively; and calculate the third loss based on the first weighted feature map and the second weighted feature map.

[0153] In some embodiments of the present application, the network optimization unit is used to: use a mean square error loss function to calculate the first weighted feature map and the second weighted feature map to obtain the third loss.

[0154] In some embodiments of the present application, the network optimization unit is used to: use a cross entropy loss function to calculate and process the first result and the predetermined result to obtain the first loss; use a relative entropy loss function to calculate and process the first result and the second result to obtain the second loss.

[0155] In some embodiments of the present application, the network optimization unit is used to: perform second-order approximate estimation processing on the network parameters in the current network to be optimized and the continuous processing results to obtain second-order approximate estimation results; and perform alternating gradient optimization on the network parameters and structural parameters in the current network to be optimized based on the second-order approximate estimation results.

[0156] The present application also provides an image classification device based on the above image classification method. The meanings of the terms are the same as those in the above image classification method, and the specific implementation details can refer to the description in the method embodiment. Figure 8 A block diagram of an image classification device according to an embodiment of the present application is shown.

[0157] like Figure 8 As shown, the image classification device 500 may include an image acquisition module 510 and a classification module 520 .

[0158] An image acquisition module is used to acquire an image to be classified; a classification module is used to classify the image to be classified using a target image processing network for image classification to obtain a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the image processing network generation method described in any embodiment of the present application.

[0159] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0160] In addition, an embodiment of the present application further provides an electronic device, which may be a terminal or a server. Fig. 9 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0161] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0162] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the computer device and processes data, thereby detecting the electronic device. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0163] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0164] The electronic device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0165] The electronic device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0166] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 will run the computer programs stored in the memory 602, thereby realizing the various functions of the aforementioned embodiments of the present application.

[0167] For example, the processor 601 may execute: obtaining an initial network structure unit, wherein the initial network structure unit includes at least one node representing a feature graph, and the nodes have connecting edges; obtaining a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each connecting edge, and the candidate operation is an operation for processing the feature graph; calculating the probability distribution of selecting the corresponding candidate operation for each connecting edge in the current forward propagation according to the structural parameters of the candidate operation corresponding to each connecting edge; according to the probability distribution corresponding to each connecting edge, selecting only one candidate operation from the corresponding candidate operations for each connecting edge and adding it to the network structure unit to obtain the current network to be optimized; performing discrete distribution continuation processing on the probability distribution, and performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result until the predetermined optimization conditions are met, thereby obtaining the target image processing network.

[0168] For example, the processor 601 can execute: obtaining an image to be classified; using a target image processing network for image classification to classify the image to be classified, and obtaining a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the image processing network generation method described in any embodiment of the present application.

[0169] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0170] To this end, an embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any method provided in the embodiment of the present application.

[0171] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0172] Since the computer program stored in the computer-readable storage medium can execute the steps in any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0173] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations in the above embodiments of the present application.

[0174] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0175] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for generating an image processing network, characterized in that: include: Acquire an initial network structure unit, wherein the initial network structure unit includes at least one node representing a feature graph, and the nodes have connecting edges; Obtain a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each of the connection edges, and the candidate operation is an operation for processing a feature graph; Generate a probability distribution of the candidate operation corresponding to each of the connection edges in the current forward propagation according to the structural parameters of the candidate operation corresponding to each of the connection edges; According to the probability distribution corresponding to each of the connection edges, only one candidate operation is selected from the corresponding candidate operations for each of the connection edges and added to the initial network structure unit to obtain a current network to be optimized; Performing discrete distribution continuation processing on the probability distribution, and performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuation processing result until the predetermined optimization conditions are met, thereby obtaining a target image processing network; The step of performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result includes: The sample image is processed by using the current network to be optimized and the preset image processing network to obtain image processing results; According to the image processing result and the network middle layer of the preset image processing network, gradient optimization is performed on the structural parameters of the candidate operation in the current network to be optimized based on the continuous processing result.

2. The method according to claim 1, characterized in that The image processing result includes a first result corresponding to the current network to be optimized and a second result corresponding to the preset image processing network, and the sample image is calibrated with a predetermined result; The step of performing gradient optimization on the structural parameters of the candidate operation in the current network to be optimized based on the processing result and the network intermediate layer of the preset image processing network and the continuous processing result comprises: Calculating a first loss according to the first result and the predetermined result; Calculating a second loss according to the first result and the second result; Calculating a third loss according to the preset image processing network and the network middle layer corresponding to the same level in the current network to be optimized; According to the first loss, the second loss and the third loss, gradient optimization is performed on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result.

3. The method according to claim 1, characterized in that The generating, according to the structural parameters of the candidate operations corresponding to the connection edges, a probability distribution of selecting the corresponding candidate operations for the connection edges in the current forward propagation includes: For each of the connection edges, perform an exponential operation on a structural parameter of each candidate operation corresponding to the connection edge to obtain a parameter operation result corresponding to each of the candidate operations; Sum the parameter operation results corresponding to each candidate operation corresponding to the connection edge to obtain the summation result corresponding to the connection edge; Divide the sum of the parameter calculation results corresponding to each candidate operation corresponding to the connection edge by the sum corresponding to the connection edge to obtain the probability of each candidate operation being selected by the corresponding connection edge; Sum the probability of each candidate operation being selected by the corresponding connection edge and the random sampling value to obtain a target probability value corresponding to each candidate operation; Based on the target probability values ​​corresponding to the candidate operations corresponding to the connection edges, the probability distribution of the candidate operations corresponding to the connection edges in the current forward propagation is obtained.

4. The method according to claim 3, characterized in that The step of selecting only one candidate operation from the corresponding candidate operations for each connection edge according to the probability distribution corresponding to each connection edge and adding the operation to the initial network structure unit includes: Determine the candidate operation corresponding to the maximum target probability value in the probability distribution corresponding to each of the connection edges, and obtain the target candidate operation corresponding to each of the connection edges; The target candidate operations corresponding to the connection edges are added to the corresponding positions of the connection edges in the initial network structure unit.

5. The method according to claim 2, characterized in that: The calculating the third loss according to the preset image processing network and the network intermediate layer corresponding to the same level in the current network to be optimized includes: Performing a mean pooling operation on two network feature maps corresponding to the same level of the network middle layer in the preset image processing network and the current network to be optimized, to obtain two target feature maps with a uniform number of channels; Converting the two target feature maps into a first weighted feature map and a second weighted feature map respectively; The third loss is calculated according to the first weighted feature map and the second weighted feature map.

6. The method according to claim 5, characterized in that The calculating the third loss according to the first weighted feature map and the second weighted feature map includes: The first weighted feature map and the second weighted feature map are calculated and processed using a mean square error loss function to obtain the third loss.

7. The method according to claim 2, 5 or 6, characterized in that: The calculating the first loss according to the first result and the predetermined result includes: Using a cross entropy loss function to calculate the first result and the predetermined result to obtain the first loss; The calculating the second loss according to the first result and the second result includes: The first result and the second result are calculated and processed using a relative entropy loss function to obtain the second loss.

8. The method according to claim 1 or 2 or 5 or 6, characterized in that: The step of performing gradient optimization on the structural parameters of the candidate operations in the current network to be optimized based on the continuous processing result includes: Performing second-order approximate estimation processing on the network parameters in the current network to be optimized and the continuous processing result to obtain a second-order approximate estimation result; Based on the second-order approximate estimation result, alternating gradient optimization is performed on the network parameters and structural parameters in the current network to be optimized.

9. An image classification method, characterized in that: include: Get the image to be classified; A target image processing network for image classification is used to classify the image to be classified to obtain a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the method according to any one of claims 1 to 8.

10. An image processing network generation device, characterized in that: include: A structural unit acquisition module, used to acquire an initial network structural unit, wherein the initial network structural unit includes at least one node representing a feature graph, and there are connecting edges between the nodes; An operation set acquisition module, used to acquire a candidate operation set, wherein the candidate operation set includes at least one candidate operation corresponding to each of the connection edges, and the candidate operation is an operation for processing a feature graph; A probability calculation module, used to generate a probability distribution of selecting a corresponding candidate operation for each of the connection edges in the current forward propagation according to the structural parameters of the candidate operation corresponding to each of the connection edges; A single operation sampling module, configured to select only one candidate operation from the corresponding candidate operations for each connection edge according to the probability distribution corresponding to each connection edge, and add it to the initial network structure unit to obtain a current network to be optimized; An optimization module is used to perform discrete distribution continuation processing on the probability distribution, and based on the continuation processing result, perform gradient optimization on the structural parameters of the candidate operations in the current network to be optimized until the predetermined optimization conditions are met to obtain a target image processing network; wherein, the gradient optimization of the structural parameters of the candidate operations in the current network to be optimized based on the continuation processing result comprises: using the current network to be optimized and the preset image processing network to process the sample image respectively to obtain an image processing result; according to the image processing result and the network middle layer of the preset image processing network, the structural parameters of the candidate operations in the current network to be optimized are gradient optimized based on the continuation processing result.

11. An image classification device, characterized in that: include: An image acquisition module, used to acquire images to be classified; A classification module is used to classify the image to be classified using a target image processing network for image classification to obtain a classification result corresponding to the image to be classified, wherein the target image processing network is generated according to the method described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

13. An electronic device, characterized in that: include: a memory storing a computer program; A processor reads a computer program stored in a memory to execute the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method of any one of claims 1 to 9 is implemented.

Images