Classification network training method, classification method, device and electronic equipment

By adding an intermediate feature extraction layer to the classification network, the problem of insufficient expression ability of the existing classification network in migration tasks is solved, the network performance and applicability are improved, and it is suitable for a variety of downstream tasks.

CN112308145BInactive Publication Date: 2024-11-19BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202011197979.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-30
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When the existing classification network structure is migrated to other tasks, its expression capabilities and performance are insufficient, making it difficult to meet the needs of multiple downstream tasks.

Method used

Based on the initial classification network, N intermediate feature extraction layers are added to connect between the first convolution layer and the second convolution layer. Through these intermediate feature extraction layers, the convolution output results are further processed to improve the goal Performance and expression capabilities of classification networks.

Benefits of technology

By adding an intermediate feature extraction layer, the performance and expression capabilities of the classification network are improved, making it perform better in a variety of downstream tasks, and is suitable for object detection, segmentation, key point detection and other scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112308145B_ABST
    Figure CN112308145B_ABST
Patent Text Reader

Abstract

The present application discloses a classification network training method, a classification method, a device, and an electronic device, etc., and relates to the fields of artificial intelligence technology such as deep learning and computer vision. The implementation scheme is: obtain a first training set; train the initial classification network based on the first training set to obtain a target classification network; wherein the initial classification network includes a first residual network, and the first residual network includes a first convolutional layer, N intermediate feature extraction layers, and a second convolutional layer, and N is an integer. That is, the initial classification network to be trained not only includes the first residual network, and the first residual network includes the first convolutional layer and the second convolutional layer, but also adds N intermediate feature extraction layers between the first convolutional layer and the second convolutional layer. The convolution output result of the first convolutional layer can be further feature extracted through the intermediate feature extraction layer, and then the obtained result is input into the second convolutional layer for further processing. In this way, the expression ability of the target classification network can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology such as deep learning and computer vision in computer technology, and in particular to a classification network training method, classification method, device and electronic equipment. Background Art

[0002] With the development of deep learning, computer vision has been applied in various scenarios, such as pedestrian detection, driverless driving, smart retail, etc., all of which rely on deep learning models to complete specific tasks. At present, the combination of industry and deep learning is a general trend. On the one hand, it can improve the effect, and on the other hand, it can reduce labor costs. At present, the basis of computer vision-related tasks based on deep learning is the classification network. A good classification network structure can be directly migrated to other related tasks, thereby realizing the upgrade of a series of tasks. Generally speaking, a classification network structure can be migrated to many downstream tasks, such as object detection, object segmentation, key point detection and other tasks.

[0003] At present, the classification network can be based on the residual network. The existing residual network structure is a direct connection structure formed by multiple convolutional layers. The input features will have a direct connection, and the fully connected layer of the classification network obtains the classification result through the output of the residual network to achieve classification. Summary of the invention

[0004] The present application provides a classification network training method, a classification method, a device and an electronic device.

[0005] In a first aspect, an embodiment of the present application provides a classification network training method, the method comprising:

[0006] Get the first training set;

[0007] The initial classification network is trained based on the first training set to obtain a target classification network; wherein the initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0008] In the classification network training method of the present embodiment, the initial classification network to be trained includes not only the first residual network, but also the first convolutional layer and the second convolutional layer in the first residual network. In addition, N intermediate feature extraction layers are added between the first convolutional layer and the second convolutional layer. The convolution output result of the first convolutional layer can be further feature extracted through the intermediate feature extraction layer, and then the obtained result is input to the second convolutional layer for further processing. In this way, by training the initial classification network including the above-mentioned first residual network to obtain the target classification network, the performance of the target classification network can be improved. At the same time, the expression ability of the target classification network can be improved.

[0009] In a second aspect, an embodiment of the present application provides a classification network training device, the device comprising:

[0010] A first acquisition module, used to acquire a first training set;

[0011] A training module, used for training the initial classification network based on the first training set to obtain a target classification network;

[0012] The initial classification network includes a first residual network, and the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0013] In a third aspect, an embodiment of the present application provides a classification method, the method comprising:

[0014] Get the object to be classified;

[0015] Inputting the object to be classified into a target classification network, and outputting a classification result of the object to be classified through the target classification network;

[0016] The target classification network is trained based on an initial classification network, and the initial classification network includes a first residual network, and the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0017] Classification is performed based on the target classification network to obtain the classification result of the object to be classified, wherein the target classification network is obtained based on the training of the initial classification network, and the initial classification network not only includes the first residual network, the first residual network includes the first convolution layer and the second convolution layer, but also adds N intermediate feature extraction layers between the first convolution layer and the second convolution layer, and the convolution output result of the first convolution layer can be further extracted through the intermediate feature extraction layer, and then the obtained result is input into the second convolution layer for further processing. In this way, by training the initial classification network including the above-mentioned first residual network to obtain the target classification network, the performance of the target classification network can be improved, and classification through the target classification network can improve the classification accuracy.

[0018] In a fourth aspect, an embodiment of the present application provides a classification device, the device comprising:

[0019] A first acquisition module, used to acquire a first training set;

[0020] A training module, used for training the initial classification network based on the first training set to obtain a target classification network;

[0021] The initial classification network includes a first residual network, and the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0022] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the classification network training method or classification method provided in each embodiment of the present application.

[0026] In a sixth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the classification network training method or classification method provided in each embodiment of the present application.

[0027] In a seventh aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the classification network training method or classification method provided in each embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0029] Figure 1 This is one of the flow charts of a classification network training method according to an embodiment of the present application;

[0030] Figure 2 is a structural diagram of a first residual network of an embodiment provided by the present application;

[0031] Figure 3 This is the second flow chart of a classification network training method according to an embodiment of the present application;

[0032] Figure 4 It is a flow chart of a classification method of an embodiment provided by the present application;

[0033] Figure 5 is a structural diagram of a classification network training device according to an embodiment of the present application;

[0034] Figure 6 is a structural diagram of a classification device according to an embodiment of the present application;

[0035] Figure 7 It is a block diagram of an electronic device used to implement the classification network training method of an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0037] like Figure 1 As shown, according to an embodiment of the present application, the present application provides a classification network training method, which can be used in electronic devices, and the method includes:

[0038] Step S101: Obtain a first training set.

[0039] The first training set may include multiple training objects, for example, the training objects may be training images, etc., or the first training set may include first features of multiple training objects, for example, the training objects are training images, and the first features of the training objects may be features obtained by convolution operations on the training objects, etc.

[0040] Step S102: training the initial classification network based on the first training set to obtain a target classification network;

[0041] Wherein, the initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, and N is an integer.

[0042] It can be understood that the N intermediate feature extraction layers are connected between the first convolution layer and the second convolution layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

[0043] In the classification network training method of the present embodiment, the initial classification network to be trained includes not only the first residual network, but also the first convolutional layer and the second convolutional layer in the first residual network. In addition, N intermediate feature extraction layers are added between the first convolutional layer and the second convolutional layer. The convolution output result of the first convolutional layer can be further feature extracted through the intermediate feature extraction layer, and then the obtained result is input to the second convolutional layer for further processing. In this way, by training the initial classification network including the above-mentioned first residual network to obtain the target classification network, the performance of the target classification network can be improved. At the same time, the expression ability of the target classification network can be improved.

[0044] As an example, there is also a correlation between two adjacent intermediate feature extraction layers. For example, the input of each of the N intermediate feature extraction layers except the first intermediate feature extraction layer includes a portion of the output results of the previous intermediate feature extraction layer adjacent to it, the input of the first intermediate feature extraction layer includes a portion of the convolution results output by the first convolution layer, and the input of each of the remaining intermediate feature extraction layers also includes a portion of another portion of the convolution results output by the first convolution layer.

[0045] In one embodiment, the output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the first intermediate feature extraction layer includes the second first sub-convolution result, the input of the second convolution layer includes the first first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

[0046] The output result of each intermediate feature extraction layer includes a part of the output result and another part of the output result. The part of the output result of each intermediate feature extraction layer in the first N-1 intermediate feature extraction layers must be input to the next intermediate feature extraction layer adjacent to it, and the other part of the output result of each intermediate feature extraction layer in the first N-1 intermediate feature extraction layers is input to the second convolution layer, and the other part of the output result of the i-1th intermediate feature extraction layer is input to the second convolution layer, 2≤i≤N, that is, the other part of the output result of the above N-1 intermediate feature extraction layers includes the other part of the output result of the 1st to N-1th intermediate feature extraction layers, and the output result of the Nth intermediate feature extraction layer is all input to the second convolution layer. The above M is an integer. In one example, the value of M can be N+1.

[0047] As an example, the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layer, and the output result of the Nth intermediate feature extraction layer are fused and input to the second convolution layer, that is, it can be understood that the input of the second convolution layer includes the fusion result of the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layer, and the output result of the Nth intermediate feature extraction layer. The above fusion can be understood as splicing, for example, N is 3, the 1st first sub-convolution result is a result in the form of 10×10×5 (which can be understood as 5 matrices of size 10×10, and 5 can be understood as the number of channels), another part of the output result of the 1st intermediate feature extraction layer is a result in the form of 10×10×5 dimensions, another part of the output result of the 2nd intermediate feature extraction layer is a matrix of the result in the form of dimensions, and another part of the output result of the 10th intermediate feature extraction layer is a result in the form of 10×10×10 dimensions. Then the fused result is a result in the form of 10×10×35 dimensions.

[0048] The initial classification network is trained based on the first training set to obtain a target classification network. In this way, the target classification network may include a trained first residual network (the result of training the first residual network in the initial classification network). That is, the structure of the trained target classification network is the same as the structure of the initial classification network, except that the values ​​of the parameters in the trained network are different. The values ​​of the parameters of the target classification network are the result of continuously adjusting the initial values ​​of the parameters in the initial classification network during the training process.

[0049] In the classification network training method of the present embodiment, the initial classification network including the first residual network is trained to obtain the target classification network. Since the first residual network in the present embodiment includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, the output of the first convolutional layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output result of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the first intermediate feature extraction layer includes the second first sub-convolution result, the input of the second convolutional layer includes the first first sub-convolution result, another part of the output result of the N-1 intermediate feature extraction layer and the output result of the N-th intermediate feature extraction layer. The result is that the output results of each intermediate feature extraction layer of the first N-1 intermediate feature extraction layers are divided into two parts, one part of the output results is used as the input of the second convolution layer, and the other part of the output results and the first sub-convolution result are input into the next adjacent intermediate feature extraction layer for further processing. The second convolution layer obtains the output of the second convolution layer based on the input of the second convolution layer. In this way, the first residual network is no longer a direct connection structure of the convolution layer, but N intermediate feature extraction layers are added to the first residual network. Through the structure of the N intermediate feature extraction layers in the first residual network, the output result of the first convolution layer can be obtained as the input of the second convolution layer through different processing, and the second convolution layer obtains the output of the second convolution layer based on the input of the second convolution layer. In this way, by training the initial classification network of the first residual network including the above structure, the performance and expression ability of the obtained target classification network are improved.

[0050] In one embodiment, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0051] It should be noted that the normalization layer is connected between the third convolution layer and the activation layer, the input of the third convolution layer includes part of the convolution result output by the first convolution layer, the normalization input includes the output of the third convolution layer, the input of the activation layer includes the output of the normalization layer, and the output of the activation layer is the output of the intermediate feature extraction layer. The intermediate feature extraction layer including the third convolution layer, the normalization layer and the activation layer connected in sequence can process part of the convolution result output by the first convolution layer, and then perform normalization processing, and the activation layer further processes the normalized result and outputs it, so that the expression ability and performance of the entire first residual network can be improved.

[0052] As an example, the initial classification network also includes a pooling layer and a fully connected layer, that is, the initial classification network may include a first residual network, a pooling layer, and a fully connected layer connected in sequence. It can be understood that the first residual network is connected to the pooling layer, the pooling layer is connected to the fully connected layer, the input of the pooling layer includes the output of the first residual network, and the input of the fully connected layer includes the output of the pooling layer. In this embodiment, the output of the first residual network is the sum of the output of the second convolutional layer and the input of the first convolutional layer. It can be understood that the first residual network also includes a summation layer, whose input includes the output of the second convolutional layer and the input of the first convolutional layer, the output of the summation layer is the sum of the output of the second convolutional layer and the input of the first convolutional layer, and the output of the first residual network is the output of the summation layer.

[0053] The initial classification network is trained based on the first training set to obtain a target classification network. In this way, the target classification network may include a trained first residual network (the result of the first residual network training in the initial classification network), a trained pooling layer (the result of the pooling layer training in the initial classification network), and a trained fully connected layer (the result of the fully connected layer training in the initial classification network) connected in sequence, the input of the trained pooling layer includes the output of the trained first residual network, and the input of the trained fully connected layer includes the output of the trained pooling layer. That is, the structure of the target classification network obtained by training is the same as that of the initial classification network, except that the values ​​of the parameters in the network after training are different, and the values ​​of the parameters of the target classification network are the result of continuously adjusting the initial values ​​of the parameters in the initial classification network during the training process.

[0054] In this example, the second convolutional layer obtains the output of the second convolutional layer based on the input of the second convolutional layer and passes it to the pooling layer. The output of the pooling layer is input to the fully connected layer, and classification is achieved through the fully connected layer.

[0055] In one embodiment, the first residual network further includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

[0056] That is, in this embodiment, a weighted layer is added to the first residual network, and the output of the second convolutional layer is weighted by the weighted layer, so that the expression ability of the first residual network can be improved and the performance of the first residual network can be improved. As an example, the output of the second convolutional layer includes the output of multiple channels, and the weighted layer can perform weighted processing on the output of each channel to obtain a weighted result. As an example, the input of the summation layer includes the sum of the output of the weighted layer and the input of the first convolutional layer, and the input of the pooling layer includes the sum of the output of the weighted layer and the input of the first convolutional layer.

[0057] In one embodiment, the convolution kernels of the first convolution layer and the second convolution layer have the same size.

[0058] This can improve the consistency of the first convolutional layer and the second convolutional layer to ensure the performance of the first residual network. For example, the convolution kernel sizes of the first convolutional layer and the second convolutional layer can both be 1×1, etc. In the implementation of this application, the convolution kernel sizes of the first convolutional layer and the second convolutional layer may not be limited.

[0059] In one embodiment, the convolution kernels of the N third convolutional layers have the same size.

[0060] This can improve the consistency of the third convolution layer of each intermediate feature extraction layer to ensure the performance of the first residual network. For example, the convolution kernel size of the third convolution layer can be 3×3, etc. In the implementation of this application, the convolution kernel size of the third convolution layer may not be limited.

[0061] The above classification network training method process is described in detail below with a specific implementation.

[0062] First of all, the basic operation of the current classification network structure is as follows:

[0063] Since the classification network structure is composed of various basic operations stacked layer by layer, we only need to obtain the specific prediction time of each layer of operation, and then we can accurately estimate the prediction time of the entire network structure through simple linear accumulation.

[0064] The following specifically introduces several common basic operations of classification network structures, including but not limited to these basic operations:

[0065] (1) Convolution operation: Corresponding to the convolution layer, convolution first appeared in the signal and linear system. The signal and linear system discuss the changes that occur after the signal passes through a linear system. Since in reality, the output of a signal at the previous moment often affects the output at this moment, the unit response of the system is used to convolve with the input of the system to obtain the output signal of the system.

[0066] Convolution is the result of multiplying two variables within a certain range and then summing them. The definition of the convolution operation is shown in the following formula:

[0067]

[0068] in, Discrete function, for the domain Ω r :[-r,r] 2 ∩Z 2 , k is defined in The size of the above is (2r+1) 2 Discrete filter function, * is the discrete convolution operator.

[0069] The convolution mentioned in this application is a convolution operation for images. The dimensions of the input data and the output data are in the form of N×C×H×W. This form indicates that the data is a tensor with a total of 4 dimensions, where N represents the number of data, C: represents the number of channels of the data, H represents the height of the data, and W represents the width of the data. The predicted time consumption of the convolution operation is controlled by multiple parameters, specifically the following parameters:

[0070] Input Channel Number (Input_Channel): refers to the number of channels of the input data of the convolution operation, corresponding to the C dimension;

[0071] Output Channel Number (Output_Channel): refers to the number of channels of output data after convolution operation, corresponding to the C dimension;

[0072] Input feature map size (Input_size): refers to the size of the input data of the convolution operation, corresponding to the H and W dimensions;

[0073] Convolution kernel size (Kernel_size): refers to the size of the convolution kernel used for convolution operations;

[0074] Stride: refers to the step size of the convolution operation;

[0075] Padding: refers to the number of padding for convolution operation;

[0076] Group number: refers to the number of groups for convolution operation.

[0077] Therefore, all available convolution operations can be generated by enumerating common parameter values.

[0078] (2) Pooling operation: corresponds to the pooling layer, which is one of the commonly used components in the current convolutional neural network, also known as downsampling. The pooling layer imitates the human visual system to reduce the dimension of the data and represents the image with higher-level features. Pooling operations include: maximum pooling, mean pooling, random pooling, and median pooling. The prediction time of the pooling operation is controlled by the following parameters:

[0079] Pooling size: refers to the size of the pool in the pooling operation;

[0080] Stride: refers to the step length of the pooling operation;

[0081] Padding: refers to the number of paddings for pooling operations;

[0082] Therefore, all available pooling operations can be generated by enumerating common parameter values.

[0083] (3) Linear operation: also refers to the fully connected operation, corresponding to the fully connected layer, which plays the role of "classifier" in the entire convolutional neural network. If the operations such as the convolution layer, pooling layer and activation function layer are to map the original data to the hidden layer feature space, the fully connected layer plays the role of mapping the learned "distributed feature representation" to the sample label space. The prediction time of the linear operation is controlled by the following parameters:

[0084] Input dimension: refers to the dimension of linear operation input data;

[0085] Output dimension: refers to the dimension of linear operation output data;

[0086] Therefore, all available linear operations can be generated by enumerating common parameter values.

[0087] (4) Activation function: Corresponding to the activation layer, the activation function is used to add nonlinear factors because the linear model is not expressive enough. Common activation functions include: Relu, PRelu, Sigmoid, Tanh, Swish, etc.

[0088] (5) Normalization operation: For the normalization layer, normalization can be achieved through operations such as BatchNorm / LayerNorm / InstanceNorm. For each hidden layer neuron, the input distribution that gradually approaches the saturation area of ​​the value range after being mapped to the nonlinear function is forced to return to a relatively standard normal distribution with a mean of 0 and a variance of 1, so that the input value of the nonlinear transformation function falls into an area that is more sensitive to the input, thereby avoiding the gradient vanishing problem.

[0089] Then, the embodiment of the present application provides a classification network structure based on an improved first residual network, such as Figure 2 As shown, it is a structural diagram of the first residual network of an embodiment of the present application. The training process of the classification network of the embodiment of the present application is as follows:

[0090] First, construct the initial classification network, which includes the first residual network, pooling layer and fully connected layer. The structure of the first residual network is as follows: Figure 2 As shown, in Figure 2 In the example, N is 4 and M is 5.

[0091] In the first residual network, the first input data input to the first convolutional layer is processed by the first convolutional layer, and the first convolution result includes 5 first sub-convolution results. It can be understood that the output of the first convolutional layer is divided into 5 parts, that is, 5 first sub-convolution results, which are respectively Figure 2The first sub-convolution result X1, the second first sub-convolution result X2, the third first sub-convolution result X3, the fourth first sub-convolution result X4 and the fifth first sub-convolution result X5 are included. The first first sub-convolution result X1 is used as the first sub-output Y1. The second first sub-convolution result X2 is processed by the first intermediate feature extraction layer A1, and the output result of the first intermediate feature extraction layer A1 includes a part of the output result X20 and another part of the output result X21. The other part of the output result X21 of the first intermediate feature extraction layer A1 is used as the second sub-output Y2. The third first sub-convolution result X3 and a part of the output result X20 of the first intermediate feature extraction layer A1 are fused (for example, spliced) as the input of the second intermediate feature extraction layer A2. After being processed by the second intermediate feature extraction layer A2, the output result of the second intermediate feature extraction layer A2 includes a part of the output result X30 and another part of the output result X31. The other part of the output result X31 of the second intermediate feature extraction layer A2 is the third sub-output Y3. The fourth first sub-convolution result X4 and a part of the output result X30 of the second intermediate feature extraction layer A2 are fused as the input of the third intermediate feature extraction layer A3. After being processed by the third intermediate feature extraction layer A3, the output result of the third intermediate feature extraction layer A3 includes a part of the output result X40 and another part of the output result X41. The other part of the output result X41 of the third intermediate feature extraction layer A3 is the fourth sub-output Y4. The 5th first sub-convolution result X5 and a part of the output result X40 of the 3rd intermediate feature extraction layer A3 are fused as the input of the 4th intermediate feature extraction layer A4, and after being processed by the 4th intermediate feature extraction layer A4, the output result of the 4th intermediate feature extraction layer A4 is used as the 5th sub-output Y5. In this way, 5 sub-outputs are obtained, including self-outputs Y1, Y2, Y3, Y4 and Y5. It can be understood that another part of the output results of the N-1 intermediate feature extraction layer include X21, X31, and X41.

[0092] The above Y1, Y2, Y3, Y4 and Y5 are fused as the input of the second convolutional layer. The output of the second convolutional layer is passed to the weighted layer (SE Block) for weighting to obtain a weighted result. The weighted result is added to the first input data input to the first convolutional layer through the addition module to obtain the output of the first residual network. The output of the first residual network is passed to the pooling layer. After the pooling layer receives the output of the first residual network for processing, the output of the pooling layer is passed to the fully connected layer.

[0093] Then, based on the constructed initial classification network, the initial classification network is trained. The training process is as follows: Figure 3As shown, the first training set is input into the initial classification network for training, the parameters in the initial classification network are updated, the first verification set (including multiple verification objects, such as verification images, and the classification identification of each verification object) is input into the network, the performance of the network is verified, and it is determined whether the training iteration round threshold is reached. If it is reached, the training ends and the target classification network is obtained. If it is not reached, the iteration round is updated, and the step of inputting the first training set into the initial classification network for training is returned to continue training until the training iteration round threshold is reached. After the training is completed, the target classification network is obtained, and the target classification network can be used for classification later.

[0094] In the classification network training method of the embodiment of the present application, an initial classification network is first constructed, and multiple intermediate feature extraction layers are added to the first residual network in the initial classification network. A part of the M first sub-convolution results of the first convolution layer in the first residual network can be directly used as the input of the second convolution layer, and each first sub-convolution result in the other part can be processed by different intermediate feature extraction layers. There is also a correlation between the intermediate feature extraction layers. For example, the input of an intermediate feature extraction layer not only includes a first sub-convolution result, but also includes a part of the output result of the previous intermediate feature extraction layer. The initial classification network includes the first residual network of the above structure, and the initial classification network is trained to obtain a target classification network, which can achieve a target classification network with better performance under the same model size and model prediction speed, and improve the network's expression ability. The first residual network in this embodiment can be applied to the classification base network, the backbone network of the detection model, and the backbone network assigned to the model. The trained target classification network can be applied in scenes such as target detection, target segmentation, key point detection, and network structure search. Only the corresponding backbone network or search space needs to be modified to obtain a large performance benefit.

[0095] like Figure 4 As shown, according to an embodiment of the present application, the present application provides a classification method, the method comprising:

[0096] Step S401: obtaining an object to be classified;

[0097] The object to be classified may include an image to be classified or a first feature of the image to be classified. The first feature of the image to be classified is a feature obtained after a convolution operation is performed on the image to be classified.

[0098] Step S402: inputting the object to be classified into the target classification network, and outputting the classification result of the object to be classified through the target classification network;

[0099] The target classification network is trained based on the initial classification network. The initial classification network includes a first residual network. The first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0100] In the classification method of this embodiment, classification is performed based on a target classification network to obtain a classification result of an object to be classified, wherein the target classification network is obtained based on training of an initial classification network, and the initial classification network includes not only a first residual network, but also a first convolutional layer and a second convolutional layer in the first residual network, and N intermediate feature extraction layers are added between the first convolutional layer and the second convolutional layer, and the convolution output result of the first convolutional layer can be further feature extracted through the intermediate feature extraction layer, and then the obtained result is input into the second convolutional layer for further processing. In this way, by training the initial classification network including the above-mentioned first residual network to obtain a target classification network, the performance of the target classification network can be improved, and classification through the target classification network can improve classification accuracy. At the same time, the expression ability of the target classification network can be improved.

[0101] It should be noted that the target classification network can be obtained by training the initial classification network based on the first training set.

[0102] In one embodiment, the output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the first intermediate feature extraction layer includes the second first sub-convolution result, the input of the second convolution layer includes the first first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

[0103] In one embodiment, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0104] In one embodiment, the first residual network further includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

[0105] In one embodiment, the convolution kernels of the first convolution layer and the second convolution layer have the same size.

[0106] This can improve the consistency of the first convolutional layer and the second convolutional layer to ensure the performance of the first residual network. For example, the convolution kernel sizes of the first convolutional layer and the second convolutional layer can both be 1×1, etc. In the implementation of this application, the convolution kernel sizes of the first convolutional layer and the second convolutional layer may not be limited.

[0107] In one embodiment, the convolution kernels of the N third convolutional layers have the same size.

[0108] like Figure 5As shown, according to an embodiment of the present application, the present application also provides a classification network training device 500, the device comprising:

[0109] A first acquisition module 501, used to acquire a first training set;

[0110] A training module 502 is used to train the initial classification network based on the first training set to obtain a target classification network;

[0111] The initial classification network includes a first residual network, and the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0112] In one embodiment, the output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the first intermediate feature extraction layer includes the second first sub-convolution result, the input of the second convolution layer includes the first first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

[0113] In one embodiment, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0114] In one embodiment, the first residual network further includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

[0115] The convolution kernel sizes of the first and second convolution layers are the same.

[0116] In one embodiment, the convolution kernels of the N third convolutional layers have the same size.

[0117] The classification network training device of each of the above embodiments is a device for implementing the classification network training method of each of the above embodiments, and the technical features and technical effects correspond to each other, which will not be repeated here.

[0118] like Figure 6 As shown, according to an embodiment of the present application, the present application further provides a classification device 600, the device comprising:

[0119] The second acquisition module 601 is used to acquire the object to be classified;

[0120] A classification module 602 is used to input the object to be classified into a target classification network, and output the classification result of the object to be classified through the target classification network;

[0121] The target classification network is trained based on the initial classification network. The initial classification network includes a first residual network. The first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer.

[0122] In one embodiment, the output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the first intermediate feature extraction layer includes the second first sub-convolution result, the input of the second convolution layer includes the first first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

[0123] In one embodiment, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0124] In one embodiment, the first residual network further includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

[0125] In one embodiment, the convolution kernels of the first convolution layer and the second convolution layer have the same size.

[0126] In one embodiment, the convolution kernels of the N third convolutional layers have the same size.

[0127] The classification device of each of the above embodiments is a device for implementing the classification method of each of the above embodiments, and the technical features and technical effects correspond to each other, which will not be repeated here.

[0128] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0129] like Figure 7 , is a block diagram of an electronic device according to a classification network training method or classification method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0130] like Figure 7As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphic information of the GUM on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0131] The memory 702 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the classification network training method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the classification network training method or classification method provided in the present application.

[0132] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the classification network training method in the embodiment of the present application (for example, the attached Figure 5 The first acquisition module 501 and the training module 502 shown in the figure, or the program instructions / modules corresponding to the classification method in the embodiment of the present application (for example, the attached Figure 7 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the classification network training method or classification method in the above method embodiment is implemented.

[0133] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device displayed by the keyboard, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device displayed by the keyboard via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The electronic device of the classification network training method or classification method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0135] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device displayed by the keyboard, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0136] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASMCs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0141] According to the technical solution of the embodiment of the present application, the initial classification network to be trained not only includes the first residual network, but also includes the first convolutional layer and the second convolutional layer. In addition, N intermediate feature extraction layers are added between the first convolutional layer and the second convolutional layer. The convolution output result of the first convolutional layer can be further feature extracted through the intermediate feature extraction layer, and then the obtained result is input to the second convolutional layer for further processing. In this way, by training the initial classification network including the above-mentioned first residual network to obtain the target classification network, the performance of the target classification network can be improved.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0143] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A classification network training method, the method comprising: Acquire a first training set, wherein the first training set includes a plurality of training objects or first features of a plurality of training objects, wherein the training objects are training images; Training the initial classification network based on the first training set to obtain a target classification network; The initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer; The output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the 1st intermediate feature extraction layer includes the 2nd first sub-convolution result, the input of the second convolution layer includes the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

2. According to the method of claim 1, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer and an activation layer connected in sequence.

3. According to the method according to claim 1, the first residual network also includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

4. A classification method, comprising: Acquire an object to be classified, where the object to be classified includes an image to be classified or a first feature of the image to be classified; Inputting the object to be classified into a target classification network, and outputting a classification result of the object to be classified through the target classification network; The target classification network is obtained based on the training of the initial classification network, the initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer; The output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the 1st intermediate feature extraction layer includes the 2nd first sub-convolution result, the input of the second convolution layer includes the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

5. According to the method of claim 4, each intermediate feature extraction layer includes a third convolutional layer, a normalization layer and an activation layer connected in sequence.

6. According to the method according to claim 4, the first residual network also includes a weighted layer, and the input of the weighted layer includes the output of the second convolutional layer.

7. A classification network training device, the device comprising: A first acquisition module is used to acquire a first training set, wherein the first training set includes a plurality of training objects or a first feature of a plurality of training objects, wherein the training objects are training images; A training module, used for training the initial classification network based on the first training set to obtain a target classification network; The initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer; The output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the 1st intermediate feature extraction layer includes the 2nd first sub-convolution result, the input of the second convolution layer includes the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

8. According to the device of claim 7, each intermediate feature extraction layer comprises a third convolution layer, a normalization layer and an activation layer connected in sequence.

9. A classification device, comprising: A second acquisition module is used to acquire an object to be classified, wherein the object to be classified includes an image to be classified or a first feature of the image to be classified; A classification module, used for inputting the object to be classified into a target classification network, and outputting a classification result of the object to be classified through the target classification network; The target classification network is obtained based on the training of the initial classification network, the initial classification network includes a first residual network, the first residual network includes a first convolutional layer, N intermediate feature extraction layers and a second convolutional layer, where N is an integer; The output of the first convolution layer includes M first sub-convolution results, the input of the i-th intermediate feature extraction layer includes the i+1-th first sub-convolution result and a part of the output results of the i-1-th intermediate feature extraction layer, i is an integer, and 2≤i≤N, the input of the 1st intermediate feature extraction layer includes the 2nd first sub-convolution result, the input of the second convolution layer includes the 1st first sub-convolution result, another part of the output results of the N-1 intermediate feature extraction layers, and the output result of the N-th intermediate feature extraction layer, and the output of the first residual network is the sum of the output of the second convolution layer and the input of the first convolution layer.

10. According to the device of claim 9, each intermediate feature extraction layer comprises a third convolution layer, a normalization layer and an activation layer connected in sequence.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the classification network training method described in any one of claims 1-3 or the classification method described in any one of claims 4-6.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the classification network training method described in any one of claims 1 to 3 or the classification method described in any one of claims 4 to 6.

Citation Information

Patent Citations

  • Vegetable image classification identification method and system

    CN108491765A

  • Image processing method and apparatus, and computer readable medium, and electronic device

    WO2019233244A1