Method, apparatus, device, and medium for constructing an image processing network

Abstract

An embodiment of the present invention provides a method, apparatus, device, and medium for constructing an image processing network. The method includes: obtaining a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch, and the backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained by fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to that layer of the distillation branch; training the network to be trained using sample images; and when the training end condition is satisfied, determining the backbone network branch in the network obtained at the end of training as the target image processing network.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, apparatus, device, and medium for constructing an image processing network. Background Art

[0002] Image processing is a core technology in computer vision. With the popularization of deep learning, image processing plays an important role in driverless, robot navigation, mobile phone image editing, etc. Image processing technology refers to obtaining the category of each pixel point from any image, so as to segment each object in the image at the pixel level. Generally speaking, the two main factors in image processing are classification expression ability and spatial expression ability.

[0003] In related technologies, a neural network model can be used to obtain classification expression ability and spatial expression ability, so as to realize image processing. When a traditional neural network model performs real-time image processing, either a small image or a small model is used to reduce the calculation amount, or a multi-branch structure is used to obtain classification expression ability and spatial expression ability.

[0004] However, the method of small images and small models will cause a serious decline in the performance of the segmentation model, and the multi-branch structure will inevitably use large images and deeper models, which makes it difficult to achieve real-time speed. Therefore, it is difficult to balance the accuracy and speed of image processing in the above two image methods. Thus, there is an urgent need to provide a method for fast and highly accurate real-time image processing. Summary of the Invention

[0005] In view of the above problems, a method, apparatus, device, and medium for constructing an image processing network according to an embodiment of the present invention are proposed to overcome or at least partially solve the above problems.

[0006] To solve the above problems, in a first aspect of the present invention, a method for constructing an image processing network is disclosed, and the method includes:

[0007] Obtain a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch, the backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained after fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input of this layer of the distillation branch;

[0008] Train the network to be trained using sample images;

[0009] When the training meets the training end condition, determine the backbone network branch in the network obtained at the end of training as the target image processing network.

[0010] Optionally, training the to-be-trained network using a sample image includes:

[0011] Determine the overall loss value of the to-be-trained network according to the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image;

[0012] Update each learnable parameter of the to-be-trained network according to the overall loss value of the to-be-trained network.

[0013] Optionally, determining the overall loss value of the to-be-trained network according to the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image includes:

[0014] Obtain a first loss value according to the output result of the decoding branch and the label of the sample image, and obtain a second loss value according to the output result of the distillation branch and the label of the sample image;

[0015] Obtain a first distillation loss value according to the difference between the output result of the distillation branch and the output result of the decoding branch;

[0016] Update each learnable parameter of the to-be-trained network according to the first loss value, the second loss value, and the first distillation loss value.

[0017] Optionally, the method further includes:

[0018] Obtain a fusion output result after fusing the output result of the decoding branch and the output result of the distillation branch;

[0019] Obtain a second distillation loss value according to the difference between the fusion output result and the output result of the decoding branch;

[0020] Determine the overall loss value of the to-be-trained network according to the first loss value, the second loss value, and the first distillation loss value, including:

[0021] Determine the overall loss value of the to-be-trained network according to the first loss value, the second loss value, the first distillation loss value, and the second distillation loss value.

[0022] Optionally, the method further includes:

[0023] The decoding branch includes an original network layer and a convolutional layer connected in series with the original network layer. The learnable parameters of the convolutional layer include: the weight values corresponding to the decoding branch and the distillation branch respectively; the fused output result is obtained according to the following steps:

[0024] According to the weight values corresponding to the decoding branch and the distillation branch respectively, perform weighted summation on the output result of the decoding branch and the output result of the distillation branch to obtain the fused output result.

[0025] Optionally, the fused feature is obtained according to the following steps:

[0026] According to the features of the same layer input by the distillation branch and the encoding branch, determine the corresponding covariance matrix;

[0027] Use the elements in the covariance matrix as the weights corresponding to the features of this layer input by the distillation branch to determine the increment corresponding to the features of this layer input by the distillation branch;

[0028] On the basis of the features of this layer input by the distillation branch, add the increment to obtain the fused feature.

[0029] Optionally, the distillation branch includes a first distillation branch and a second distillation branch; or

[0030] The distillation branch includes a first distillation branch group and a second distillation branch group. The first distillation branch group includes the first distillation branch and the second distillation branch, and the second distillation branch group includes a third distillation branch and a fourth distillation branch;

[0031] Wherein, the first distillation branch processes the sample image in the spatial dimension, the second distillation branch processes the sample image in the feature dimension, the third distillation branch processes the sample image in the feature dimension, and the fourth distillation branch processes the sample image in the spatial dimension.

[0032] Optionally, after obtaining the target image processing network, the method further includes:

[0033] Obtain an image to be processed;

[0034] Input the image to be processed into the target image processing network to obtain an image processing result.

[0035] In the second aspect of the embodiments of the present invention, an apparatus for constructing an image processing network is disclosed, including:

[0036] A network acquisition module, configured to acquire a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch, the backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained after fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to this layer of the distillation branch.

[0037] A training module, configured to train the network to be trained using sample images.

[0038] A determination module, configured to, when the training end condition is satisfied, determine the backbone network branch in the network obtained at the end of training as the target image processing network.

[0039] In a third aspect of the embodiments of the present invention, an electronic device is further disclosed, including:

[0040] One or more processors; and

[0041] One or more machine-readable media storing instructions thereon, which when executed by the one or more processors cause the device to execute the method for constructing an image processing network as described in the embodiments of the first aspect of the present invention.

[0042] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is further disclosed, and the computer program stored thereon causes a processor to execute the method for constructing an image processing network as described in the embodiments of the first aspect of the present invention.

[0043] The embodiments of the present invention have the following advantages:

[0044] In the embodiments of the present invention, the network to be trained includes a backbone network branch and a distillation branch. The backbone network branch includes an encoding branch, a decoding branch, and a distillation branch connected in series with the encoding branch. The network to be trained is trained using sample images, and the backbone network branch is retained at the end of training to obtain the target image processing network.

[0045] On the one hand, since each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained after fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to this layer of the distillation branch. Therefore, the input of the distillation branch incorporates some input features of the decoding branch, enabling the distillation branch to obtain better output features. During the training process, when using the distillation branch to distill the encoding branch, it can distill the decoding branch based on better output features, enabling the decoding branch to learn better parameters. Furthermore, the decoding branch can have a higher expression ability, thereby improving the accuracy of image processing.

[0046] On the other hand, since the embodiments of the present invention start from the decoding branch and introduce the method of distillation to improve the segmentation accuracy, at the end of training, the backbone network branch is retained, thus realizing a small model structure. Therefore, the efficiency of image processing can be guaranteed.

[0047] In summary of the above two aspects, it is possible to optimize the accuracy and speed during real-time image segmentation and achieve a balance between the accuracy and speed of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 is a flowchart of the steps of a method for constructing an image processing network in an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the network structure of the original image processing network in an embodiment of the present invention;

[0051] Figure 3 is a network structure diagram of a network to be trained in an embodiment of the present invention;

[0052] Figure 4 is a network structure diagram of another network to be trained in an embodiment of the present invention;

[0053] Figure 5 is a network structure diagram of another network to be trained in an embodiment of the present invention;

[0054] Figure 6-A is a schematic diagram of obtaining the fused feature of the same-layer input of the distillation branch and the decoding branch in an embodiment of the present invention;

[0055] Figure 6-B is a schematic diagram of obtaining the fused feature of the same-layer input of the distillation branch and the decoding branch in an embodiment of the present invention;

[0056] Figure 7 is a flowchart of the steps for determining the overall loss value in an embodiment of the present invention;

[0057] Figure 8 is in an embodiment of the present invention as Figure 3 shown in another schematic diagram of the network structure of the network to be trained;

[0058] Figure 9 is a structural block diagram of an image processing network construction device in an embodiment of the present invention. Detailed Implementation Manner

[0059] To make the above objects, features, and advantages of the present invention more apparent and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0060] In related technologies, when using a neural network model to obtain classification expression ability and spatial expression ability for real-time image segmentation, generally to improve the accuracy of image processing, better classification expression ability and spatial expression ability need to be achieved. If a deeper network structure is used, better classification ability can be achieved. However, because too deep a structure often leads to too low a resolution of the feature layer, thus losing the spatial description ability. Therefore, the main approach in related technologies is to design a neural network including an encoding layer and a decoding layer. The decoding layer is connected in series with the encoding layer. The encoding layer encodes the image into a set of high-dimensional features, and the decoding layer restores the high-dimensional features to an output of the same size as the original image. The high-dimensional features often correspond to the feature map after the original image is downsampled many times, so as to strengthen the influence of the shallow network on the final output result to increase the spatial description ability.

[0061] However, when adopting the above neural network, in order to achieve better efficiency, only small images or small models can be used to reduce the computational amount; and in order to achieve higher accuracy, only a multi-branch structure can be used, with one branch learning feature expression and the other branch increasing the spatial description ability.

[0062] However, when applied to the scenario of real-time image segmentation, the method of small images and small models will cause a serious decline in the performance of the segmentation model, and the multi-branch structure will inevitably use large images and deeper models, resulting in it being difficult to achieve real-time speed. Therefore, the above neural network cannot meet the requirements of both accuracy and efficiency at the same time.

[0063] In view of this, the inventor of the present invention has proposed a new network structure and training method. Specifically, starting from a very small backbone network, the method of distillation is introduced to improve the segmentation accuracy. At the end of training, only the backbone network is retained, so as to achieve optimization in terms of accuracy and speed.

[0064] Refer to Figure 1 , Figure 1 shows a flowchart of the steps of a method for constructing an image processing network according to an embodiment of the present invention. Among them, the method described in the present invention can be applied to a terminal device or a server, and specifically may include the following steps:

[0065] Step S101: Obtain a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch. The backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch.

[0066] Wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained by fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input of this layer of the distillation branch.

[0067] Specifically, the process of obtaining the network to be trained can be as described below:

[0068] First, obtain an original image processing network, where the original image processing network includes an encoding branch and a decoding branch connected in series with the encoding branch. Herein, the original image processing network is the backbone network branch in the network to be trained. Then, add a distillation branch connected in series with the encoding branch to the original image processing network to obtain the network to be trained.

[0069] In practice, a terminal device or a server can obtain the corresponding original image processing network according to the user's selection of the image processing network. Among them, the selected original image processing network is a neural network with image processing functions, and the image processing functions can be, but are not limited to, the following functions: image processing, image classification. In practice, the original image processing network can be a convolutional neural network.

[0070] Refer to Figure 2 , Figure 2 shows a schematic diagram of the network structure of the original image processing network. As Figure 2 shown, the original image processing network can include an encoding branch and a decoding branch. Among them, the combination of the encoding branch and the decoding branch can be called the backbone network branch in the network to be trained. Herein, the original image processing network in this embodiment can be a network structure of Xception.

[0071] In practice, the encoding branch is used to encode the input image into a set of high-dimensional features, and the decoding branch is used to restore the high-dimensional features to an output with the same size as the original image. The high-dimensional features often correspond to the feature map after the original image is downsampled many times.

[0072] In this embodiment, both the encoding branch and the decoding branch have multiple layers. Each layer of the encoding branch is used to downsample the input image, and each layer of the decoding branch is used to upsample the input image. From Figure 1It can be seen that each layer of the encoding branch is connected in series with the same layer of the decoding branch. For example, the encoding branch includes four layers, namely L1, L2, L3, and L4; while the decoding branch also includes four layers, namely L1', L2', L3', and L4'. Among them, L4' of the decoding branch is connected in series with L1 of the encoding branch, L3' is connected in series with L2 of the encoding branch, L2' is connected in series with L3 of the encoding branch, and L1' is connected in series with L4 of the encoding branch.

[0073] In practice, a distillation branch corresponding to the decoding branch can be preset in the terminal device or the server, and then the distillation branch can be automatically added to the original image processing network.

[0074] In this embodiment, the number of layers included in the distillation branch can be the same as the number of layers of the decoding branch. Specifically, the connection in series between the encoding branch and the distillation branch can also mean that each layer of the encoding branch is connected in series with the corresponding layer of the distillation branch, so that the distillation branch and the decoding branch can form two parallel branch structures.

[0075] In practice, the distillation branch can also perform the same function as the decoding branch, that is, restoring the high-dimensional features output by the encoding branch to an output with the same size as the original image.

[0076] Among them, the connection in series between each layer of the decoding branch and each layer of the distillation branch can mean that for each layer of the decoding branch, this layer can be connected in series with the same layer of the distillation branch, and the input to this layer of the decoding branch and the input to the same layer of the distillation branch are subjected to feature fusion to obtain a fused feature, and then this fused feature is used as the input to the same layer of the distillation branch.

[0077] In this way, the fusion of input features is achieved between each layer of the decoding branch and each layer of the distillation branch, thereby achieving the purpose of the decoding branch guiding the distillation branch, and the distillation branch can obtain better output features. When the distillation branch obtains better output features, in the subsequent training process, the distillation branch can in turn guide the learning of the decoding branch, so that the decoding branch has better performance. Thus, in the training process, the decoding branch and the distillation branch are in a mutual distillation manner, and in this way, the image processing performance of the decoding branch can be improved.

[0078] Step S102: Train the network to be trained using the sample images.

[0079] In this embodiment, multiple sample images can be pre-stored in the terminal device or the server. After adding the distillation branch to the original image processing network, multiple sample images can be automatically obtained, and then the sample images can be automatically input into the network to be trained, and then the network to be trained can be automatically trained.

[0080] In this embodiment, the sample image may refer to an image prepared in advance for training the network. Among them, the training of the network to be trained can be supervised training. Specifically, the loss values of the distillation branch and the decoding branch can be calculated according to the labels of the sample images, the outputs of the distillation branch and the decoding branch respectively, and then the parameters of the network to be trained can be updated according to the loss values of the distillation branch and the decoding branch respectively.

[0081] Since the input of the distillation branch fuses the input features of the corresponding layer of the decoding branch, the distillation branch can obtain better output features. In this way, when updating the parameters of the network to be trained according to the loss values of the distillation branch and the decoding branch respectively, the distillation branch can guide the learning of the decoding branch, so that the decoding branch has better performance, that is, the network to be trained has better spatial and feature expression capabilities, thereby improving the accuracy of the network to be trained for image processing.

[0082] Step S103: When the training end condition is satisfied, the backbone network branch in the network obtained at the end of training is determined as the target image processing network.

[0083] In this embodiment, the training end condition may refer to that the number of training times reaches a preset number, or the network loss obtained by training is less than the preset loss. At the end of training, the backbone network branch in the obtained network can be retained, and then the distillation branch can be discarded to obtain the target image processing network. Since the backbone network branch is retained at the end of training, the backbone network branch can be used for image processing subsequently, thereby reducing the branch structure of the target image processing network and realizing a small model structure. Therefore, the efficiency of image processing can be guaranteed.

[0084] By adopting the technical solution of the embodiment of the present invention, the optimization of the accuracy and speed during real-time image segmentation can be realized, and the balance between the accuracy and speed of image processing can be achieved.

[0085] Next, a detailed introduction will be given to several different networks to be trained obtained by adding a distillation branch connected in series with the encoding branch to the original image processing network.

[0086] In one implementation manner, when adding a distillation branch connected in series with the encoding branch to the original image processing network, it can be added in any of the following ways:

[0087] Method 1: Add a distillation branch, and the distillation branch can process the sample image from the feature dimension or the spatial dimension.

[0088] Refer to Figure 3 As shown in Figure 3The network structure diagram of a network to be trained obtained after adding a distillation branch is shown. Among them, each layer of the distillation branch is connected in series with each layer of the decoding branch, and the distillation branch can process the input image from the feature dimension or from the spatial dimension.

[0089] Method 2: Add two distillation branches. One distillation branch in the two distillation branches processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension.

[0090] Refer to Figure 4 as shown Figure 4 The network structure diagram of another network to be trained obtained after adding a distillation branch is shown. Among them, each layer of each added distillation branch is connected in series with the corresponding layer of the decoding branch, and different distillation branches process the input image from different dimensions. For example, distillation branch 1 processes the input image from the feature dimension, and distillation branch 2 processes the input image from the spatial dimension. Or, distillation branch 1 processes the input image from the spatial dimension, and distillation branch 2 processes the input image from the feature dimension.

[0091] Method 3: Add two groups of distillation branches. Each group of distillation branches includes two distillation branches. One distillation branch in each group processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension, and the order of the dimensions for which the two groups of distillation branches process the sample image is different.

[0092] Refer to Figure 5 as shown Figure 5 The network structure diagram of yet another network to be trained obtained after adding two groups of distillation branches is shown. Among them, when adding two groups of distillation branches, the order of the dimensions for which the two groups of distillation branches process the sample image being different can be: for the same group of distillation branches, one distillation branch in this distillation branch group can perform image processing from the feature dimension during the nth training and perform image processing from the spatial dimension during the (n + 1)th training; while the other distillation branch performs image processing from the spatial dimension during the nth training and performs image processing from the feature dimension during the (n + 1)th training.

[0093] In this embodiment, the spatial dimension can refer to the pixel dimension, and the feature dimension can refer to the channel dimension.

[0094] Combined with the network structures shown in the above 3 to Figure 5 as shown, the process of how to fuse the features input to the same layer of the distillation branch and the decoding branch in the embodiments of the present application is described:

[0095] In one example, the obtained fused feature can be the fused feature of the same-layer input of the distillation branch and the decoding branch obtained according to the following steps:

[0096] First, determine the corresponding covariance matrix according to the features of the same-layer input of the distillation branch and the decoding branch; second, use the elements in the covariance matrix as the weights corresponding to the features of the input of this layer of the distillation branch to determine the increment corresponding to the features of the input of this layer of the distillation branch; then, on the basis of the features of the input of this layer of the distillation branch, add the increment to obtain the fused feature.

[0097] In this embodiment, as Figure 3 shown, since the distillation branch can process the image in terms of the feature dimension or the spatial dimension, correspondingly, for each distillation branch, the covariance matrix of the corresponding dimension can be determined according to the dimension in which the distillation branch processes the sample image and the features of the input of the same layer of the decoding branch.

[0098] Among them, if the distillation branch is set to process the image in terms of the feature dimension, for the input of each layer of the decoding branch, the original input of this layer can be transposed corresponding to the feature dimension to obtain the transposed input, and the covariance matrix can be determined according to the original input and the transposed input, so that the elements in the covariance matrix are used as the weights corresponding to the features of the input of this layer of the distillation branch. Furthermore, according to the weights and the features of the input of this layer of the distillation branch, the increment can be determined, and then the increment is added to the features of the input of this layer of the distillation branch as the input of this layer of the distillation branch.

[0099] Exemplarily, as Figure 6-A shown, a schematic diagram of obtaining the fused feature of the same-layer input of the distillation branch and the decoding branch is shown. As Figure 6-A shown, Conv10 is the first layer in the decoding branch, and Conv20 is the first layer in the distillation branch. Assume that the input to the first layer of the decoding branch is C*H*W. Express this input as the input in the feature dimension, that is, obtain the input of C*HW. Then, transpose this input of C*HW, that is, rotate it 90 degrees to obtain HW*C, and calculate the feature of C_HW*HW_C = C*C. This C*C feature is the covariance matrix. Then, C*C is multiplied by the input C'*HW' of the first layer of the distillation branch to obtain the corresponding increment, and this increment is added to C'*HW' to obtain the fused feature.

[0100] Among them, if the distillation branch is set to process the image in the spatial dimension, for the input of each layer of the decoding branch, the original input of this layer can be transposed corresponding to the spatial dimension to obtain the transposed input, and based on the original input and the transposed input, the covariance matrix is determined. Thus, the elements in the covariance matrix are used as the weights corresponding to the features of the input of this layer of the distillation branch. Furthermore, based on the weights and the features of the input of this layer of the distillation branch, the increment can be determined, and then the increment is added to the features of the input of this layer of the distillation branch as the input of this layer of the distillation branch.

[0101] Exemplarily, as Figure 6-B shown, Conv10 is the first layer in the decoding branch, and Conv20 is the first layer in the distillation branch. Assume that the input to the first layer of the decoding branch is C*H*W. Expressing this input as an input in the spatial dimension, we get an input of HW*C. Then, transpose this input of HW*C, that is, rotate it by 90 degrees to get C*HW, and calculate the feature of HW_C*C_HW = HW*HW. This feature of HW*HW is the covariance matrix. Then, multiply HW*HW by the input HW'*C' of the first layer of the distillation branch to obtain the corresponding increment, and add this increment to HW'*C' to obtain the fused feature.

[0102] In one example, the process of training the network to be trained is introduced in detail. Specifically, the process of updating the parameters of the network to be trained is introduced, including the following process:

[0103] First, based on the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image, the overall loss value of the network to be trained is determined.

[0104] Next, based on the overall loss value of the network to be trained, each learnable parameter of the network to be trained is updated.

[0105] Since both the decoding branch and each distillation branch output an output result, the loss between each output result and the label of the sample image can be determined respectively. Furthermore, the losses obtained are weighted and summed to obtain the overall loss value, and then each learnable parameter of the network to be trained is updated based on this overall loss value.

[0106] Referring to Figure 7 shown, a flowchart of the steps for determining the overall loss value is shown. As Figure 7 shown, it specifically includes the following steps:

[0107] Step S701: Obtain a first loss value based on the output result of the decoding branch and the label of the sample image, and obtain a second loss value based on the output result of the distillation branch and the label of the sample image.

[0108] In this embodiment, when training the network to be trained, the label of the sample image can represent the true category to which the sample image belongs. Referring to Figure 4 as shown, taking the example of having two distillation branches, during the training process, the output result F0 of the decoding branch, and the output results F1 and F2 respectively output by the two distillation branches can be obtained. Since the distillation branches and the decoding branch have the same function, therefore, the outputs of both the decoding branch and the distillation branches represent the prediction results of segmenting the same sample image. Thus, the first loss value of the decoding branch and the second loss value of the distillation branch can be calculated respectively.

[0109] As Figure 4 shown, the first loss value corresponding to the decoding branch can be L0, and the loss values corresponding to the two distillation branches are L1 and L2 respectively.

[0110] Step S702: Obtain a first distillation loss value according to the difference between the output result of the distillation branch and the output result of the decoding branch.

[0111] In this embodiment, the differences between the output results of the distillation branches and the output result of the decoding branch can be determined. This difference can reflect the performance difference between the distillation branches and the decoding branch in object segmentation. Subsequently, the learning of the decoding branch can be guided according to this first distillation loss value. In practice, when there are multiple distillation branches, the differences between the output results of the distillation branches and the output result of the decoding branch can be calculated respectively, so as to obtain multiple first distillation loss values.

[0112] Specifically, when implementing, the difference between the output result corresponding to the decoding branch and the output result corresponding to the distillation branch can be used as the first distillation loss value. For example, as Figure 4 shown, the output result corresponding to the decoding branch is F0, and the output results F1 and F2 respectively output by the two distillation branches. Then the first distillation loss value between the decoding branch and distillation branch 1 can be loss_n1, loss_n1 = mse(F1, F0), and the first distillation loss value loss_n2 between the decoding branch and distillation branch 2, loss_n2 = mse(F2, F0). Where mse represents the mean square error.

[0113] Step S703: Determine the overall loss value of the network to be trained according to the first loss value, the second loss value, and the first distillation loss value.

[0114] In this embodiment, the above-obtained first loss value, second loss value, and first distillation loss value can be weighted and summed to obtain the overall loss of the model to be trained, so as to iteratively update each learnable parameter of the entire network to be trained according to the overall loss.

[0115] In one example, it is also possible to obtain a fusion output result after fusing the output result of the decoding branch and the output result of the distillation branch, and obtain a second distillation loss value according to the difference between the fusion output result and the output result of the decoding branch. Furthermore, the overall loss value of the network to be trained can be determined according to the first loss value, the second loss value, the first distillation loss value, and the second distillation loss value.

[0116] In this example, the output result of the decoding branch can be fused with the output result of the distillation branch to obtain a fused output result after fusion. The fused output result after fusion can represent the comprehensive performance of the decoding branch and the distillation branch in image processing. Among them, the second distillation loss value can be used to evaluate the comprehensive performance of the decoding branch and the distillation branch in image processing. In this way, in the follow-up, mutual distillation between the decoding branch and the distillation branch can be realized according to the second distillation loss value. Mutual distillation can be understood as follows: while the distillation branch guides the decoding branch to learn, the decoding branch can also guide the learning of the distillation branch, so that the decoding branch and the distillation branch can be made as consistent as possible, achieving a further improvement in the accuracy of the decoding branch.

[0117] Among them, in one example, the specific process of obtaining the fusion output result is described. In this example, a convolutional layer can be added after the last layer of the decoding branch, referring to Figure 8 as shown. Figure 8 shows the schematic diagram of the structure after adding a convolutional layer in the network structure to be trained as shown in Figure 3 as shown. As shown in Figure 8 a convolutional layer can be added after the last layer of the decoding branch. This convolutional layer is mainly used to assign respective weights to the decoding branch and the distillation branch.

[0118] Among them, the learnable parameters of the convolutional layer include: the respective weight values corresponding to the decoding branch and the distillation branch; then the fusion output result can be obtained by performing weighted summation on the output result of the decoding branch and the output result of the distillation branch according to the respective weight values corresponding to the decoding branch and the distillation branch.

[0119] In this example, the respective weight values corresponding to the decoding branch and the distillation branch can represent the proportion of their respective output results. Thus, the comprehensive performance of the decoding branch and the distillation branch can be obtained. Among them, the respective weight values corresponding to the decoding branch and the distillation branch can be learned during the training of the network to be trained. Of course, in practice, initial weight values can be set at the beginning of the training, and then, during the training process, these initial weight values can be continuously updated.

[0120] Exemplarily, as shown in Figure 8As shown, after the weight assignment of the convolutional layer, a weighted sum of the output result of the decoding branch and the output result of the distillation branch is performed to obtain a fused output result Ft, and the corresponding second distillation loss value is Lt = mse(Ft, gt), where gt represents the label of the sample image.

[0121] In practice, when iteratively updating a preset number of times according to the overall loss value, or until the loss value corresponding to the decoding branch is less than the preset loss value, the update can be ended, thereby obtaining the network after training is completed. Subsequently, in the network after training is completed, the distillation branch can be discarded, and the remaining encoding branch and decoding branch can be used as the target image processing network. Among them, the image processing function implemented by the target image processing network is consistent with the image processing function implemented by the original image processing network.

[0122] After obtaining the above-mentioned target image processing network, the target image processing network can be used for image processing. Specifically, an image to be processed can be obtained, and the image to be processed is input into the target image processing network to obtain an image processing result.

[0123] Of course, in some examples, the backbone network branch can be an image processing network for image classification. In this way, the target image processing network can also be an image processing network for image classification, so as to realize the classification processing of images. That is to say, the method for constructing the image processing network of the present application is not limited to the neural network for image processing, and can also be applied to other image processing, such as the neural network for image classification.

[0124] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0125] Refer to Figure 9 As shown, a structural block diagram of an apparatus for constructing an image processing network according to an embodiment of the present invention is shown. As Figure 9 shown, the apparatus may specifically include the following modules:

[0126] A network acquisition module 901, configured to acquire a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch, the backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained by fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to this layer of the distillation branch;

[0127] A training module 902, configured to train the network to be trained by using a sample image;

[0128] A determination module 903, configured to, when a training end condition is satisfied, determine the backbone network branch in the network obtained at the end of training as a target image processing network.

[0129] Optionally, the training module 902 may specifically include the following units:

[0130] An overall loss determination unit, configured to determine an overall loss value of the network to be trained according to the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image;

[0131] An update unit, configured to update each learnable parameter of the network to be trained according to the overall loss value of the network to be trained.

[0132] Optionally, the overall loss determination unit may specifically include the following subunits:

[0133] A first loss value acquisition subunit, configured to acquire a first loss value according to the output result of the decoding branch and the label of the sample image, and acquire a second loss value according to the output result of the distillation branch and the label of the sample image;

[0134] A second loss value acquisition subunit, configured to acquire a first distillation loss value according to the difference between the output result of the distillation branch and the output result of the decoding branch;

[0135] A determination subunit, configured to determine the overall loss value of the network to be trained according to the first loss value, the second loss value, and the first distillation loss value.

[0136] Optionally, the overall loss determination unit may specifically include the following subunits:

[0137] A third loss value acquisition unit, configured to acquire a fused output result after fusing the output result of the decoding branch and the output result of the distillation branch, and acquire a second distillation loss value according to the difference between the fused output result and the output result of the decoding branch;

[0138] The determining subunit is specifically configured to determine the overall loss value of the network to be trained according to the first loss value, the second loss value, the first distillation loss value, and the second distillation loss value.

[0139] Optionally, the decoding branch includes an original network layer and a convolutional layer connected in series with the original network layer. The learnable parameters of the convolutional layer include: the weight values corresponding to the decoding branch and the distillation branch respectively; the apparatus further includes the following modules:

[0140] A convolutional layer adding module, configured to add a convolutional layer after the last layer of the decoding branch. The learnable parameters of the convolutional layer include: the weight values corresponding to the decoding branch and the distillation branch respectively; the fused output result is obtained according to the following steps:

[0141] A fusion module, configured to perform weighted summation on the output result of the decoding branch and the output result of the distillation branch according to the weight values corresponding to the decoding branch and the distillation branch respectively, to obtain the fused output result.

[0142] Optionally, the fused feature is obtained according to the following steps: determining a corresponding covariance matrix according to the features input to the same layer of the distillation branch and the decoding branch; using the elements in the covariance matrix as the weights corresponding to the features input to this layer of the distillation branch, determining the increment corresponding to the features input to this layer of the distillation branch; and adding the increment to the features input to this layer of the decoding branch to obtain the fused feature.

[0143] Optionally, the distillation branch processes the sample image from the feature dimension or the spatial dimension; or

[0144] The distillation branch includes two distillation branches. One distillation branch processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension; or

[0145] The distillation branch includes two groups of distillation branches. Each group of distillation branches includes two distillation branches. One distillation branch processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension, and the order of the dimensions for which the two groups of distillation branches process the sample image is different.

[0146] Optionally, the apparatus may further include the following modules:

[0147] An image obtaining module, configured to obtain an image to be processed;

[0148] An input module for inputting the image to be processed into the target image processing network to obtain an image processing result.

[0149] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For related parts, please refer to the method embodiments.

[0150] An embodiment of the present invention also provides an electronic device, which can be used to execute the method for constructing an image processing network. It may include a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor is configured to execute the method for constructing the image processing network.

[0151] An embodiment of the present invention also provides a computer-readable storage medium, and the computer program stored thereon causes the processor to execute the method for constructing an image processing network as described in the embodiment of the present invention.

[0152] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0153] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for realizing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0155] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0157] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

[0158] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or terminal device comprising the element.

[0159] The above has introduced in detail a method, apparatus, device and storage medium for constructing an image processing network provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for constructing an image processing network, characterized in that, Including: Obtain a network to be trained, where the network to be trained includes a backbone network branch and a distillation branch. The backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fused feature obtained after fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to this layer of the distillation branch; Train the network to be trained using a sample image; the distillation branch is used to process the sample image in terms of feature dimension or spatial dimension; When the training end condition is met, determine the backbone network branch in the network obtained at the end of training as the target image processing network; Wherein, the fused feature is obtained according to the following steps: Determine the corresponding covariance matrix according to the features input to the same layer of the distillation branch and the decoding branch; Determine the increment corresponding to the feature input to this layer of the distillation branch with the elements in the covariance matrix as the weights corresponding to the feature input to this layer of the distillation branch; On the basis of the feature input to this layer of the distillation branch, add the increment to obtain the fused feature.

2. The method according to claim 1, characterized in that Training the network to be trained using a sample image includes: Determine the overall loss value of the network to be trained according to the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image; Update each learnable parameter of the network to be trained according to the overall loss value of the network to be trained.

3. The method according to claim 2, characterized in that, Determining the overall loss value of the network to be trained according to the output result of the decoding branch, the output result of the distillation branch, and the label of the sample image includes: Obtain a first loss value according to the output result of the decoding branch and the label of the sample image, and obtain a second loss value according to the output result of the distillation branch and the label of the sample image; Obtain a first distillation loss value according to the difference between the output result of the distillation branch and the output result of the decoding branch; Determine the overall loss value of the network to be trained according to the first loss value, the second loss value, and the first distillation loss value.

4. The method according to claim 3, characterized in that The method further includes: Obtain the fused output result after fusing the output result of the decoding branch and the output result of the distillation branch; Determine a second distillation loss value according to the difference between the fused output result and the output result of the decoding branch; Determining the overall loss value of the network to be trained according to the first loss value, the second loss value, and the first distillation loss value includes: Determine the overall loss value of the network to be trained according to the first loss value, the second loss value, the first distillation loss value, and the second distillation loss value.

5. The method according to claim 4, wherein The decoding branch includes an original network layer and a convolutional layer connected in series with the original network layer. The learnable parameters of the convolutional layer include: the weight values corresponding to the decoding branch and the distillation branch respectively; The fused output result is obtained according to the following steps: According to the respective weight values corresponding to the decoding branch and the distillation branch, perform weighted summation on the output result of the decoding branch and the output result of the distillation branch to obtain the fusion output result.

6. The method according to claim 1, wherein The distillation branch processes the sample image from the feature dimension or the spatial dimension; or The distillation branch includes two distillation branches, one of the two distillation branches processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension; or The distillation branch includes two groups of distillation branches, each group of distillation branches includes two distillation branches, one of the distillation branches processes the sample image from the feature dimension, and the other distillation branch processes the sample image from the spatial dimension, and the order of the dimensions for which the two groups of distillation branches process the sample image is different.

7. The method according to claim 1, characterized in that After obtaining the target image processing network, the method further includes: Obtain the image to be processed; Input the image to be processed into the target image processing network to obtain an image segmentation result.

8. An apparatus for constructing an image processing network, characterized in that Including: A network obtaining module, configured to obtain a network to be trained, the network to be trained includes a backbone network branch and a distillation branch, the backbone network branch includes an encoding branch and a decoding branch connected in series with the encoding branch, and the distillation branch is connected in series with the encoding branch; wherein, each layer of the decoding branch is connected in series with each layer of the distillation branch, and the fusion feature obtained after fusing the features input to the same layer of the distillation branch and the decoding branch is used as the input to the corresponding layer of the distillation branch; A training module, configured to train the network to be trained using a sample image; the distillation branch is configured to process the sample image from the feature dimension or the spatial dimension; A determination module, configured to, when the training end condition is met, determine the backbone network branch in the network obtained at the end of training as the target image processing network; Wherein, the fusion feature is obtained according to the following steps: Determine the corresponding covariance matrix according to the features input to the same layer of the distillation branch and the decoding branch; Use the elements in the covariance matrix as the weights corresponding to the features input to the corresponding layer of the distillation branch to determine the increment corresponding to the features input to the corresponding layer of the distillation branch; On the basis of the features input to the corresponding layer of the distillation branch, add the increment to obtain the fusion feature.

9. An electronic device, characterized in that, Including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes, it implements the method for constructing an image processing network according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The stored computer program causes the processor to execute the method for constructing an image processing network according to any one of claims 1-7.

Images