Neural network training method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a neural network training method and device, which include: inputting multiple training samples into multiple sub-networks respectively, wherein each sub-network corresponds to a training sample, and each sub-network corresponds to a different training sample; performing a feature fusion operation on a first feature map output by a target network unit of a target sub-network and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit; obtaining a prediction result according to the image fusion result; determining a target loss according to the prediction result and supervision information of the last target sub-network, and updating multiple sub-networks based at least on the target loss.

Description

Technical Field

[0001] The present application relates to a field, and specifically to a neural network training method, device, electronic device and storage medium. Background Art

[0002] At present, in the framework for training multiple sub-networks simultaneously, such as the CBNet framework for training multiple backbone networks simultaneously, each time multiple sub-networks are trained, the same input is input into each sub-network respectively, and the outputs of related network units are fused to obtain corresponding fusion results. The corresponding fusion results are used to train multiple sub-networks so that each sub-network learns complementary feature information. Summary of the invention

[0003] The embodiments of the present application provide a neural network training method, device, electronic device and storage medium.

[0004] The present application provides a neural network training method, including:

[0005] Inputting multiple training samples into multiple sub-networks respectively, wherein each sub-network corresponds to one training sample, and the training samples corresponding to each sub-network are different;

[0006] Performing a feature fusion operation on a first feature map output by a target network unit of the target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit, a position of the associated network unit in the associated subnetwork of the target subnetwork is the same as a position of the target network unit in the target subnetwork, and the associated subnetwork of the target subnetwork is a previous subnetwork of the target subnetwork;

[0007] Obtaining a prediction result according to the image fusion result;

[0008] A target loss is determined according to the prediction result and the supervision information of the last target sub-network, and a plurality of sub-networks are updated based at least on the loss.

[0009] The present application provides a neural network training device, comprising:

[0010] A parallel input unit is configured to input a plurality of training samples into a plurality of sub-networks respectively, wherein each sub-network corresponds to a training sample, and the training samples corresponding to each sub-network are different;

[0011] a fusion unit configured to perform a feature fusion operation on a first feature map output by a target network unit of the target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit, a position of the associated network unit in an associated subnetwork of the target subnetwork is the same as a position of the target network unit in the target subnetwork, and the associated subnetwork of the target subnetwork is a previous subnetwork of the target subnetwork; and obtain a prediction result according to the image fusion result;

[0012] A prediction unit, configured to obtain a prediction result according to the image fusion result;

[0013] An updating unit is configured to determine a target loss according to the prediction result and the supervision information of the last target sub-network, and to update multiple sub-networks based at least on the target loss.

[0014] The neural network training method and device provided in the embodiments of the present application realize that when training multiple sub-networks, the input of each sub-network in the multiple sub-networks is input into each sub-network respectively. Compared with using the same input to input each sub-network respectively when training multiple sub-networks each time, multiple different inputs are input in parallel each time training multiple sub-networks, so that the output of the relevant network unit is subjected to a preset feature fusion operation to obtain the corresponding image fusion result including richer features, and accordingly, each sub-network learns more complementary feature information. After completing the training of multiple sub-networks, the performance of each sub-network is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] Figure 1 A flowchart of a neural network training method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing the effect of training multiple sub-networks;

[0018] Figure 3 A structural block diagram of a neural network training device provided in an embodiment of the present application is shown;

[0019] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] Figure 1 A flowchart of a neural network training method provided in an embodiment of the present application is shown, and the method includes:

[0023] Step 101 : Input of each sub-network in a plurality of sub-networks is input into each sub-network respectively.

[0024] In the embodiment of the present application, the network structures of the multiple sub-networks are the same, and the multiple sub-networks can be multiple backbone networks with the same structure. For example, the multiple sub-networks can be multiple backbone networks ResNeXt-152.

[0025] In the embodiment of the present application, each time multiple sub-networks are trained, steps 101-104 are performed on the multiple sub-networks. The operations in steps 101-104 are exemplary operations performed in the process of training multiple sub-networks at the same time.

[0026] By repeatedly executing steps 101 to 104, the multiple sub-networks are continuously trained until the training completion condition is met, thereby completing the training of the multiple sub-networks.

[0027] In an embodiment of the present application, the training completion condition may be that the number of training times is greater than a threshold indicating the completion of training, the training completion condition may be that the accuracy of each sub-network reaches an accuracy threshold, or the training completion condition may be that the accuracy of the last target sub-network reaches an accuracy threshold.

[0028] In practice, since the number of sub-networks involved in training is not proportional to the training efficiency and the accuracy of each sub-network after training, preferably, the number of the multiple sub-networks is 2 or 3.

[0029] In the embodiment of the present application, during the process of training multiple sub-networks at one time, the input of each of the multiple sub-networks is different.

[0030] In the embodiment of the present application, in the process of training multiple sub-networks at one time, each of the multiple sub-networks corresponds to a training image, i.e., a training sample, and each sub-network corresponds to a different training image. In the process of training multiple sub-networks at this time, the training image corresponding to each sub-network comes from the training image set used for training the multiple sub-networks at this time.

[0031] In the embodiment of the present application, in the process of training multiple sub-networks at one time, the input of each sub-network in the multiple sub-networks is first determined. The input of each sub-network in the multiple sub-networks is different, that is, the training samples input to each sub-network are different.

[0032] In the process of training multiple sub-networks at one time, for each of the multiple sub-networks, the training image corresponding to the sub-network in the process of training the multiple sub-networks at one time can be used as the input of the sub-network in the process of training the multiple sub-networks at one time.

[0033] In some embodiments, before inputting multiple training samples into multiple sub-networks respectively, it also includes: pre-training one of the multiple sub-networks using a pre-training image set; initializing the parameter values ​​of the parameters of each other sub-network except the pre-trained sub-network using the parameter values ​​of the parameters of the pre-trained sub-network after the pre-training of the pre-trained sub-network is completed.

[0034] In an embodiment of the present application, before training the multiple sub-networks for the first time, one of the multiple sub-networks may be pre-trained using a pre-training image set.

[0035] For example, the multiple sub-networks are multiple backbone networks ResNeXt-152 with the same structure. The training set ImageNet is used to pre-train one of the multiple backbone networks.

[0036] The structures of the multiple sub-networks are the same, and the multiple sub-networks have multiple parameters of the same type. For each parameter in any other sub-network except the pre-trained sub-network, the parameter value of the parameter is initialized to the parameter value of the parameter of the pre-trained sub-network of the same type as the parameter after the pre-training of the pre-trained sub-network is completed.

[0037] After initializing the parameter values ​​of the parameters of each other subnetwork except the pretrained subnetwork using the parameter values ​​of the parameters of the pretrained subnetwork after the pretraining of the pretrained subnetwork is completed, the same type of parameters in the multiple subnetworks have the same initial value. Then, the training of the multiple subnetworks can be started.

[0038] Step 102: Perform a feature fusion operation on a first feature map output by a target network unit of a target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result.

[0039] In the embodiment of the present application, the target subnetwork is a subnetwork other than the first subnetwork among the multiple subnetworks.

[0040] In the process of training multiple sub-networks at one time, for each sub-network, in the process of training the multiple sub-networks at one time, the output of the first network unit in the sub-network is obtained based on the input of the sub-network during the process of training the multiple sub-networks at one time.

[0041] In the process of training multiple sub-networks at one time, for each network unit except the first network unit in the first sub-network, in the process of training multiple sub-networks at one time, the input of the network unit is the output of the previous network unit of the network unit in the first sub-network.

[0042] In the process of training multiple sub-networks at one time, for each target sub-network in the multiple sub-networks, in the process of training the multiple sub-networks at one time, the input of each network unit in the target sub-network except the first network unit is generated by performing a parallel fusion operation on the corresponding feature map.

[0043] In the embodiment of the present application, for a network unit, the number of network units the network unit is refers to the number of network units the network unit is in the subnetwork to which the network unit belongs in the direction from input to output.

[0044] In the embodiment of the present application, each of the multiple sub-networks may be divided in advance in the same division manner, so that each sub-network has multiple network units.

[0045] Since the network structures of the multiple subnetworks are the same, for example, the multiple subnetworks may be multiple backbone networks ResNeXt-152. Therefore, each of the multiple subnetworks is divided in the same division manner, and accordingly, the network structures of the network units at the same position in the multiple subnetworks are the same.

[0046] For example, the multiple sub-networks are 2 sub-networks, and each sub-network is divided into 3 network units in the same division manner.

[0047] The structure of the first network unit in the first sub-network is the same as the structure of the first network unit in the second sub-network. The structure of the second network unit in the first sub-network is the same as the structure of the second network unit in the second sub-network. The structure of the third network unit in the first sub-network is the same as the structure of the third network unit in the second sub-network.

[0048] In this embodiment of the present application, the target network unit may be the last network unit in the target subnetwork.

[0049] In the embodiment of the present application, each target network unit may have an associated network unit.

[0050] In some embodiments, the position of the associated network unit of the target network unit in the associated subnetwork of the target subnetwork is the same as the position of the target network unit in the target subnetwork, and the associated subnetwork of the target subnetwork is the previous subnetwork of the target subnetwork.

[0051] For each target network unit, the associated network unit of the target network unit belongs to the associated subnetwork of the target subnetwork, the position of the associated network unit of the target network unit in the associated subnetwork of the target subnetwork is the same as the position of the target network unit in the target subnetwork, and the associated subnetwork of the target subnetwork is the previous subnetwork of the target subnetwork.

[0052] For each target network element, the structure of the target network element is the same as the structure of the associated network element of the target network element.

[0053] In the embodiment of the present application, the first feature map does not specifically refer to the feature map output by a certain network unit, and the second feature map does not specifically refer to the feature map output by a certain network unit or the fused feature map of a certain network unit. The first feature map and the second feature map are relative to the target network unit and the associated network unit of the target network unit. For a target network unit and the associated network unit of the target network unit, the feature map output by the target network unit can be referred to as the first feature map. The feature map output by the associated network unit of the target network unit or the fused feature map of the associated network unit of the target network unit can be referred to as the second feature map.

[0054] In an embodiment of the present application, the target network unit may be the last network unit in the target subnetwork. The output of each target subnetwork may be subjected to a preset feature fusion operation with the output of the first subnetwork to obtain a fused feature map, and the fused image map may be used as an image fusion result. Since the output of the target subnetwork is the output of the target network unit of the target subnetwork, the output of each target subnetwork is subjected to a preset feature fusion operation with the output of the first subnetwork, that is, the output of the target network unit of each target subnetwork is subjected to a preset feature fusion operation with the output of the first subnetwork.

[0055] In some embodiments, each network unit in the target subnetwork is a target network unit, and the image fusion result includes: a fused feature map of each target network unit in the last target subnetwork in multiple subnetworks; and a feature fusion operation is performed on a first feature map output by the target network unit of the target subnetwork and a second feature map corresponding to the first feature map, and the image fusion result is obtained, including: for each target network unit in the target subnetwork, a preset feature fusion operation is performed on the first feature map output by the target network unit and the second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit, wherein the fused feature map of the target network unit is used as the input of the next network unit of the target network unit.

[0056] In the embodiment of the present application, the target subnetwork is a subnetwork other than the first subnetwork among the multiple subnetworks. The last target subnetwork is the last subnetwork among the multiple networks.

[0057] In the embodiment of the present application, each network unit in the target subnetwork may be a target network unit. For example, the multiple subnetworks are 2 subnetworks. The second subnetwork is the target subnetwork. Each network unit in the second subnetwork is a target network unit.

[0058] In the process of training multiple sub-networks at one time, for each target sub-network in the multiple sub-networks, in the process of training the multiple sub-networks at one time, the input of each network unit in the target sub-network except the first network unit is generated by performing a parallel fusion operation on the corresponding feature map.

[0059] In an embodiment of the present application, during the process of training multiple sub-networks at one time, if the number of the multiple sub-networks is 2, for each target network unit in the second sub-network, a preset feature fusion operation is performed on the first feature map output by the target network unit and the second feature map corresponding to the first feature map output by the associated network unit of the target network unit.

[0060] In an embodiment of the present application, in the process of training multiple subnetworks at one time, if the number of the multiple subnetworks is greater than 2, for each target network unit in the second subnetwork, a preset feature fusion operation is performed on the first feature map output by the target network unit and the second feature map corresponding to the first feature map output by the associated network unit of the target network unit to obtain the fused feature map of the target network unit. For each target network unit in any subnetwork after the second subnetwork, a preset feature fusion operation is performed on the first feature map output by the target network unit and the fused feature map of the associated network unit of the target network unit corresponding to the first feature map to obtain the fused feature map of the target network unit.

[0061] The following example illustrates the process of performing training and parallel fusion operations on each target sub-network in multiple sub-networks:

[0062] Assume that the multiple subnetworks are 3 subnetworks. The second subnetwork and the third subnetwork are both target subnetworks, and the third subnetwork is the last target subnetwork. The first subnetwork is the previous subnetwork of the second subnetwork, and the second subnetwork is the previous subnetwork of the third subnetwork.

[0063] Assume that the first sub-network, the second sub-network, and the third sub-network each have three network units. The first network unit in the second sub-network, the second network unit in the second sub-network, and the third network unit in the second sub-network are all target network units. The first network unit in the third sub-network, the second network unit in the third sub-network, and the third network unit in the third sub-network are all target network units.

[0064] The associated network unit of the first network unit in the second sub-network is the first network unit in the first sub-network.

[0065] The associated network unit of the second network unit in the second sub-network is the second network unit in the first sub-network.

[0066] The associated network unit of the third network unit in the second sub-network is the third network unit in the first sub-network.

[0067] The associated network unit of the first network unit in the third sub-network is the first network unit in the second sub-network.

[0068] The associated network unit of the second network unit in the third sub-network is the second network unit in the second sub-network.

[0069] The associated network unit of the 3rd network unit in the 3rd sub-network is the 3rd network unit in the 2nd sub-network.

[0070] In the process of training multiple sub-networks at one time, a preset feature fusion operation is performed on the first feature map output by the first network unit in the second sub-network and the second feature map output by the first network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the first network unit in the second sub-network.

[0071] The fusion result of the first network unit in the second sub-network is used as the input of the second network unit in the second sub-network during the training of multiple sub-networks.

[0072] A preset feature fusion operation is performed on the feature map of the output of the second network unit in the second sub-network and the second feature map of the output of the second network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the second network unit in the second sub-network.

[0073] The fusion result of the second network unit in the second sub-network is used as the input of the third network unit in the second sub-network during the training of multiple sub-networks.

[0074] A preset feature fusion operation is performed on the feature map of the output of the third network unit in the second sub-network and the second feature map of the output of the third network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the third network unit in the second sub-network.

[0075] A preset feature fusion operation is performed on the first feature map outputted by the first network unit in the third sub-network and the fused feature map of the first network unit in the second sub-network corresponding to the first feature map to obtain the fused feature map of the first network unit in the third sub-network.

[0076] The fusion result of the first network unit in the third sub-network is used as the input of the second network unit in the third sub-network during the training of multiple sub-networks.

[0077] A preset feature fusion operation is performed on the first feature map outputted by the second network unit in the third sub-network and the fused feature map of the second network unit in the second sub-network corresponding to the first feature map to obtain the fused feature map of the second network unit in the third sub-network.

[0078] The fusion result of the second network unit in the third sub-network is used as the input of the third network unit in the third sub-network during the training of multiple sub-networks.

[0079] A preset feature fusion operation is performed on the first feature map outputted by the third network unit in the third sub-network and the fused feature map of the third network unit in the second sub-network corresponding to the first feature map to obtain the fused feature map of the third network unit in the third sub-network.

[0080] In the process of training multiple sub-networks at one time, for a target network unit and the associated network units of the target network unit, a preset feature fusion operation can be performed on each first feature map output by the target network unit and the second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit.

[0081] In an embodiment of the present application, the preset feature fusion operation may be to perform corresponding calculations on pixel values ​​of pixels at the same position in two feature maps.

[0082] For example, in the process of training multiple sub-networks at one time, the preset feature fusion operation performed can be one of the preset feature fusion operations such as calculating the average value of pixel values ​​of pixels in the same position in two feature maps, calculating the product of pixel values ​​of pixels in the same position in two feature maps, etc.

[0083] After performing a preset feature fusion operation on a first feature map and a second feature map corresponding to the first feature map, a fused feature map can be obtained.

[0084] For each pixel in the fused feature map, the pixel value of the pixel is a calculation result obtained by correspondingly calculating the pixel values ​​of two pixels at the same position in the first feature map and the second feature map corresponding to the first feature map.

[0085] In an embodiment of the present application, in the process of training multiple sub-networks at one time, the input of each sub-network may be a training image corresponding to each sub-network.

[0086] The supervision information of the last target sub-network may be the annotation information of the fused image obtained by fusing the inputs of the multiple sub-networks. The annotation information of the fused image obtained by fusing the inputs of the multiple sub-networks is obtained by annotating the fused image.

[0087] During the process of training the multiple sub-networks, when fusing the inputs of the multiple sub-networks, a fusion method adapted to the preset feature fusion operation performed during the process of training the multiple sub-networks can be used to fuse the inputs of the multiple sub-networks to obtain a fused image.

[0088] The type of calculation performed on the pixel values ​​of pixels at the same position in the performed preset feature fusion operation is the same as the type of calculation performed on the pixel values ​​of pixels at the same position in the input of multiple sub-networks in a fusion method adapted to the preset feature fusion operation.

[0089] For example, in the process of training multiple subnetworks, the preset feature fusion operation performed in the process of training multiple subnetworks is to calculate the average value of the pixel values ​​of the pixels at the same position in two feature maps. Accordingly, in the process of training multiple subnetworks, the fusion mode adapted to the preset feature fusion operation is: calculating the average value of the pixel values ​​of the pixels at the same position in the inputs of the multiple subnetworks. Accordingly, the pixel value of each pixel in the obtained fused image is the average value of the pixel values ​​of the pixels at the same position in the inputs of the multiple subnetworks.

[0090] In some embodiments, the preset feature fusion operation is a coupling operation, and the fused feature map of the target network unit is the coupled feature map of the target network unit; the preset feature fusion operation is performed on the first feature map output by the target network unit and the second feature map corresponding to the first feature map to obtain the fused feature map of the target network unit, including: for each pixel in the first feature map, the pixel value of the first feature map pixel is added to the pixel value of the corresponding pixel of the second feature map to obtain the coupled feature map of the target network unit.

[0091] In the process of training multiple sub-networks at one time, when the inputs of the multiple sub-networks are separate inputs, the preset feature fusion operation performed in the process of training the multiple sub-networks at this time may be a coupling operation. The coupling operation may also be called a point-by-point addition operation.

[0092] The separate input situation is that during the training of multiple sub-networks, for each sub-network, the input of the sub-network is a single image, and the input of each sub-network is not a fused image.

[0093] A pixel in the first feature map may be referred to as a first feature map pixel. A pixel in the second feature map may be referred to as a second feature map pixel. Each first feature map pixel corresponds to a second feature map pixel. For a first feature map pixel, a second feature map corresponding pixel added to a pixel value of the first feature map pixel may refer to a second feature map pixel in the second feature map corresponding to the first feature map pixel. For a first feature map pixel, a position of the first feature map pixel in the first feature map is the same as a position of the second feature map corresponding pixel in the second feature map.

[0094] For each first feature map pixel in the first feature map, a pixel value of the first feature map pixel is added to a pixel value of a corresponding pixel in the second feature map to obtain a coupled feature map.

[0095] In other words, the pixel values ​​of the pixels at the same position in the first feature map and the second feature map are added to obtain the coupled feature map. For each pixel in the coupled feature map, the pixel value of the pixel is the result of adding the pixel values ​​of the two pixels at the same position in the first feature map and the second feature map.

[0096] In the process of training multiple sub-networks at one time, the inputs of the multiple sub-networks are separate inputs, and the supervision information of the last target sub-network is the annotation information of the fused image obtained by fusing the inputs of each sub-network.

[0097] The inputs of multiple sub-networks can be fused in a fusion method adapted to the coupling operation to obtain a fused image. The fusion method adapted to the coupling operation is to fuse the inputs of multiple sub-networks as follows: the sum of the pixel values ​​of the pixels at the same position in the inputs of the multiple sub-networks is calculated. Accordingly, the pixel value of each pixel in the obtained fused image is the sum of the pixel values ​​of the pixels at the same position in the inputs of the multiple sub-networks.

[0098] For example, the multiple subnetworks are 2 subnetworks. In the process of training the multiple subnetworks once, the training image corresponding to the first subnetwork is training image A, and the training image corresponding to the second subnetwork is training image B. In the process of training the multiple subnetworks this time, the inputs of the multiple subnetworks are separate inputs, the input of the first subnetwork is training image A corresponding to the first subnetwork, and the input of the second subnetwork is training image B corresponding to the second subnetwork. The preset feature fusion operation performed in the process of training the multiple subnetworks this time is a coupling operation.

[0099] The second sub-network is the last target sub-network. The supervision information of the last target sub-network, i.e., the second sub-network, is the annotation information of the fused image obtained by fusing the input of the first sub-network, i.e., the training image A, with the input of the second sub-network, i.e., the training image B.

[0100] The fusion method for fusing the training image A and the training image B may be: adding pixel values ​​of pixels in the training image A and the training image B at the same position.

[0101] For each pixel in the fused image obtained by fusing the training image A with the training image B, the pixel value of the pixel is the sum of the pixel values ​​of the pixels in the training image A and the training image B at the same position.

[0102] In some embodiments, the preset feature fusion operation is a decoupling operation, and the fused feature map of the target network unit includes: a decoupled feature map of the target network unit; performing a preset feature fusion operation on a first feature map output of the target network unit and a second feature map corresponding to the first feature map includes: for each pixel in the first feature map, subtracting the pixel value of the first feature map pixel from the pixel value of the corresponding pixel of the second feature map to obtain the decoupled feature map of the target network unit.

[0103] In the process of training multiple sub-networks at one time, when the inputs of the multiple sub-networks are related inputs, the preset feature fusion operation performed in the process of training the multiple sub-networks at this time may be a decoupling operation. The decoupling operation may also be called a point-by-point reduction operation.

[0104] The associated input is: the input of the first sub-network is a single image, not a fused image. For each target sub-network, the input of the target sub-network is a fused image obtained by fusing the input-related image of the target sub-network with the input of the previous sub-network of the target sub-network, i.e., the training image corresponding to the previous sub-network. Each target sub-network has a different input-related image.

[0105] When the inputs of multiple sub-networks are associated inputs, the input of the target sub-network, that is, the training image corresponding to the target sub-network, is an image related to the input of the target sub-network and is fused with the input of the previous sub-network of the target sub-network. The fusion method can be: adding the pixel values ​​of the pixels at the same position in the image related to the input of the target sub-network and the input of the previous sub-network of the target sub-network.

[0106] When the inputs of multiple sub-networks are associated inputs, for each pixel in the input of the target sub-network obtained by fusing the image associated with the input of the target sub-network with the input of the previous sub-network of the target sub-network, the pixel value of the pixel is the sum of the pixel values ​​of the pixels in the same position in the image associated with the input of the target sub-network and the input of the previous sub-network of the target sub-network.

[0107] When the inputs of multiple sub-networks are associated inputs, when a decoupling operation is performed on the first feature map outputted by the target network unit and the second feature map corresponding to the first feature map, for each first feature map pixel in the first feature map, the pixel value of the first feature map pixel can be subtracted from the pixel value of the corresponding pixel of the second feature map to obtain the decoupled feature map of the target network unit.

[0108] When the inputs of multiple sub-networks are associated inputs, for each pixel in a decoupled feature map, the pixel value of the pixel is the result obtained by subtracting the pixel value of the pixel in the first feature map from the pixel value of the corresponding pixel in the second feature map.

[0109] In the process of training multiple sub-networks at one time, when the inputs of the multiple sub-networks are related inputs and the preset feature fusion operation performed is a decoupling operation, the supervision information of the last target sub-network is the annotation information of the image related to the input of the last target sub-network in the process of training the multiple sub-networks.

[0110] For example, the multiple subnetworks are 2 subnetworks. In the process of training multiple subnetworks at one time, the inputs of the multiple subnetworks are associated inputs, the training image corresponding to the first subnetwork, i.e., the training sample, is the training image A, and the training image corresponding to the second subnetwork is the fused image obtained by fusing the image ImageB associated with the input of the second subnetwork with the input of the first subnetwork, i.e., ImageA+B. The supervision information of the last target subnetwork is the annotation information of the image associated with the input of the last target subnetwork, i.e., the training image B, in the process of training multiple subnetworks at this time.

[0111] Step 103, obtaining a prediction result according to the image fusion result.

[0112] In an embodiment of the present application, a prediction result may be generated by fusion results of target detection network images.

[0113] The target detection network may be Mask R-CNN. An image fusion result, such as an image fusion result composed of fusion feature maps of each target network unit in the last target subnetwork among multiple subnetworks, may be used as an input of the target detection network. Mask R-CNN performs prediction based on the input of Mask R-CNN to generate a prediction result.

[0114] Step 104, determining a target loss according to the prediction result and the supervision information of the last target sub-network, and updating the multiple sub-networks based at least on the target loss.

[0115] In an embodiment of the present application, the target loss is the loss between the prediction result and the annotation information of the last neural network.

[0116] In the process of training multiple sub-networks at once, the target loss can be determined based on the prediction results and the supervision information of the last target sub-network.

[0117] Then, back propagation can be performed according to the target loss to update multiple sub-networks, that is, to update the parameter values ​​of the parameters of each sub-network.

[0118] In an embodiment of the present application, after completing the training of multiple subnetworks, the last target subnetwork, that is, the last subnetwork among the multiple subnetworks, can be directly integrated with the target detection network to obtain a target detector, and the target detector is used to perform a target detection task. When using the target detector to perform a target detection task, the last target subnetwork is used to receive an image that needs to be detected, extract features for target detection from the image that needs to be detected, and the output of the last target subnetwork is used as the output of the target detection network as the input of the target detection network, and the target detection network outputs a target detection result.

[0119] In some embodiments, after completing the training of multiple sub-networks, it also includes: integrating the last target sub-network with the target detection network to obtain a target detector; testing the target detector using a test image set to obtain an indicator of the target detector indicating the target detection accuracy of the target detector.

[0120] In an embodiment of the present application, after completing the training of multiple sub-networks, the last target sub-network can be used as the target backbone network, and the target backbone network can be integrated with the target detection network to obtain a target detector. Then, the obtained target detector can be tested using a test set to obtain a mAP indicator of the target detector indicating the target detection accuracy of the target detector.

[0121] Please refer to Figure 2 , which shows a schematic diagram of the effect of training multiple sub-networks.

[0122] Multiple subnets are 2 subnets. Figure 2 In , the subnet on the left is the first subnet, and the subnet on the right is the second subnet. Figure 2 In , “+” indicates a coupling operation, and “-” indicates a decoupling operation. The two subnetworks may be two backbone networks ResNeXt-152, and the target detection network used to generate the prediction results may be Mask R-CNN.

[0123] The separate input is for each sub-network, and the input of the sub-network is the training image corresponding to the sub-network.

[0124] When the inputs of the two sub-networks are separate inputs, the input of the first sub-network is ImageA, and the input of the second sub-network is ImageB.

[0125] The associated input is: the input of the first sub-network is a single image, not a fused image. For each target sub-network, the input of the target sub-network is a fused image obtained by fusing the input-related image of the target sub-network with the input of the previous sub-network of the target sub-network.

[0126] When the inputs of the two subnetworks are related inputs, the input of the first subnetwork is imageA, and the input of the second subnetwork is a fused image, i.e., ImageA+B, obtained by fusing the image ImageB related to the input of the second subnetwork with the input of the first subnetwork, i.e., ImageA.

[0127] The first sub-network and the second sub-network each have three network units. The second sub-network is the target sub-network, and the first network unit in the second sub-network, the second network unit in the second sub-network, and the third network unit in the second sub-network are all target network units.

[0128] The associated network unit of the first network unit 202 in the second sub-network is the first network unit 201 in the first sub-network. The associated network unit of the second network unit 204 in the second sub-network is the second network unit 203 in the first sub-network. The associated network unit of the third network unit 206 in the second sub-network is the third network unit 205 in the first sub-network.

[0129] In the process of training multiple sub-networks at one time, when the inputs of two sub-networks are separate inputs, the preset feature fusion operation performed in the process of training the multiple sub-networks at this time is a coupling operation.

[0130] A coupling operation is performed on the first feature map outputted by the first network unit in the second sub-network and the second feature map outputted by the first network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the first network unit in the second sub-network.

[0131] The fused feature map of the first network unit in the second sub-network is used as the input of the second network unit in the second sub-network during the training of multiple sub-networks.

[0132] A coupling operation is performed on the first feature map outputted by the second network unit in the second sub-network and the second feature map outputted by the second network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the second network unit in the second sub-network.

[0133] The fused feature map of the second network unit in the second sub-network is used as the input of the third network unit in the second sub-network during the training of multiple sub-networks.

[0134] A coupling operation is performed on the first feature map outputted by the third network unit in the second sub-network and the second feature map outputted by the third network unit in the first sub-network corresponding to the first feature map to obtain a fused feature map of the third network unit in the second sub-network.

[0135] The fused feature map of the first network unit in the second subnetwork, the fused feature map of the second network unit in the second subnetwork, and the fused feature map of the third network unit in the second subnetwork constitute the input of the target detection network Mask R-CNN.

[0136] Mask R-CNN generates prediction results based on the input of Mask R-CNN. When the inputs of the two sub-networks are separate inputs, the supervision information of the second sub-network is the annotation information of the fused image ImageA+B obtained by fusing the input of the first sub-network, ImageA, with the input of the second sub-network, ImageB.

[0137] The loss between the prediction result and the annotation information of ImageA+B can be calculated. Then, back propagation can be performed based on the loss to update the parameter values ​​of the parameters of the first sub-network and the parameter values ​​of the parameters of the second sub-network.

[0138] In the process of training multiple sub-networks at one time, when the inputs of two sub-networks are related inputs, the preset feature fusion operation performed in the process of training the multiple sub-networks at this time is a decoupling operation.

[0139] A decoupling operation is performed on a first feature map outputted by the first network unit in the second sub-network and a second feature map corresponding to the first feature map outputted by the first network unit in the first sub-network to obtain a fused feature map of the first network unit in the second sub-network.

[0140] A decoupling operation is performed on a first feature map outputted by the second network unit in the second sub-network and a second feature map outputted by the second network unit in the first sub-network and corresponding to the first feature map, so as to obtain a fused feature map of the second network unit in the second sub-network.

[0141] A decoupling operation is performed on the first feature map outputted by the third network unit in the second sub-network and the second feature map outputted by the third network unit in the first sub-network corresponding to the first feature map, so as to obtain a fused feature map of the third network unit in the second sub-network.

[0142] The fused feature map of the first network unit in the second subnetwork, the fused feature map of the second network unit in the second subnetwork, and the fused feature map of the third network unit in the second subnetwork constitute the input of the target detection network Mask R-CNN.

[0143] Mask R-CNN generates prediction results based on the input of Mask R-CNN. When the inputs of the two sub-networks are related inputs, that is, when the input of the first sub-network is imageA and the input of the second sub-network is imageA+B, the supervision information of the second sub-network is the annotation information of the image related to the input of the second sub-network, that is, ImageB.

[0144] The loss between the prediction result and the annotation information of Image B can be calculated. Then, back propagation can be performed based on the loss to update the parameter values ​​of the parameters of the first sub-network and the parameter values ​​of the parameters of the second sub-network.

[0145] In some embodiments, it also includes: calculating a loss function value based on a first value and a second value, wherein the first value is a value of a prediction result obtained based on the output of the last target sub-network for a non-fused image, and the second value is a value of a prediction result obtained based on the output of a previous sub-network for a non-fused image, the previous sub-network is located before the last target sub-network, and the value of the prediction result indicates the probability that the prediction result is a correct result; updating multiple sub-networks based at least on the target loss includes: updating multiple sub-networks based on the target loss and the loss function value, that is, updating the parameter values ​​of the parameters of each sub-network.

[0146] In an embodiment of the present application, the loss function value indicates whether the difference between the first value and the second value is large enough. When calculating the loss function value based on the first value and the second value, the loss function can be arranged at a preset interval to calculate the loss function value based on the first value and the second value.

[0147] In the process of training multiple sub-networks at one time, the non-fused image comes from the training image set used in the process of training multiple sub-networks at this time. A training image can be selected from the training image set used in the process of training multiple sub-networks at this time as the non-fused image.

[0148] For example, the multiple sub-networks are 2 sub-networks, and the training image set used in the process of training the multiple sub-networks at one time consists of training image A and training image B. The non-fused image may be training image A or training image B.

[0149] For each sub-network, the output of the sub-network for the non-fused image may refer to the output of the sub-network when the non-fused image is input to the sub-network.

[0150] For the same non-fused image, each of the multiple sub-networks has an output for the non-fused image.

[0151] In each process of training multiple sub-networks, the non-fused image can be input into each sub-network respectively to obtain the output of each sub-network for the non-fused image.

[0152] For each sub-network, the output of the sub-network for the non-fused image is used as the input of the target detection network to obtain the prediction result corresponding to the sub-network output by the target detection network and the numerical value of the prediction result corresponding to the sub-network. The prediction result corresponding to the sub-network is the prediction result obtained based on the output of the sub-network for the non-fused image.

[0153] In the embodiment of the present application, the interval arrangement loss function can be expressed as: loss(S1, S2)=max(0, S1-S2+δ)

[0154] Among them, S2 can represent the numerical value of the prediction result obtained based on the output of the last target sub-network for the non-fused image, the prediction result obtained based on the output of the last target sub-network for the non-fused image can be a classified logit vector of a non-fused image, and the numerical value of the prediction result obtained based on the output of the last target sub-network for the non-fused image can be the value of the classified logit vector of the non-fused image at the category position (ground-truth position) of the non-fused image. S1 can represent the numerical value of the prediction result obtained based on the output of the previous sub-network for the non-fused image, and similarly, the prediction result obtained based on the output of the previous sub-network for the non-fused image can also be a classified logit vector of a non-fused image.

[0155] In the case where the multiple subnetworks are 2 subnetworks, the previous subnetwork may refer to the first subnetwork. In the case where the multiple subnetworks are 2 subnetworks, the previous subnetwork may refer to the subnetwork used when obtaining the prediction result with the highest value among all other prediction results, and all other prediction results are composed of each prediction result except the prediction result obtained based on the output of the last target subnetwork for the non-fused image.

[0156] δ is a positive number, which indicates how much larger S2 must be than S1 to avoid loss. When S2 is not at least δ larger than S1, the loss function value is always positive, that is, it is in a penalty state.

[0157] In an embodiment of the present application, by arranging the loss function value calculated by the loss function to participate in the update of the parameter values ​​of the parameters of each sub-network, the performance of the last target sub-network can be promoted to be better than the performance of other sub-networks located before the last target sub-network in multiple sub-networks.

[0158] In each training process of multiple sub-networks, the parameter values ​​of the parameters of each sub-network are updated by combining the loss between the prediction result and the supervision information of the last target sub-network, i.e., the target loss, and the loss function value obtained by permuting the loss function. The performance of multiple sub-networks can be improved, and at the same time, the performance of each sub-network can be enhanced in a progressive manner. When the training of multiple sub-networks is completed, the performance of the last target sub-network is better than the performance of other sub-networks located before the last target sub-network in the multiple sub-networks.

[0159] Please refer to Figure 3 , which shows a structural block diagram of a neural network training device provided in an embodiment of the present application. The neural network training device comprises: a parallel input unit 301, a fusion unit 302, a prediction unit 303, and an update unit 304.

[0160] The parallel input unit 301 is configured to input the multiple training samples into the multiple sub-networks respectively, wherein each sub-network corresponds to a training sample, and the training samples corresponding to each sub-network are different;

[0161] The fusion unit 302 is configured to perform a feature fusion operation on a first feature map output by a target network unit of the target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit, a position of the associated network unit in the associated subnetwork of the target subnetwork is the same as a position of the target network unit in the target subnetwork, and the associated subnetwork of the target subnetwork is a previous subnetwork of the target subnetwork; and a prediction result is obtained according to the image fusion result;

[0162] The prediction unit 303 is configured to obtain a prediction result according to the image fusion result;

[0163] The updating unit 304 is configured to determine a target loss according to the prediction result and the supervision information of the last target sub-network, and to update multiple sub-networks based at least on the target loss.

[0164] In some embodiments, each network unit in the target subnetwork is a target network unit, and the image fusion result includes: a fused feature map of each target network unit in the last target subnetwork in the multiple subnetworks; the fusion unit 302 is further configured to perform a feature fusion operation on a first feature map output by the target network unit of the target subnetwork and a second feature map corresponding to the first feature map, and the image fusion result includes: for each target network unit in the target subnetwork, a preset feature fusion operation is performed on the first feature map output by the target network unit and the second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit, wherein the fused feature map of the target network unit is used as the input of the next network unit of the target network unit.

[0165] In some embodiments, the preset feature fusion operation is a coupling operation, and the fused feature map of the target network unit is a coupled feature map of the target network unit; the fusion unit 302 is further configured to add the pixel value of the pixel of the first feature map to the pixel value of the corresponding pixel of the second feature map for each pixel in the first feature map to obtain a coupled feature map.

[0166] In some embodiments, the preset feature fusion operation is a decoupling operation, and the fused feature map of the target network unit is a decoupled feature map of the target network unit; the fusion unit 302 is further configured to, for each pixel in the first feature map, subtract the pixel value of the pixel of the first feature map from the pixel value of the corresponding pixel of the second feature map to obtain a decoupled feature map.

[0167] In some embodiments, the neural network training apparatus further comprises:

[0168] The initialization unit is configured to, before inputting the input of each sub-network in the plurality of sub-networks into each sub-network respectively, the method further comprises:

[0169] A sub-network among the plurality of sub-networks is pre-trained using a pre-trained image set; and parameter values ​​of parameters of the pre-trained sub-network are initialized using parameter values ​​of the parameters of each other sub-network except the pre-trained sub-network after the pre-training of the pre-trained sub-network is completed.

[0170] In some embodiments, the neural network training device also includes: a loss function value calculation unit, configured to calculate the loss function value based on a first numerical value and a second numerical value, wherein the first numerical value is a numerical value of a prediction result obtained based on the output of the last target sub-network for a non-fused image, and the second numerical value is a numerical value of a prediction result obtained based on the output of a previous sub-network for a non-fused image, the previous sub-network is located before the last target sub-network, and the numerical value of the prediction result indicates the probability that the prediction result is a correct result; the update unit is further configured to update the multiple sub-networks based on the target loss and the loss function value.

[0171] In some embodiments, the neural network training apparatus further comprises:

[0172] The testing unit is configured to integrate the last target sub-network with the target detection network to obtain a target detector after completing the training of multiple sub-networks; test the target detector using a test image set to obtain an indicator of the target detector indicating the target detection accuracy of the target detector.

[0173] Figure 4 4 is a block diagram of a structure of an electronic device provided by this embodiment. The electronic device includes a processing component 422, which further includes one or more processors, and a memory resource represented by a memory 432, which is used to store instructions executable by the processing component 422, such as an application. The application stored in the memory 432 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 422 is configured to execute instructions to perform the above method.

[0174] The electronic device may also include a power supply component 426 configured to perform power management of the electronic device, a wired or wireless network interface 450 configured to connect the electronic device to a network, and an input / output (I / O) interface 458. The electronic device may operate based on an operating system stored in the memory 432, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0175] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0176] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0177] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A neural network training method, characterized in that: The method comprises: Inputting a plurality of training samples into a plurality of sub-networks respectively, wherein each sub-network corresponds to a training sample, and each sub-network corresponds to a different training sample; wherein the plurality of sub-networks are a plurality of backbone networks with the same structure, and the training samples are training images; Performing a feature fusion operation on a first feature map output by a target network unit of a target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit; wherein the target subnetwork is a subnetwork other than the first subnetwork among the multiple subnetworks, the position of the associated network unit in the associated subnetwork of the target subnetwork is the same as the position of the target network unit in the target subnetwork, the associated subnetwork of the target subnetwork is a previous subnetwork of the target subnetwork, and the image fusion result includes a fused feature map of each target network unit in the last target subnetwork among the multiple subnetworks; Obtaining a prediction result according to the image fusion result; Determine a target loss according to the prediction result and the supervision information of the last target sub-network, and update the multiple sub-networks based at least on the target loss.

2. The method according to claim 1, characterized in that Performing a feature fusion operation on a first feature map output by a target network unit of the target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result includes: For each target network unit in the target subnetwork, a preset feature fusion operation is performed on a first feature map output by the target network unit and a second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit, wherein the fused feature map serves as an input of a subsequent network unit of the target network unit.

3. The method according to claim 2, characterized in that The preset feature fusion operation is a coupling operation, and the fusion feature graph of the target network unit is a coupling feature graph of the target network unit; Performing a preset feature fusion operation on a first feature map output by the target network unit and a second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit includes: For each pixel in the first feature map, a pixel value of the pixel in the first feature map is added to a pixel value of a corresponding pixel in the second feature map to obtain the coupled feature map.

4. The method according to claim 2, characterized in that: The preset feature fusion operation is a decoupling operation, and the fusion feature map of the target network unit is a decoupled feature map of the target network unit; Performing a preset feature fusion operation on a first feature map output by the target network unit and a second feature map corresponding to the first feature map to obtain a fused feature map of the target network unit includes: For each pixel in the first feature map, a pixel value of the pixel in the first feature map is subtracted from a pixel value of a corresponding pixel in the second feature map to obtain the decoupled feature map.

5. The method according to claim 1, characterized in that Before inputting the plurality of training samples into the plurality of sub-networks respectively, the method further comprises: A sub-network among the plurality of sub-networks is pre-trained using a pre-trained image set; and parameter values ​​of parameters of the pre-trained sub-network are initialized using parameter values ​​of the parameters of each other sub-network except the pre-trained sub-network after the pre-training of the pre-trained sub-network is completed.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Calculating a loss function value based on a first value and a second value, wherein the first value is a prediction result obtained based on an output of a last target sub-network for a non-fused image, and the second value is a prediction result obtained based on an output of a previous sub-network for the non-fused image, the previous sub-network being located before the last target sub-network; Based at least on the target loss, updating the plurality of sub-networks comprises: Based on the target loss and the loss function value, parameters of the multiple sub-networks are updated.

7. The method according to claim 6, characterized in that After completing the training of the multiple sub-networks, the method further includes: Integrate the last target sub-network with the target detection network to obtain the target detector; The object detector is tested using the test image set to obtain an indicator of the object detector indicating the object detection accuracy of the object detector.

8. A sub-network training device, characterized in that: The device comprises: A parallel input unit is configured to input a plurality of training samples into a plurality of sub-networks respectively, wherein each sub-network corresponds to a training sample, and each sub-network corresponds to a different training sample, and the training sample is a training image; A fusion unit is configured to perform a feature fusion operation on a first feature map output by a target network unit of a target subnetwork and a second feature map corresponding to the first feature map to obtain an image fusion result, wherein the second feature map is a feature map output by an associated network unit of the target network unit or a fused feature map of an associated network unit of the target network unit; wherein the target subnetwork is a subnetwork other than the first subnetwork among the multiple subnetworks, the position of the associated network unit in the associated subnetwork of the target subnetwork is the same as the position of the target network unit in the target subnetwork, the associated subnetwork of the target subnetwork is a previous subnetwork of the target subnetwork, and the image fusion result includes a fused feature map of each target network unit in the last target subnetwork among the multiple subnetworks; A prediction unit, configured to obtain a prediction result according to the image fusion result; An updating unit is configured to determine a target loss according to the prediction result and the supervision information of the last target sub-network, and to update each sub-network at least based on the target loss.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7. 10 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to claim 1 .

Images