Method and device for neural network training, target detection, equipment and storage medium

Abstract

The present disclosure provides a method and device for neural network training and target detection, equipment and a storage medium, wherein the method comprises: obtaining first image samples collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image samples; the second neural network is used for feature extraction, the image generation network is used for generating new images, and the new images conform to the overall distribution of the first image samples; the new images generated based on the image generation network are subjected to feature extraction according to the second neural network and the first neural network to be trained respectively; and the first neural network to be trained is trained based on the extracted first image features and second image features, so as to obtain the trained first neural network. The present disclosure can better guide the training of the first neural network through two image features, thereby improving the performance in the downstream task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a neural network training method and device, a target detection method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, end-to-end deep learning technology is also becoming mature. A pre-trained neural network (i.e., a pre-trained model) can be learned for various tasks together in the upstream using a large-scale data set, and the model can directly share pre-trained early weights and has a powerful feature representation capability.

[0003] However, in the process of migrating the pre-trained model to the downstream specific task, since the amount of data that can be actually obtained in the downstream is relatively small, the performance of the pre-trained model in the downstream task is not good no matter whether the model is directly migrated or the pre-trained model is fine-tuned and then migrated. SUMMARY

[0004] The embodiments of the present disclosure at least provide a neural network training method and device, a target detection method and device, equipment and a storage medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a neural network training method, which comprises:

[0006] obtaining a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the second neural network is used for feature extraction, the image generation network is used for generating a new image, and the new image conforms to the overall distribution of the first image sample;

[0007] respectively extracting features from the new image generated based on the image generation network according to the second neural network and the first neural network to be trained, to obtain first image features and second image features;

[0008] training the first neural network to be trained based on the first image features and the second image features, to obtain a trained first neural network.

[0009] By using the method for training the neural network, the new image generated based on the image generation network can be subjected to feature extraction according to the first neural network to be trained in the downstream task and the second neural network trained based on the first image sample collected in the upstream task, and then the first neural network can be trained according to the obtained first image feature and second image feature. Since the new image generated based on the image generation network is an image sample more conforming to the overall distribution of the first image sample, such an image sample is more conducive to adapting to the network environment of the second neural network, and meanwhile, the first image feature output by the trained second neural network and the second image feature output by the first neural network to be trained can better guide the training of the first neural network, thereby further improving the performance in the downstream task.

[0010] In a possible implementation, the image generation network is trained according to the following steps:

[0011] obtaining a first image output based on a codebook generation network; the codebook generation network is configured to generate a codebook for decomposing the first image sample into a plurality of primitives;

[0012] inputting the first image into the image generation network to be trained to obtain a second image output by the image generation network;

[0013] determining a loss function value of the image generation network to be trained based on an image similarity between the second image and the first image;

[0014] training the image generation network to be trained based on the loss function value to obtain a trained image generation network.

[0015] Here, the first image output by the codebook encoding manner can be used to train the image generation network to be trained, so that the trained image generation network is more consistent with the overall distribution of the first image sample, thereby being conducive to improving the subsequent network training performance.

[0016] In a possible implementation, the inputting the first image into the image generation network to be trained comprises:

[0017] performing a covering process on part of the image region in the first image to obtain a first image subjected to the covering process;

[0018] inputting the first image subjected to the covering process into the image generation network to be trained.

[0019] In a possible implementation, the codebook generation network comprises an encoder and a decoder, and the codebook generation network is trained according to the following steps:

[0020] The following steps are repeatedly performed until the similarity between the image output by the decoder and the first image sample input into the encoder is greater than a preset threshold value:

[0021] The first image sample is input into an encoder to be trained to obtain a codebook output by the encoder, and the codebook output by the encoder is input into a decoder to be trained to obtain an image output by the decoder.

[0022] The codebook here can be image encoding implemented based on the adversarial network composed of the encoder and the decoder, and the accuracy is high.

[0023] In a possible implementation, the first image output based on the codebook generation network is obtained according to the following steps:

[0024] The first image sample is input into an encoder included in the codebook generation network to obtain a codebook output by the encoder.

[0025] The codebook output by the encoder is input into a decoder included in the codebook generation network to obtain the first image output by the decoder.

[0026] Here, the first image sample can be re-represented by using the codebook output by the encoder, and the represented first image can be more suitable for subsequent network training.

[0027] In a possible implementation, the image generation network includes a first generation sub-network for generating a codebook for decomposing the first image sample into a plurality of primitives, and a second generation sub-network for generating the new image based on the image output by the first generation sub-network; and the image generation network is trained according to the following steps:

[0028] The first image sample is input into the trained first generation sub-network to obtain a first image output by the first generation sub-network.

[0029] The first image is input into the second generation sub-network to be trained to obtain a second image output by the second generation sub-network.

[0030] Based on the first image similarity between the first image and the input first image sample and the second image similarity between the second image and the first image, a loss function value of the image generation network to be trained is determined.

[0031] The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network.

[0032] Here, the training of the image generation network can be performed in combination with the first generation sub-network and the second generation sub-network, so that the trained image generation network can better balance the generation effect and generation efficiency of the codebook generation and the image generation.

[0033] In a possible implementation, the training of the first neural network based on the first image feature and the second image feature to obtain the trained first neural network comprises:

[0034] determining a loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature;

[0035] In a case where the loss function value corresponding to the current round is greater than a preset threshold, adjusting a network parameter value of the first neural network based on the loss function value, and performing the next round of training according to the adjusted first neural network until the loss function value is less than or equal to the preset threshold.

[0036] In a possible implementation, after the trained first neural network is obtained, the method further comprises:

[0037] obtaining a third image sample collected in a downstream task;

[0038] re-training the trained first neural network based on the third image sample to obtain a final trained first neural network.

[0039] Here, the first neural network can be fine-tuned based on the third image sample collected in the downstream task, and the generalization performance of the network in the downstream task is expanded.

[0040] In a possible implementation, the re-training of the trained first neural network based on the third image sample to obtain the final trained first neural network comprises:

[0041] inputting the third image sample into the first neural network to obtain a task output result of the network;

[0042] determining a loss function value of the first neural network based on a comparison relationship between the task output result and a task annotation result annotated for the third image sample;

[0043] re-training the first neural network based on the loss function value to obtain the final trained first neural network.

[0044] In a possible implementation, the second neural network is trained according to the following steps:

[0045] obtain an original neural network; the original neural network at least comprises a feature extraction layer;

[0046] perform feature extraction on the first image sample based on the feature extraction layer comprised in the original neural network, to obtain image feature information output by the feature extraction layer;

[0047] adjust network parameter values of the feature extraction layer based on the image feature information, to obtain an adjusted feature extraction layer;

[0048] determine the original neural network comprising the adjusted feature extraction layer as a second neural network trained.

[0049] Here, the second neural network can be obtained based on training of the original neural network comprising the feature extraction layer, and the network can output more general feature information, facilitating subsequent task migration.

[0050] In a second aspect, the embodiments of the present disclosure further provide a method for target detection, and the method comprises:

[0051] obtain a target image collected in a downstream task;

[0052] input the target image into a first neural network trained by the method for neural network training according to any one of the first aspect and various embodiments thereof, to obtain a detection result of a target object in the target image.

[0053] In a third aspect, the embodiments of the present disclosure further provide a device for neural network training, and the device comprises:

[0054] an obtaining module, configured to obtain a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the second neural network is used for feature extraction, the image generation network is used for generating a new image, and the new image conforms to an overall distribution of the first image sample;

[0055] an extracting module, configured to perform feature extraction on a new image generated based on the image generation network according to the second neural network and the first neural network to be trained respectively, to obtain first image features and second image features;

[0056] a training module, configured to train the first neural network to be trained based on the first image features and the second image features, to obtain a trained first neural network.

[0057] In a fourth aspect, the embodiments of the present disclosure further provide a device for target detection, and the device comprises:

[0058] An acquisition module is configured to acquire a target image collected in a downstream task;

[0059] A detection module is configured to input the target image into a first neural network trained by the method for training a neural network according to any one of the first aspect and the various embodiments thereof, to obtain a detection result of a target object in the target image.

[0060] In a fifth aspect, the embodiments of the present disclosure further provide an electronic device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the method for training a neural network according to any one of the first aspect and the various embodiments thereof or the steps of the method for target detection according to the second aspect.

[0061] In a sixth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the steps of the method for training a neural network according to any one of the first aspect and the various embodiments thereof or the steps of the method for target detection according to the second aspect are performed.

[0062] The effects of the above device, electronic device and computer readable storage medium are described in the description of the above method, and will not be repeated here.

[0063] In order to make the above objectives, characteristics and advantages of the present disclosure more apparent and understandable, the following preferred embodiments are specifically described below, and the accompanying drawings are described briefly, and form a part of the present disclosure, and the drawings show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments, the drawings herein are incorporated into the description and form a part of the present disclosure, and the drawings show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 A flow chart of a method for training a neural network provided by the embodiments of the present disclosure is shown;

[0066] Figure 2 A flow chart of a method for target detection provided by the embodiments of the present disclosure is shown;

[0067] Figure 3A schematic diagram of an apparatus for neural network training is shown.

[0068] Figure 4 A schematic diagram of an apparatus for target detection is shown.

[0069] Figure 5 A schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0070] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to clearly and completely describe the technical solutions of the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0071] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0072] The term "and / or" herein is only used to describe an association relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0073] It is found through research that in the process of migrating a pre-trained model to a downstream specific task, the performance in the downstream task is usually improved through model fine-tuning in the related technology.

[0074] The existing fine-tuning methods mainly include the following two types: the first type can be to screen the features extracted by the mapping backbone network. In specific applications, the above screening process can be implemented by adding additional network layers after the backbone network, that is, the general features extracted by the backbone network can be screened and mapped through the additional network layers, and the features required by the downstream task are retained and strengthened. Here, the additional network layers can be convolutional layers, normalization layers, etc. The second type can be to operate the weight parameters of the backbone network. In specific applications, instead of directly using back propagation for downstream task migration, the weight increment and offset value are predicted in the specified weight parameter space for the downstream task to help the backbone network adapt to the downstream task.

[0075] However, the above two methods have the following disadvantages: the first method may cause overfitting of the feature mapping layer in the case of small amount of data in the downstream task; the weight update range of the second method is limited to the specified weight parameter space, which cannot guarantee the optimization of the weights to the best state. Therefore, the performance of model migration needs to be improved.

[0076] In addition, related pre-trained models often have a specific network structure, and in the actual migrated downstream scene, it may be necessary to migrate to different network structures, which puts higher requirements on the performance of model migration.

[0077] Based on the above research, the present disclosure provides a scheme for implementing network migration based on a teaching and training method for guiding a student network to be trained based on a trained teacher network, so as to improve the performance of the pre-trained model in the downstream task.

[0078] In order to facilitate the understanding of the present embodiment, first, a neural network training method disclosed by the present embodiment is described in detail. The execution subject of the neural network training method provided by the present embodiment is generally an electronic device with certain computing power, which includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, etc. In some possible implementation manners, the neural network training method can be implemented by a processor calling computer readable instructions stored in a memory.

[0079] Referring to Figure 1 The flowchart of the neural network training method provided by the present embodiment is shown. The method includes steps S101-S103, wherein:

[0080] S101: acquire a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the second neural network is used for feature extraction, and the image generation network is used for generating a new image, and the new image conforms to the overall distribution of the first image sample;

[0081] S102: respectively according to the second neural network trained and the first neural network to be trained, performing feature extraction on the new image generated based on the image generation network to obtain first image features and second image features;

[0082] S103: training the first neural network to be trained based on the first image features and the second image features to obtain a trained first neural network.

[0083] In order to facilitate understanding of the method of training the neural network provided by the embodiments of the present disclosure, next, first, the application scenario of the method is simply explained. The method of training the neural network in the embodiments of the present disclosure can be mainly applied to the network training in the downstream task under the visual scene migration. The downstream task here can be a related task based on the current migrated scene, for example, it can be a target detection task in a natural scene, and it can also be a semantic segmentation task in a collection scene.

[0084] Among them, the number of training samples that can be collected in the downstream task is relatively small. Corresponding to the downstream task is the upstream task, which can be a related task with more training samples. Taking a target classification task as an example, a target classification neural network trained by a training database composed of various target objects is currently available. However, for the specific application scene of automatic driving, the training data corresponding to this scene is relatively small, so it is often necessary to use the pre-trained model obtained from the upstream to support the training of the downstream, for example, the pre-trained model can be fine-tuned before migration.

[0085] However, due to a series of problems in the fine-tuning scheme in the related art, the performance of the pre-trained model in the downstream task is not good. At the same time, in the migration process, there is a restriction in the related art that the upstream pre-trained model must use the same model structure when migrating to the downstream data set, which leads to the pre-trained model cannot be well migrated to the downstream task with different model structures.

[0086] It is precisely to solve the above problems that the embodiments of the present disclosure provide a scheme of network migration based on the training of the teacher network to guide the student network to be trained, so as to improve the performance of the pre-trained model in the downstream task.

[0087] In the embodiments of the present disclosure, the pre-trained model herein can be a second neural network trained using first image samples collected in an upstream task. In addition, the image generation network herein can be a related network that generates new images conforming to the overall distribution of the first image samples.

[0088] In a specific application, an upstream data set for the upstream task and a downstream data set for the downstream task can be prepared in advance, the upstream data set serving as a large-scale pre-training data set and having a large number of first image samples, and the downstream data set serving as a data set to be migrated and having a small number of second image samples.

[0089] The first image samples can be images collected in multiple tasks in multiple application scenarios, the application scenarios herein can be natural scenes, monitoring scenes, collection scenes, and the like, and the tasks herein can be image classification, target detection, semantic segmentation, and the like. The second image samples can be images collected in a specific scene and a specific task to be migrated, such as street pedestrian images in a detection task.

[0090] Based on the first image samples, an original neural network including a feature extraction layer can be trained. Herein, the first image samples can be subjected to feature extraction based on the feature extraction layer, and the network parameter values of the feature extraction layer can be adjusted based on the image feature information output by the feature extraction layer. The original neural network trained in this way can be determined as the second neural network trained as described above.

[0091] The original neural network described above is any network structure having a feature providing function. The second neural network obtained by training the original neural network using large-scale upstream data (corresponding to the first image samples) can output a general feature representation for any image based on the backbone network part (corresponding to the feature extraction layer).

[0092] It should be noted that the original neural network described above can further include a task layer for processing a task after the feature extraction layer. In this case, the training of the entire original neural network can be performed based on the matching degree between the task output result of the task layer and the task annotation result for the large-scale upstream data. Details are not described herein.

[0093] In order to better train the second neural network in the downstream task, the trained second neural network described above can be used as a teacher model in knowledge distillation, and the first neural network to be migrated in the downstream can be used as a student model in knowledge distillation. The first neural network is trained in a manner that the teacher model is fixed and the student model is trained.

[0094] During the training, the second neural network and the first neural network can be used for feature extraction respectively, and then the training of the first neural network is guided based on the similarity between the first image features output by the second neural network as the teacher model and the second image features output by the first neural network as the student model, so that the representation output by the student model is as close as possible to the teacher model, and thus can better adapt to the downstream task. In the case where the representations of the two are similar, even if the same network structure is not used for the downstream task and the upstream task, the task index can also be well completed.

[0095] In actual application, the first image sample is derived from the images in the large-scale pre-training data set, which makes different first image samples possibly collected based on different application scenarios. The characteristics of the first image samples collected in different application scenarios may have certain differences, which may to some extent interfere with network training. Here, in order to fully tap the feature information contained in the upstream data set and reduce the interference of irrelevant information, a new image conforming to the overall distribution of the first image sample can be generated by using the image generation network, and then the training of the teacher-student model described above is performed based on the generated new image, so that the pre-training model to which the second neural network belongs can be efficiently and accurately migrated to the specific task field of the downstream, and even in the case of less downstream data, it also has good migration effect.

[0096] Considering the key role of the training process of the image generation network in model migration, the scheme for training the image generation network will be mainly explained next. The image generation network in the embodiments of the present disclosure can be trained under the premise of the codebook generation network, or can be trained synchronously with the codebook generation network. Specifically, it can be developed in the following two aspects.

[0097] First aspect: the image generation network in the embodiments of the present disclosure can be trained according to the following steps:

[0098] Step one, obtaining a first image output based on a codebook generation network; the codebook generation network is used for generating a codebook for decomposing the first image sample into a plurality of base elements;

[0099] Step two, inputting the first image into the image generation network to be trained to obtain a second image output by the image generation network;

[0100] Step three, determining a loss function value of the image generation network to be trained based on the image similarity between the second image and the first image;

[0101] Step four, training the image generation network to be trained based on the loss function value to obtain a trained image generation network.

[0102] Here, the first image output by the codebook generation network can be taken as the input image of the image generation network to be trained, and the loss function value of the image generation network is determined based on the image similarity between the second image output by the image generation network and the first image. The greater the loss function value, the greater the gap between the output second image and the input first image to some extent, and the network training needs to be performed again. The smaller the loss function value, the smaller the gap between the output second image and the input first image to some extent. When the gap is small to a certain extent, it can be determined that the output image is basically consistent with the input image, and the training can be ended at this time.

[0103] In order to better train the image generation network, part of the image region in the first image can be covered before the first image is input to the image generation network to be trained, to obtain a first image after covering processing. In the case of inputting the first image after covering processing to the image generation network to be trained, the uncovered part of the image region can guide the generation of the covered part of the image region, and then the network training can be realized based on the closeness between the generated image and the original first image.

[0104] In the embodiment of the present disclosure, the codebook generation network described above can also be trained based on the first image sample. The codebook generation network here is mainly to train a codebook that can encode the visual features in the upstream data, and then the image restoration can be performed through the plurality of primitives contained in the codebook generated by the codebook generation network, and then the first image output by the codebook generation network is obtained.

[0105] Next, the training process and application process of the codebook generation network will be described in detail.

[0106] In the embodiment of the present disclosure, the training of the codebook generation network can be performed by using the adversarial network composed of the paired encoder and decoder. Here, the first image sample can be input to the encoder to be trained to obtain the codebook output by the encoder; the codebook output by the encoder is input to the decoder to be trained to obtain the image output by the decoder, and then it is verified whether the similarity between the image output by the decoder and the first image sample input to the encoder is greater than a preset threshold. If it is not greater than the preset threshold, the above process of inputting the first image sample to the encoder to be trained is repeated until the similarity between the two images is greater than the preset threshold.

[0107] Here, the trained codebook generation network can make an image decomposed into a codebook composed of a plurality of primitives by the encoder, and the primitives can be restored to the image by the decoder.

[0108] Here, the first image sample can be input to the encoder included in the codebook generation network to obtain the codebook output by the encoder, and in the case of inputting the codebook output by the encoder to the decoder included in the codebook generation network, the image can be restored using each base element contained in the codebook, and then the first image after re-representation is obtained.

[0109] It should be noted that in actual application, the determination process of the first image can be determined by the training process of the joint codebook generation network, that is, the steps of inputting the first image sample to the encoder to be trained to obtain the codebook output by the encoder, and inputting the codebook output by the encoder to the decoder to be trained to obtain the image output by the decoder can be repeatedly executed until the similarity between the image output by the decoder and the first image sample input to the encoder is greater than a preset threshold, and the image output by the decoder is determined as the first image.

[0110] Second aspect: in the case that the image generation network includes a first generation subnetwork for generating a codebook for decomposing a first image sample into a plurality of base elements, and a second generation subnetwork for generating a new image based on the image output by the first generation subnetwork, the embodiments of the present disclosure can train the image generation network according to the following steps:

[0111] Step one, input the first image sample to the trained first generation subnetwork to obtain the first image output by the first generation subnetwork;

[0112] Step two, input the first image to the second generation subnetwork to be trained to obtain the second image output by the second generation subnetwork;

[0113] Step three, determine the loss function value of the image generation network to be trained based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image;

[0114] Step four, train the image generation network to be trained based on the loss function value to obtain the trained image generation network.

[0115] Here, the loss function value of the image generation network to be trained can be determined jointly based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image. No matter the first image similarity or the second image similarity will affect the adjustment of the related network parameter value, that is, the synchronous training of the first generation subnetwork and the second generation subnetwork is realized here, and the training efficiency is higher.

[0116] In the case of training to obtain the first generation sub-network and the second generation sub-network, a corresponding new image can be generated for any input first image sample, and the new image contains rich data information of the upstream task, thereby being more suitable for the training needs of the network.

[0117] In the case of inputting a large number of new images generated by the image-based generation network into the second neural network (i.e., a pre-training model) trained in the upstream task, a general representation can be obtained, and the knowledge of the general representation can be distilled into the first neural network to be migrated in the downstream task through knowledge distillation.

[0118] In the embodiments of the present disclosure, the training of the first neural network in the downstream task can be guided based on the image similarity between the two image features (i.e., the first image feature and the second image feature) extracted by the trained second neural network and the first neural network to be trained, and the training can be implemented through the following steps:

[0119] Step one, determining the loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature;

[0120] Step two, in the case that the loss function value corresponding to the current round is greater than a preset threshold, adjusting the network parameter value of the first neural network based on the loss function value, and performing the next round of training according to the adjusted first neural network until the loss function value is less than or equal to the preset threshold.

[0121] Here, the image similarity between the two image features is negatively correlated with the loss function value of the first neural network, that is, in the case of a smaller image similarity, the determined loss function value is larger, and in the case of a larger image similarity, the determined loss function value is smaller. The purpose of training the first neural network in the embodiments of the present disclosure is to make the representations output by the two neural networks (the second neural network and the first neural network) as similar as possible.

[0122] In order to further expand the generalization performance of the first neural network in the downstream task field, here, the first neural network can be fine-tuned using the second image samples collected in the downstream task, and the fine-tuning can be implemented through the following steps:

[0123] Step one, inputting the second image sample into the first neural network to obtain the task output result of the network;

[0124] Step two, determining the loss function value of the first neural network based on the comparison relationship between the task output result and the task annotation result annotated for the second image sample;

[0125] Step three, re-training the first neural network based on the loss function value, to obtain the final trained first neural network.

[0126] Here, feature extraction can be performed through the feature extraction layer included in the first neural network, and in the case that the feature information output by the feature extraction layer is input to the task layer included in the first neural network, multiple rounds of training of the first neural network can be performed based on the matching result of the task output result and the task annotation result for the second image sample.

[0127] In the embodiments of the present disclosure, in the case that the task output result and the task annotation result do not match, it is indicated that the current network performance is poor, and the adjustment of the network parameter value is needed for the next round of training until the two results match or until other network convergence conditions are met, for example, the iteration round reaches a preset number, and again, the loss function value is less than a preset threshold.

[0128] The task annotation result here is also different for different downstream tasks. For example, some image samples can be labeled with information such as the position and size of the target object for a target detection task, and some image samples can be labeled with object semantic information for a target semantic segmentation task. Here, annotation can be performed for different downstream tasks, and no specific limitation is made thereto.

[0129] In the process of fine-tuning the network based on the second image sample, the embodiments of the present disclosure can adjust the entire process of each network layer included in the network. Here, all parameters of each network layer can be released, a small learning rate is used to perform the final adjustment of the network, so that the generalization performance of the network in the downstream task field can be significantly improved.

[0130] Based on the above-mentioned neural network training method provided by the embodiments of the present disclosure, the embodiments of the present disclosure also provide a target detection method, as shown in Figure 2 The specific steps include the following steps:

[0131] S201: acquiring a target image collected in a downstream task;

[0132] S202: inputting the target image into the first neural network trained by the method of training the neural network to obtain a detection result of the target object in the target image.

[0133] Here, in the case that the target image collected in the downstream task is acquired, the target object in the target image can be detected based on the first neural network trained for target detection to obtain a detection result of the target object in the target image.

[0134] The detection result of the target object in the target image can be information such as the position and size of the target object in the target image.

[0135] In the embodiments of the present disclosure, the target images collected by different downstream tasks are also different, and details can be referred to the collection process of the second image sample, which will not be repeated here. The training process of the first neural network is described in the above embodiments, which will not be repeated here.

[0136] It should be noted that the method for training a neural network provided in the embodiments of the present disclosure can not only be applied to the field of target detection, but also be applied to the fields of image classification, semantic segmentation, etc., which will not be repeated here.

[0137] Those skilled in the art can understand that the writing order of each step in the above method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0138] Based on the same inventive concept, the embodiments of the present disclosure also provide a device corresponding to the method. Since the principle of solving problems in the device of the embodiments of the present disclosure is similar to the above-mentioned method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be repeated.

[0139] Referring to Figure 3 Fig. 1 is a schematic diagram of a device for training a neural network provided in the embodiments of the present disclosure. The device comprises an acquisition module 301, an extraction module 302 and a training module 303; wherein,

[0140] The acquisition module 301 is configured to acquire a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample. The second neural network is configured to perform feature extraction, and the image generation network is configured to generate a new image, and the new image conforms to the overall distribution of the first image sample.

[0141] The extraction module 302 is configured to perform feature extraction on the new image generated based on the image generation network according to the second neural network trained and the first neural network to be trained respectively, to obtain first image features and second image features.

[0142] The training module 303 is configured to train the first neural network to be trained based on the first image features and the second image features, to obtain a trained first neural network.

[0143] The device trained by the neural network can respectively extract features of the new image generated by the image generation network based on the first neural network to be trained in the downstream task and the second neural network trained based on the first image sample collected in the upstream task, and then can train the first neural network based on the obtained first image features and second image features. Since the new image generated by the image generation network is an image sample more consistent with the overall distribution of the first image sample, such an image sample is more conducive to adapting to the network environment of the second neural network, and at the same time, the first image features output by the trained second neural network and the second image features output by the first neural network to be trained can better guide the training of the first neural network, thereby further improving the performance in the downstream task.

[0144] In a possible implementation, the obtaining module 301 is configured to train the image generation network according to the following steps:

[0145] obtain a first image output by a codebook generation network, the codebook generation network being configured to generate a codebook for decomposing the first image sample into a plurality of primitives;

[0146] input the first image into the image generation network to be trained to obtain a second image output by the image generation network;

[0147] determine a loss function value of the image generation network to be trained based on an image similarity between the second image and the first image;

[0148] train the image generation network to be trained based on the loss function value to obtain a trained image generation network.

[0149] In a possible implementation, the obtaining module 301 is configured to input the first image into the image generation network to be trained:

[0150] perform a covering process on part of the image region in the first image to obtain a first image after the covering process;

[0151] input the first image after the covering process into the image generation network to be trained.

[0152] In a possible implementation, the codebook generation network includes an encoder and a decoder, and the obtaining module 301 is configured to train the codebook generation network according to the following steps:

[0153] repeat the following steps until a similarity between an image output by the decoder and the first image sample input into the encoder is greater than a preset threshold:

[0154] input the first image sample to the encoder to be trained to obtain a codebook output by the encoder; and input the codebook output by the encoder to the decoder to be trained to obtain a first image output by the decoder.

[0155] In a possible implementation, the obtaining module 301 is configured to obtain the first image based on the codebook generation network output according to the following steps:

[0156] input the first image sample to the encoder included in the codebook generation network to obtain a codebook output by the encoder;

[0157] input the codebook output by the encoder to the decoder included in the codebook generation network to obtain a first image output by the decoder.

[0158] In a possible implementation, the image generation network includes a first generation subnetwork configured to generate a codebook for decomposing the first image sample into a plurality of primitives, and a second generation subnetwork configured to generate a new image based on an image output by the first generation subnetwork; and the obtaining module 301 is configured to train the image generation network according to the following steps:

[0159] input the first image sample to the trained first generation subnetwork to obtain a first image output by the first generation subnetwork;

[0160] input the first image to the second generation subnetwork to be trained to obtain a second image output by the second generation subnetwork;

[0161] determine a loss function value of the image generation network to be trained based on a first image similarity between the first image and the input first image sample, and a second image similarity between the second image and the first image;

[0162] train the image generation network to be trained based on the loss function value to obtain a trained image generation network.

[0163] In a possible implementation, the training module 303 is configured to train the first neural network to be trained based on the first image feature and the second image feature according to the following steps to obtain a trained first neural network:

[0164] determine a loss function value of the first neural network to be trained based on an image similarity between the first image feature and the second image feature;

[0165] if the loss function value corresponding to the current round is greater than a preset threshold, adjust a network parameter value of the first neural network based on the loss function value, and perform the next round of training according to the adjusted first neural network until the loss function value is less than or equal to the preset threshold.

[0166] In a possible implementation, the training module 303 is further configured to:

[0167] After the first trained neural network is obtained, a second image sample collected in a downstream task is acquired; the first trained neural network is retrained based on the second image sample, and a final trained first neural network is obtained.

[0168] In a possible implementation, the training module 303 is configured to retrain the first trained neural network based on the second image sample according to the following steps, and obtain a final trained first neural network:

[0169] The second image sample is input into the first neural network, and a task output result of the network is obtained;

[0170] Based on a comparison relationship between the task output result and a task annotation result annotated for the second image sample, a loss function value of the first neural network is determined;

[0171] The first neural network is retrained based on the loss function value, and a final trained first neural network is obtained.

[0172] In a possible implementation, the acquisition module 301 is configured to train the second neural network according to the following steps:

[0173] An original neural network is acquired; the original neural network at least includes a feature extraction layer;

[0174] The first image sample is subjected to feature extraction based on the feature extraction layer included in the original neural network, and image feature information output by the feature extraction layer is obtained;

[0175] The network parameter value of the feature extraction layer is adjusted based on the image feature information, and an adjusted feature extraction layer is obtained;

[0176] The original neural network including the adjusted feature extraction layer is determined as the second neural network trained.

[0177] Referring to Figure 4 FIG. 1 is a schematic diagram of a device for target detection provided by an embodiment of the present disclosure, and the device includes an acquisition module 401 and a detection module 402; wherein

[0178] The acquisition module 401 is configured to acquire a target image collected in a downstream task;

[0179] The detection module 402 is configured to input the target image into a first neural network trained by a method of training a neural network, and obtain a detection result of a target object in the target image.

[0180] The description of the processing flow of each module in the device and the interaction flow between the modules can refer to the related description in the above method embodiments, and will not be described in detail here.

[0181] Corresponding to Figure 1 and Figure 2 The method, the present embodiment of the disclosure also provides an electronic device, as shown in Figure 5 , the structural schematic diagram of the electronic device provided by the present embodiment of the disclosure, comprising:

[0182] The processor 501, the memory 502, and the bus 503; the memory 502 is used to store the execution instructions, including the internal memory 5021 and the external memory 5022; the internal memory 5021 here is also called the internal memory, which is used to temporarily store the operation data in the processor 501 and the data exchanged with the external memory 5022 such as the hard disk, the processor 501 exchanges data with the external memory 5022 through the internal memory 5021, when the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes Figure 1 The steps of the method of neural network training shown in Figure 2 The steps of the method of target detection shown in.

[0183] The present embodiment of the disclosure also provides a computer readable storage medium, the computer readable storage medium has a computer program stored thereon, the computer program is run by the processor to execute the steps of the method described in the above method embodiments. Wherein, the storage medium can be a volatile or non-volatile computer readable storage medium.

[0184] The present embodiment of the disclosure also provides a computer program product, the computer program product carries a program code, the instructions included in the program code can be used to execute the steps of the method described in the above method embodiments, and specific reference can be made to the above method embodiments, which will not be described here.

[0185] Among them, the above-mentioned computer program product can be realized by hardware, software or combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as software development kit (Software Development Kit, SDK) and the like.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0187] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0188] In addition, each functional unit in each embodiment of the present disclosure can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0189] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present disclosure. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0190] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for training a neural network, characterized in that, The method includes: The system acquires a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample. The second neural network is used for feature extraction, and the image generation network is used to generate a new image, wherein the new image conforms to the overall distribution of the first image sample. The new image generated based on the image generation network is subjected to feature extraction based on the second neural network trained and the first neural network to be trained, respectively, to obtain the first image features and the second image features. The first neural network to be trained is trained based on the first image features and the second image features to obtain a trained first neural network. The image generation network is trained according to the following steps: Obtain the first image output by the codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into multiple primitives; The first image is input into the image generation network to be trained, and the second image output by the image generation network is obtained. Based on the image similarity between the second image and the first image, the loss function value of the image generation network to be trained is determined; The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network.

2. The method according to claim 1, characterized in that, The step of inputting the first image into the image generation network to be trained includes: A portion of the image region in the first image is masked to obtain the masked first image. The first image after the masking process is input into the image generation network to be trained.

3. The method according to claim 1, characterized in that, The codebook generation network includes an encoder and a decoder, and is trained according to the following steps: Repeat the following steps until the similarity between the image output by the decoder and the first image sample input into the encoder is greater than a preset threshold: The first image sample is input into the encoder to be trained to obtain the codebook output by the encoder; the codebook output by the encoder is input into the decoder to be trained to obtain the image output by the decoder.

4. The method according to claim 3, characterized in that, Obtain the first image output by the codebook-based generation network by following these steps: The first image sample is input into the encoder included in the codebook generation network to obtain the codebook output by the encoder; The codebook output by the encoder is input into the decoder included in the codebook generation network to obtain the first image output by the decoder.

5. The method according to claim 1, characterized in that, The image generation network includes a first generation sub-network for generating a codebook that decomposes the first image sample into multiple primitives, and a second generation sub-network for generating the new image based on the image output by the first generation sub-network. The image generation network is trained according to the following steps: The first image sample is input into the trained first generation sub-network to obtain the first image output by the first generation sub-network. The first image is input into the second generative sub-network to be trained, and the second image output by the second generative sub-network is obtained. The loss function value of the image generation network to be trained is determined based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image. The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network.

6. The method according to any one of claims 1 to 5, characterized in that, The step of training the first neural network to be trained based on the first image features and the second image features to obtain a trained first neural network includes: Based on the image similarity between the first image features and the second image features, the loss function value of the first neural network to be trained is determined; If the loss function value in the current round is greater than a preset threshold, the network parameter values ​​of the first neural network are adjusted based on the loss function value, and the next round of training is performed based on the adjusted first neural network until the loss function value is less than or equal to the preset threshold.

7. The method according to any one of claims 1 to 5, characterized in that, After obtaining the trained first neural network, the method further includes: Acquire the second image sample collected in the downstream task; The first neural network, which has already been trained, is trained again based on the second image samples to obtain the final trained first neural network.

8. The method according to claim 7, characterized in that, The step of retraining the trained first neural network based on the second image samples to obtain the final trained first neural network includes: The second image sample is input into the first neural network to obtain the network's task output result; Based on the comparison between the task output and the task annotation results for the second image sample, the loss function value of the first neural network is determined. The first neural network is trained again based on the loss function value to obtain the final trained first neural network.

9. The method according to any one of claims 1 to 5, characterized in that, Train the second neural network using the following steps: Obtain the original neural network; the original neural network includes at least a feature extraction layer; Based on the feature extraction layer included in the original neural network, feature extraction is performed on the first image sample to obtain the image feature information output by the feature extraction layer; The network parameter values ​​of the feature extraction layer are adjusted based on the image feature information to obtain the adjusted feature extraction layer; The original neural network containing the adjusted feature extraction layer is identified as the trained second neural network.

10. A method for target detection, characterized in that, The method includes: Acquire target images collected in downstream tasks; The target image is input into a first neural network trained using the neural network training method according to any one of claims 1 to 9, to obtain the detection result of the target object in the target image.

11. A device for training a neural network, characterized in that, The device includes: The acquisition module is used to acquire a first image sample collected in the upstream task, a first neural network to be trained in the downstream task, and a second neural network and an image generation network trained based on the first image sample; the second neural network is used to perform feature extraction, and the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image sample. The extraction module is used to extract features from the new image generated based on the image generation network according to the trained second neural network and the first neural network to be trained, respectively, to obtain first image features and second image features; The training module is used to train the first neural network to be trained based on the first image features and the second image features to obtain the trained first neural network; The acquisition module is used to train the image generation network according to the following steps: Obtain the first image output by the codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into multiple primitives; The first image is input into the image generation network to be trained, and the second image output by the image generation network is obtained. Based on the image similarity between the second image and the first image, the loss function value of the image generation network to be trained is determined; The image generation network to be trained is trained based on the loss function value, resulting in a trained image generation network.

12. A target detection device, characterized in that, The device includes: The acquisition module is used to acquire target images collected in downstream tasks; The detection module is used to input the target image into a first neural network trained using the neural network training method according to any one of claims 1 to 9, and obtain the detection result of the target object in the target image.

13. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the neural network training method as described in any one of claims 1 to 9 or the steps of the object detection method as described in claim 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the neural network training method as described in any one of claims 1 to 9 or the object detection method as described in claim 10.

Images