Neural network training and classification method, apparatus, device, and storage medium

Abstract

The present disclosure provides a neural network training and classification method, device, equipment and storage medium, wherein the method comprises: obtaining each training picture sample and text description content corresponding to each training picture sample; extracting picture feature information from the training picture sample, and extracting text feature information from the text description content corresponding to the training picture sample; determining a loss function value of a neural network to be trained based on the extracted picture feature information and text feature information; in the case of not satisfying an iterative training stop condition of the neural network, adjusting a network parameter value of the neural network based on the determined loss function value, and performing retraining of the adjusted neural network until the iterative training stop condition of the neural network is satisfied, to obtain a trained neural network for processing a target picture. The present disclosure provides an additional supervision signal for picture training by using text feature information, which greatly improves the performance of network training.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of machine learning, and in particular, to a neural network training and classification method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous development of deep learning, various machine learning models have achieved greater and greater success in various industries and can be widely applied to various application scenarios such as image classification and target detection.

[0003] In actual application scenarios, the training samples that can be used are likely to be small, and directly training a neural network model on a small number of training samples often results in poor performance. In order to solve this problem, a larger computer vision dataset can be used in related technologies to pre-train a feature extractor, and then the feature extractor is migrated to a specific application scenario for training of a neural network model to improve the model performance.

[0004] However, due to the large domain span between the upstream task corresponding to the computer vision dataset and the downstream task corresponding to the current application scenario, the model performance improvement effect is not good. SUMMARY

[0005] The embodiments of the present disclosure at least provide a neural network training and classification method, device, equipment and storage medium.

[0006] In a first aspect, the embodiments of the present disclosure provide a neural network training method, comprising:

[0007] obtaining each training picture sample and text description content corresponding to each training picture sample;

[0008] extracting picture feature information from the training picture sample and text feature information from the text description content corresponding to the training picture sample;

[0009] determining a loss function value of a neural network to be trained based on the extracted picture feature information and text feature information;

[0010] In the case where the iterative training stop condition of the neural network is not met, adjusting the network parameter value of the neural network based on the determined loss function value, and performing re-training of the adjusted neural network until the iterative training stop condition of the neural network is met, to obtain a trained neural network for processing a target picture.

[0011] With the above neural network training method, after obtaining each training picture sample and the text description content corresponding to each training picture sample, picture feature information and text feature information can be extracted from the training picture sample and the corresponding text description content respectively, so as to determine the loss function value of the neural network to be trained, and the network training can be adjusted based on the loss function value to train the neural network. The present disclosure provides an additional supervision signal for picture training by using text feature information, which can effectively reduce the adverse effects of cross-scene data migration on network performance, greatly improving the performance of network training, and the accuracy and robustness are better.

[0012] In a possible implementation, the neural network comprises a first feature extractor and a second feature extractor.

[0013] The picture feature information is extracted from the training picture sample, comprising:

[0014] The first feature extractor is used for picture feature extraction on the training picture sample, to obtain the picture feature information corresponding to the training picture sample.

[0015] The text feature information is extracted from the text description content corresponding to the training picture sample, comprising:

[0016] The second feature extractor is used for text feature extraction on the text description content corresponding to the training picture sample, to obtain the text feature information corresponding to the training picture sample.

[0017] In a possible implementation, the neural network comprises a classifier; and the loss function value of the neural network to be trained is determined based on the extracted picture feature information and text feature information, comprising:

[0018] A first sub-loss function value is determined based on the extracted picture feature information and text feature information; and a second sub-loss function value is determined based on the extracted picture feature information and the classifier.

[0019] The loss function value is determined based on the first sub-loss function value and the second sub-loss function value.

[0020] Here, the first sub-loss function value and the second sub-loss function value are used to supervise network training respectively, and the former can provide a supervision signal for network training on the similarity between pictures and texts, which can be used as a complementary signal for the latter to provide a supervision signal for network training on picture classification, thereby effectively improving the network training performance under cross-domain data migration.

[0021] In a possible implementation, the determining the first sub-loss function value based on the extracted picture feature information and the text feature information comprises:

[0022] determining a feature similarity between the extracted picture feature information and the text feature information;

[0023] determining the first sub-loss function value based on the feature similarity.

[0024] In a possible implementation, when the text description content corresponding to the training picture sample is multiple, the determining the first sub-loss function value based on the feature similarity comprises:

[0025] determining the first sub-loss function value based on a feature similarity between the extracted picture feature information and each of the text feature information and a weight value corresponding to each of the text description content.

[0026] Here, the first sub-loss function value can be determined based on the weight for the multiple text description content corresponding to the training picture sample, and is more accurate.

[0027] In a possible implementation, the determining the second sub-loss function value based on the extracted picture feature information and the classifier comprises:

[0028] inputting the extracted picture feature information into the classifier to obtain a classification result output by the classifier;

[0029] determining the second sub-loss function value based on the classification result output by the classifier and a label result pre-labeled for the training picture sample.

[0030] In a possible implementation, before the inputting the extracted picture feature information into the classifier, the method further comprises:

[0031] initializing the classifier by using the extracted text feature information to obtain an initialized classifier;

[0032] the inputting the extracted picture feature information into the classifier to obtain the classification result output by the classifier comprises:

[0033] inputting the extracted picture feature information into the initialized classifier to obtain the classification result output by the classifier.

[0034] In a possible implementation, the initializing the classifier by using the extracted text feature information to obtain the initialized classifier comprises:

[0035] determine an initial weight of the classifier based on the extracted text feature information and a preset classification result matched by the text feature information corresponding to the training picture sample;

[0036] determine the initial weight as a network parameter value of the initialized classifier.

[0037] Here, the network parameter value of the initialized classifier can be determined based on the text feature information, which can improve the classification accuracy of the classifier to some extent since the text feature can describe the classification information from the text dimension.

[0038] In a possible implementation, the text description content corresponding to the training picture sample is determined in at least one of the following manners:

[0039] obtain the text description content corresponding to the training picture sample in response to a manual annotation request for the training picture sample;

[0040] obtain a plurality of picture-text pairs through a network, compare the training picture sample with pictures included in each picture-text pair, and determine text content included in a picture-text pair that is successfully compared as the text description content corresponding to the training picture sample;

[0041] extract text from the training picture sample by using a trained text generation network to obtain the text description content corresponding to the training picture sample.

[0042] In a second aspect, the embodiments of the present disclosure further provide a classification method, comprising:

[0043] obtain a target picture;

[0044] input the target picture into a neural network trained by the method according to any one of the first aspect and various embodiments thereof to obtain a classification result corresponding to the target picture.

[0045] In a third aspect, the embodiments of the present disclosure further provide a neural network training apparatus, comprising:

[0046] an obtaining module configured to obtain each training picture sample and text description content corresponding to each training picture sample;

[0047] an extracting module configured to extract picture feature information from the training picture sample and text feature information from the text description content corresponding to the training picture sample;

[0048] a determining module configured to determine a loss function value of a neural network to be trained based on the extracted picture feature information and text feature information;

[0049] The training module is configured to, in a case where the iterative training stop condition of the neural network is not met, adjust the network parameter value of the neural network based on the determined loss function value, and perform re-training on the adjusted neural network until the iterative training stop condition of the neural network is met, to obtain the trained neural network for processing the target picture.

[0050] In a fourth aspect, the embodiments of the present disclosure further provide a classification device, comprising:

[0051] The acquisition module is configured to acquire a target picture.

[0052] The classification module is configured to input the target picture into the neural network trained by the method according to any one of the first aspect and the various embodiments thereof, to obtain a classification result corresponding to the target picture.

[0053] In a fifth aspect, the embodiments of the present disclosure further provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the method for training the neural network according to any one of the first aspect and the various embodiments thereof or the steps of the method for classification according to the second aspect.

[0054] In a sixth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the steps of the method for training the neural network according to any one of the first aspect and the various embodiments thereof or the steps of the method for classification according to the second aspect are performed.

[0055] The effects of the above device, electronic device and computer readable storage medium are described in the above method, which will not be repeated here.

[0056] In order to make the above objectives, features and advantages of the present disclosure more apparent, the following will specifically describe preferred embodiments in conjunction with the accompanying drawings, and make a detailed description as follows. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments, the drawings herein are incorporated into the description and form a part of the description, the drawings show the embodiments consistent with the present disclosure, and are used to illustrate the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without paying creative labor on the basis of the drawings.

[0058] Figure 1 A flow chart of a method for training a neural network is shown according to an embodiment of the present disclosure.

[0059] Figure 2 A specific flow chart of a method for training a neural network is shown according to an embodiment of the present disclosure.

[0060] Figure 3 A schematic diagram of an apparatus for training a neural network is shown according to an embodiment of the present disclosure.

[0061] Figure 4 A schematic diagram of an apparatus for classification is shown according to an embodiment of the present disclosure.

[0062] Figure 5 A schematic diagram of an electronic device is shown according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0064] It should be noted that: similar reference numerals and letters in the following drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0065] The term “and / or” herein is only used to describe an associated relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term “at least one” herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0066] It is found through research that a feature extractor can be pre-trained using a large computer vision dataset, and then the feature extractor is migrated to a specific application scenario for training of a neural network model to improve model performance.

[0067] When migrating the upstream-trained feature extractor to a downstream dataset, common migration methods include fine-tuning, multi-task learning, meta-learning, etc. However, due to a large domain span between the upstream task corresponding to the computer vision dataset and the downstream task corresponding to the current application scenario, the model performance improvement effect is not good.

[0068] Based on the above research, the present disclosure provides a neural network training and classification method, device, equipment, and storage medium to improve training performance.

[0069] To facilitate understanding of the present embodiment, first a neural network training method disclosed by the present embodiment is described in detail. The execution subject of the neural network training method provided by the present embodiment is generally an electronic device with certain computing power, which includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the neural network training method can be realized by a processor invoking computer-readable instructions stored in a memory.

[0070] Referring to Figure 1 FIG. 1 shows a flowchart of the neural network training method provided by the present embodiment. The method includes steps S101-S104, wherein:

[0071] S101: Obtain each training picture sample and text description content corresponding to each training picture sample;

[0072] S102: Extract picture feature information from the training picture sample, and extract text feature information from the text description content corresponding to the training picture sample;

[0073] S103: Determine a loss function value of a neural network to be trained based on the extracted picture feature information and text feature information;

[0074] S104: In a case where an iterative training stop condition of the neural network is not met, adjust a network parameter value of the neural network based on the determined loss function value, and perform re-training of the adjusted neural network until the iterative training stop condition of the neural network is met, to obtain a trained neural network for processing a target picture.

[0075] In order to facilitate understanding of the method of neural network training provided by the embodiments of the present disclosure, the application scenario of the method is first simply described as follows. The method of neural network training in the embodiments of the present disclosure can be mainly applied to a downstream training task under scene migration. The downstream training task can be a related task based on the current migrated scene, and has a relatively small number of training samples. The downstream training task corresponds to an upstream training task, which can be a related task with a large number of training samples. Taking a target classification task as an example, a target classification neural network trained by a training database composed of various target objects is currently available. However, for the specific application scenario of autonomous driving, the training data corresponding to this scenario is relatively small, and therefore the training of the downstream task often needs to be supported by the training results of the upstream task.

[0076] In the related art, the feature extractor of the upstream training can be adjusted by fine-tuning and the like, and the adjusted feature extractor can be applied to the downstream task. However, due to the large domain span, the performance of the trained neural network is not good.

[0077] In order to solve the above problems, the embodiments of the present disclosure provide a neural network training scheme combining text features for network supervision. The performance of the trained neural network is better.

[0078] Before the neural network training, the embodiments of the present disclosure can obtain a plurality of training picture samples and text description content corresponding to each training picture sample. The training picture sample corresponds to a picture collected by the downstream task, and the pictures collected for different application scenarios are different. The text description content can be related text obtained by text description of the training picture sample.

[0079] The text description content can be determined based on various methods.

[0080] Firstly, the text description content corresponding to the training picture sample can be obtained based on manual annotation, that is, by responding to a manual annotation request for the training picture sample. The text description generated by manual annotation for the picture is relatively accurate.

[0081] Secondly, a plurality of picture-text pairs can be obtained through the network, the training picture sample is compared with the pictures included in each picture-text pair, and the text content included in the picture-text pair that is successfully compared is determined as the text description content corresponding to the training picture sample. It is mainly considered that a large number of picture-text pairs naturally exist on the Internet, and therefore the text can be filtered by matching the similarity of the downstream picture and the picture on the network, which is convenient and fast and has scalability.

[0082] Third, a trained text generation network can be used to extract text from training image samples to obtain the corresponding text descriptions. This text generation network can be based on the ImageCaption task, which is a task that integrates computer vision and natural language processing. It requires the use of a computer to establish a mapping method to map data from the visual modality to the text modality, offering convenience, speed, and scalability.

[0083] In addition to the above, the embodiments of this disclosure can also generate text description content based on other methods. For example, a corresponding structured description statement can be preset for the image. First, the corresponding entity elements are extracted from the image, and then the extracted entity elements are substituted into the structured description statement to obtain the text description content. In practical applications, the specific text description content generation method can be determined based on different application requirements, and the embodiments of this disclosure do not impose specific limitations on this.

[0084] Given training image samples and corresponding text descriptions, image feature information and text feature information can be extracted from the training image samples and text descriptions respectively. Then, based on the two extracted feature information (i.e., image feature information and text feature information), the loss function value of the neural network to be trained is determined. Finally, the neural network is trained by adjusting the neural network based on the loss function value.

[0085] The extracted image feature information can be image features related to the training image samples. These image features can include scene features related to the image itself, features such as image texture and color, and features related to the target object in the image, such as the position of the target person in the image. In this embodiment, image feature information can be determined based on image processing or deep learning. Specifically, if the neural network includes a first feature extractor, the aforementioned image feature information can be extracted using the first feature extractor.

[0086] Furthermore, the extracted text feature information can be text features related to the text description content. These text features can include information such as keywords / groups, or features including semantic information. Embodiments of this disclosure can also extract text feature information based on deep learning. Here, if the neural network includes a second feature extractor, the aforementioned text feature information can be extracted using the second feature extractor.

[0087] Based on the extracted picture feature information and text feature information, the loss function value of the neural network to be trained can be determined. The loss function value here can be determined in part by the contrast loss corresponding to the combination of picture feature information and text feature information, and in part by the classification loss corresponding to the picture feature information. The specific implementation can be achieved by the following steps:

[0088] Step one, based on the extracted picture feature information and text feature information, determine the first sub-loss function value; and based on the extracted picture feature information and the classifier, determine the second sub-loss function value;

[0089] Step two, based on the first sub-loss function value and the second sub-loss function value, determine the loss function value.

[0090] Here, on the one hand, the first sub-loss function value can be determined based on the feature similarity between the extracted picture feature information and the text feature information, and on the other hand, the extracted picture feature information can be input into the classifier to obtain the classification result output by the classifier, and then based on the classification result output by the classifier and the annotation result pre-annotated for the training picture sample, the second sub-loss function value can be determined.

[0091] Among them, the first sub-loss function value can be used as a contrast supervision signal. For a training picture sample, the closer its picture feature information and corresponding text feature information are, the more accurately the text description content can describe the training picture sample. Through this contrast supervision information, the present embodiment can well use the text feature as the reference feature of the picture feature, providing more training reference dimensions, which will significantly improve the training performance of the network.

[0092] In order to further improve the training performance of the network, the multiple text description contents corresponding to a training picture sample can be weighted, and then the first sub-loss function value is determined.

[0093] In actual application, for any training picture sample, the text description corresponding to the sample can be generated by using the above text generation scheme, so as to obtain multiple text features. After weighting and summing, the contrast supervision loss is calculated, which can effectively improve the performance of the neural network compared with using only a single text description.

[0094] In addition, the second sub-loss function value can be used as a classification supervision signal. The disclosed embodiments can initialize the classifier using the extracted text feature information, and then determine the classification supervision signal using the initialized classifier. This is mainly because the random initialization method used in the related art for the classifier cannot well meet the training performance of the neural network. The text feature information determined for the training picture sample in the disclosed embodiments can to some extent represent the classification features of the picture, and thus the classifier initialized by the text feature information can further improve the performance of subsequent network training. The initialization of the classifier can be performed according to the following steps in the disclosed embodiments:

[0095] Step 1: Determine the initial weights of the classifier based on the extracted text feature information and the preset classification result matched by the training picture sample corresponding to the text feature information.

[0096] Step 2: Determine the initial weights as the network parameter values of the initialized classifier.

[0097] Here, the preset classification result matched by the training picture sample corresponding to the text feature information can be determined first, and then the initial values are assigned to the network parameters of the classifier based on the determined preset classification result. That is, the classification features corresponding to the text feature information can be pre-assigned to the classifier for learning, which can greatly improve the classification performance of the classifier compared with random initialization.

[0098] In order to further understand the method for training the neural network provided by the disclosed embodiments, the following can be combined with Figure 2 The process of training the neural network is described in detail.

[0099] As shown in Figure 2 The picture-text pair includes a picture (corresponding to a training picture sample) and a text (corresponding to the text description content of the training picture sample). The text description content is input into the second feature extractor to generate text feature information of the text. The text feature information and the picture feature information are compared and learned to calculate the cosine similarity, and the similarity is used as a contrast supervision loss (corresponding to the first sub-loss function value). For the training picture sample, the ResNet50 pre-trained model can be used as a visual feature extractor (i.e., the first feature extractor). The picture data of the downstream task is input into the ResNet50 model to extract the picture feature information, which is then sent to the classifier (Classfier) initialized by the text feature information to calculate the cross-entropy loss (corresponding to the second sub-loss function value).

[0100] After determining the contrast supervision loss and the cross-entropy loss, the two losses are added together as the final loss, and back propagation is performed. If the iterative training termination condition of the neural network is not met, the network parameter value of the neural network is adjusted based on the determined loss function value, and the adjusted neural network is retrained until the iterative training termination condition of the neural network is met, and a trained neural network for processing target pictures is obtained.

[0101] The iterative training termination condition can be that the number of training reaches a preset number, or that the finally determined loss is less than a preset value, or that all training picture samples are traversed, or other medium conditions, which are not specifically limited by the embodiments of the present disclosure.

[0102] Based on the trained neural network, the embodiments of the present disclosure can realize picture classification, which can be achieved by the following steps:

[0103] Step one, obtaining a target picture;

[0104] Step two, inputting the target picture into the trained neural network according to the method for training the neural network to obtain a classification result corresponding to the target picture.

[0105] The target picture can be a picture collected in the current environment, and the trained neural network can be used to realize picture classification.

[0106] It should be noted that the picture classification is only a specific application example of the neural network, and in actual application, the neural network can also be used to perform picture detection, recognition and other specific tasks, which are not specifically limited by the embodiments of the present disclosure.

[0107] Those skilled in the art can understand that the writing order of the steps in the above method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by its function and possible internal logic.

[0108] Based on the same inventive concept, the embodiments of the present disclosure also provide a device corresponding to the method. Since the principle of solving problems by the device in the embodiments of the present disclosure is similar to the above-mentioned method of the present disclosure, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described.

[0109] Referring to Figure 3 FIG. 1 shows a schematic diagram of a neural network training device provided by the embodiments of the present disclosure. The device includes an obtaining module 301, an extracting module 302, a determining module 303 and a training module 304. The obtaining module 301 is configured to obtain a plurality of training picture samples, and the extracting module 302 is configured to extract a plurality of feature vectors from the plurality of training picture samples.

[0110] The acquisition module 301 is configured to acquire each training picture sample and text description content corresponding to each training picture sample.

[0111] The extraction module 302 is configured to extract picture feature information from the training picture sample and text feature information from the text description content corresponding to the training picture sample.

[0112] The determination module 303 is configured to determine a loss function value of the neural network to be trained based on the extracted picture feature information and text feature information.

[0113] The training module 304 is configured to, in a case where the iterative training termination condition of the neural network is not met, adjust a network parameter value of the neural network based on the determined loss function value, and perform re-training of the adjusted neural network until the iterative training termination condition of the neural network is met, to obtain a trained neural network for processing the target picture.

[0114] The above-described neural network training apparatus can extract picture feature information and text feature information from the training picture sample and the corresponding text description content, respectively, in a case where each training picture sample and text description content corresponding to each training picture sample are acquired, so as to determine a loss function value of the neural network to be trained, and perform network training adjustment based on the loss function value to train the neural network. The present disclosure provides an additional supervision signal for picture training by using text feature information, which can effectively reduce the adverse effects of cross-scene data migration on network performance, greatly improve the performance of network training, and has better accuracy and robustness.

[0115] In a possible implementation, the neural network comprises a first feature extractor and a second feature extractor.

[0116] The extraction module 302 is configured to extract picture feature information from the training picture sample according to the following steps:

[0117] The first feature extractor is used to perform picture feature extraction on the training picture sample, to obtain picture feature information corresponding to the training picture sample.

[0118] The extraction module 302 is configured to extract text feature information from the text description content corresponding to the training picture sample according to the following steps:

[0119] The second feature extractor is used to perform text feature extraction on the text description content corresponding to the training picture sample, to obtain text feature information corresponding to the training picture sample.

[0120] In a possible implementation, the neural network comprises a classifier; and the determining module 303 is configured to determine the loss function value of the neural network to be trained based on the extracted picture feature information and the text feature information according to the following steps:

[0121] determining a first sub-loss function value based on the extracted picture feature information and the text feature information; and determining a second sub-loss function value based on the extracted picture feature information and the classifier;

[0122] determining the loss function value based on the first sub-loss function value and the second sub-loss function value.

[0123] In a possible implementation, the determining module 303 is configured to determine the first sub-loss function value based on the extracted picture feature information and the text feature information according to the following steps:

[0124] determining a feature similarity between the extracted picture feature information and the text feature information;

[0125] determining the first sub-loss function value based on the feature similarity.

[0126] In a possible implementation, in a case where the text description content corresponding to the training picture sample is multiple, the determining module 303 is configured to determine the first sub-loss function value based on the feature similarity according to the following steps:

[0127] determining the first sub-loss function value based on the feature similarity between the extracted picture feature information and each text feature information, and a weight value corresponding to each text description content.

[0128] In a possible implementation, the determining module 303 is configured to determine the second sub-loss function value based on the extracted picture feature information and the classifier according to the following steps:

[0129] inputting the extracted picture feature information into the classifier to obtain a classification result output by the classifier;

[0130] determining the second sub-loss function value based on the classification result output by the classifier and a label result pre-labeled for the training picture sample.

[0131] In a possible implementation, the determining module 303 is configured to input the extracted picture feature information into the classifier to obtain the classification result output by the classifier according to the following steps:

[0132] initializing the classifier based on the extracted text feature information before the extracted picture feature information is input into the classifier, to obtain an initialized classifier;

[0133] Input the extracted picture feature information into the initialized classifier to obtain a classification result output by the classifier.

[0134] In a possible implementation, the determining module 303 is configured to initialize the classifier by using the extracted text feature information according to the following steps, to obtain the initialized classifier.

[0135] Determine an initial weight of the classifier based on the extracted text feature information and a preset classification result matched by the text feature information to the training picture sample.

[0136] Determine the initial weight as a network parameter value of the initialized classifier.

[0137] In a possible implementation, the obtaining module 301 is configured to determine the text description content corresponding to the training picture sample in at least one of the following manners:

[0138] Obtain the text description content corresponding to the training picture sample in response to a manual labeling request for the training picture sample.

[0139] Obtain a plurality of groups of picture-text pairs through a network, compare the training picture sample with pictures included in each group of picture-text pairs, and determine text content included in a picture-text pair that is successfully compared as the text description content corresponding to the training picture sample.

[0140] Extract text from the training picture sample by using the trained text generation network to obtain the text description content corresponding to the training picture sample.

[0141] Referring to Figure 4 FIG. 1 is a schematic diagram of a classification device provided by an embodiment of the present disclosure, which includes an obtaining module 401 and a classification module 402, where

[0142] The obtaining module 401 is configured to obtain a target picture.

[0143] The classification module 402 is configured to input the target picture into a neural network trained according to the method to obtain a classification result corresponding to the target picture.

[0144] The processing flow of each module in the device and the interaction flow between the modules can refer to the related descriptions in the above method embodiments, which will not be described in detail here.

[0145] The present disclosure also provides an electronic device, as shown in Figure 5 FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure, which at least includes a processor 501, a memory 502, and a bus 503. The memory 502 stores machine-readable instructions executable by the processor 501 (such as Figure 3corresponding to the obtaining module 301, the extracting module 302, the determining module 303 and the training module 304 in the device in the foregoing embodiment, and so on, and again, for example, Figure 4 corresponding to the obtaining module 401, the classification module 402 in the device in the foregoing embodiment, and so on), when the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503, and the machine readable instructions are executed by the processor 501 to execute the method for training the neural network or the method for classification.

[0146] In the process of executing the method for training the neural network, the electronic device provided by the embodiment of the disclosure can further include the transceiver 504, so that the transceiver 504 can receive each training picture sample and the text description content corresponding to each training picture sample, thereby facilitating the processor 501 to perform the related operation of neural network training; on the other hand, in the process of executing the method for classification, the electronic device provided by the embodiment of the disclosure can receive the target picture through the transceiver 504, thereby facilitating the processor 501 to perform the related operation of classification.

[0147] The embodiment of the disclosure further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by a processor, the steps of the method described in the method embodiment are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0148] The embodiment of the disclosure further provides a computer program product, and the computer program product carries a program code. The instructions included in the program code can be used to execute the steps of the method described in the method embodiment. For details, refer to the method embodiment, which will not be repeated here.

[0149] The computer program product can be specifically implemented by hardware, software or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and another division can be made in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0151] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0152] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0153] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0154] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for training a neural network, characterized in that, The method includes: Obtain each training image sample and the corresponding text description content for each training image sample; Image feature information is extracted from the training image samples, and text feature information is extracted from the text description content corresponding to the training image samples; Based on the extracted image feature information and text feature information, the loss function value of the neural network to be trained is determined; If the iterative training cutoff condition of the neural network is not met, the network parameter values ​​of the neural network are adjusted based on the determined loss function value, and the adjusted neural network is retrained until the iterative training cutoff condition of the neural network is met, thus obtaining a trained neural network for processing the target image. The neural network includes a classifier; determining the loss function value of the neural network to be trained based on the extracted image feature information and text feature information includes: Based on the extracted image feature information and text feature information, a first sub-loss function value is determined; and based on the extracted image feature information and the classifier, a second sub-loss function value is determined. The loss function value is determined based on the first sub-loss function value and the second sub-loss function value; The step of determining the value of the first sub-loss function based on the extracted image feature information and text feature information includes: Determine the feature similarity between the extracted image feature information and the text feature information; Based on the feature similarity, the value of the first sub-loss function is determined; The step of determining the value of the second sub-loss function based on the extracted image feature information and the classifier includes: The extracted image feature information is input into the classifier to obtain the classification result output by the classifier; The value of the second sub-loss function is determined based on the classification result output by the classifier and the annotation result of the pre-annotated training image samples.

2. The method according to claim 1, characterized in that, The neural network includes a first feature extractor and a second feature extractor; The step of extracting image feature information from the training image samples includes: The first feature extractor is used to extract image features from the training image samples to obtain the image feature information corresponding to the training image samples. The step of extracting text feature information from the text description content corresponding to the training image samples includes: The second feature extractor is used to extract text features from the text description content corresponding to the training image sample to obtain the text feature information corresponding to the training image sample.

3. The method according to claim 1, characterized in that, When there are multiple text descriptions corresponding to the training image samples, determining the value of the first sub-loss function based on the feature similarity includes: The first sub-loss function value is determined based on the feature similarity between the extracted image feature information and each of the text feature information, as well as the weight value corresponding to each text description content.

4. The method according to claim 1, characterized in that, Before inputting the extracted image feature information into the classifier, the method further includes: The classifier is initialized using the extracted text feature information to obtain the initialized classifier; The step of inputting the extracted image feature information into the classifier to obtain the classification result output by the classifier includes: The extracted image feature information is input into the initialized classifier to obtain the classification result output by the classifier.

5. The method according to claim 4, characterized in that, The process of initializing the classifier using the extracted text feature information to obtain the initialized classifier includes: Based on the extracted text feature information and the preset classification results of the training image samples corresponding to the text feature information, the initial weights of the classifier are determined. The initial weights are determined as the network parameter values ​​of the initialized classifier.

6. The method according to any one of claims 1 to 5, characterized in that, The text description content corresponding to the training image sample shall be determined in at least one of the following ways: In response to a request for manual annotation of the training image samples, obtain the text description content corresponding to the training image samples; Multiple sets of image-text pairs are obtained through the network. The training image sample is compared with the images included in each set of image-text pairs, and the text content included in the successfully matched image-text pairs is determined as the text description content corresponding to the training image sample. The trained text generation network is used to extract text from the training image samples to obtain the text description content corresponding to the training image samples.

7. A classification method, characterized in that, include: Obtain the target image; The target image is input into the neural network trained according to any one of claims 1 to 6 to obtain the classification result corresponding to the target image.

8. A device for training a neural network, characterized in that, The device includes: The acquisition module is used to acquire each training image sample and the text description content corresponding to each training image sample; The extraction module is used to extract image feature information from the training image samples and extract text feature information from the text description content corresponding to the training image samples. The determination module is used to determine the loss function value of the neural network to be trained based on the extracted image feature information and text feature information. The training module is used to adjust the network parameter values ​​of the neural network based on the determined loss function value when the iterative training cutoff condition of the neural network is not met, and to retrain the adjusted neural network until the iterative training cutoff condition of the neural network is met, so as to obtain a trained neural network for processing the target image. The neural network includes a classifier; determining the loss function value of the neural network to be trained based on the extracted image feature information and text feature information includes: Based on the extracted image feature information and text feature information, a first sub-loss function value is determined; and based on the extracted image feature information and the classifier, a second sub-loss function value is determined. The loss function value is determined based on the first sub-loss function value and the second sub-loss function value; The step of determining the value of the first sub-loss function based on the extracted image feature information and text feature information includes: Determine the feature similarity between the extracted image feature information and the text feature information; Based on the feature similarity, the value of the first sub-loss function is determined; The step of determining the value of the second sub-loss function based on the extracted image feature information and the classifier includes: The extracted image feature information is input into the classifier to obtain the classification result output by the classifier; The value of the second sub-loss function is determined based on the classification result output by the classifier and the annotation result of the pre-annotated training image samples.

9. A sorting device, characterized in that, include: The acquisition module is used to acquire the target image; The classification module is used to input the target image into the neural network trained according to any one of the methods described in claims 1-6 to obtain the classification result corresponding to the target image.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the neural network training method as described in any one of claims 1 to 6 or the steps of the classification method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the neural network training method as described in any one of claims 1 to 6 or the classification method as described in claim 7.

Images