Neural network training method and device, electronic equipment and readable storage medium

Abstract

The application discloses a neural network training method and device, electronic equipment and a readable storage medium. The method comprises the following steps: obtaining a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature; determining a training loss according to a first result and a second result, wherein the first result is obtained by predicting the first training feature by a target neural network, and the second result is obtained by predicting the second training feature by the target neural network; and training the target neural network according to the training loss. The application solves the technical problem that the text recognition model has low prediction accuracy due to the fact that the related technical text recognition model cannot recognize similar features.

Description

Technical Field

[0001] This invention relates to the field of database technology, and more specifically, to a neural network training method and apparatus, an electronic device, and a readable storage medium. Background Technology

[0002] Among related technologies, the BERT model (Bidirectional Encoder Representation from Transformers) aims to address the issue of text context. However, the BERT model's loss function does not provide special handling for synonyms, synonym phrases, and synonym sentences.

[0003] For example, in the sentences "solve one problem" and "solve two problems", after the third character is masked, the BERT model cannot predict the two sentences after data augmentation. In other words, the BERT model cannot distinguish this character and cannot identify synonyms, synonym phrases, and synonym sentences.

[0004] It is evident that no effective solution has yet been proposed in the relevant technologies to address the aforementioned problems. Summary of the Invention

[0005] This invention provides a neural network training method and apparatus, an electronic device, and a readable storage medium to at least solve the technical problem that text recognition models have low prediction accuracy because they cannot recognize similar features in related technologies.

[0006] According to one aspect of the present invention, a neural network training method is provided, comprising: acquiring a first training feature and a second training feature, wherein the second training feature and the first training feature are similar features; determining a training loss based on a first result and a second result, wherein the first result is obtained by a target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature; and training the target neural network based on the training loss.

[0007] Furthermore, the target neural network is a text recognition model, and the first training feature and the second training feature are text. Obtaining the first training feature and the second training feature includes at least one of the following: performing data augmentation on words in the first training feature to obtain the second training feature; performing data augmentation on phrases in the first training feature to obtain the second training feature; and performing data augmentation on sentences in the first training feature to obtain the second training feature.

[0008] Further, determining the training loss based on the first result and the second result includes: when the first training feature and the second training feature are different words, mutually exclusiveizing the first result and the second result to obtain the word training loss.

[0009] Further, determining the training loss based on the first result and the second result includes: when the first training feature and the second training feature are different phrases, mutually exclusiveizing the first result and the second result to obtain the phrase training loss.

[0010] Further, determining the training loss based on the first result and the second result includes: when the first training feature and the second training feature are different sentences, mutually exclusiveizing the first result and the second result to obtain the sentence training loss.

[0011] Further, determining the training loss based on the first result and the second result includes: inputting the second result into a preset decoder to obtain a first decoding loss corresponding to the preset decoder.

[0012] Furthermore, determining the training loss based on the first result and the second result further includes: attracting the first result and the decoding loss to obtain a second decoding loss.

[0013] According to another aspect of the present invention, a neural network training apparatus is also provided, comprising: an acquisition unit for acquiring a first training feature and a second training feature, wherein the second training feature and the first training feature are similar features; a determination unit for determining a training loss based on a first result and a second result, wherein the first result is obtained by a target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature; and a training unit for training the target neural network based on the training loss.

[0014] According to another aspect of the present invention, an electronic device is also provided, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the neural network training method as described above.

[0015] According to another aspect of the present invention, a readable storage medium is also provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the neural network training method as described above.

[0016] In this embodiment of the invention, by acquiring a first training feature and a second training feature, wherein the second training feature and the first training feature are similar features; determining a training loss based on a first result and a second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature; and training the target neural network based on the training loss, the purpose of adding a corresponding optimization objective to the training process of similar features is achieved, thereby realizing the technical effect of improving the accuracy of the target neural network in recognizing similar features, and thus solving the technical problem that the text recognition model has low prediction accuracy because the text recognition model in related technologies cannot recognize similar features. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart illustrating an optional neural network training method according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram illustrating data augmentation using an optional BERT model according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of an optional word training loss according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram illustrating data augmentation using an optional BERT model according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of an optional method for obtaining phrase training loss according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram illustrating an optional method for obtaining a second decoding loss according to an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of an optional neural network training device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Example 1

[0028] According to embodiments of the present invention, a neural network training method is provided, such as... Figure 1 As shown, the method includes:

[0029] S102, obtain the first training feature and the second training feature, wherein the second training feature and the first training feature are similar features;

[0030] Specifically, in this embodiment, the first training feature and the second training feature are similar features, including but not limited to similar sentence features and similar image features. Similar features are two features that differ in some aspects, such as a different character, word, or sentence in the sentence features, or a different image element in the image features.

[0031] In one example, the sentence features "solve one problem" and "solve two problems" are similar features with different characters, and the sentence features "solve one kind of problem" and "solve one problem" are similar features with different words.

[0032] In this embodiment, the second training feature corresponding to the first training feature is obtained by performing data augmentation masking on the training features in the preset training dataset. After performing data augmentation on some features or some elements in the first training feature, similar features, i.e., the second training feature, are obtained.

[0033] S104, determine the training loss based on the first result and the second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature;

[0034] Specifically, in this embodiment, a pre-trained target neural network is used to predict the first training feature and the second training feature, respectively, to obtain a first result corresponding to the first training feature and a second result corresponding to the second training feature. In specific application scenarios, this can be a text recognition model recognizing text features, or an image recognition model recognizing image features.

[0035] For example, a pre-trained text recognition model is used to predict the first text (first training feature) to obtain the first result corresponding to the first text. The text recognition model is then used to predict the second text (second training feature) to obtain the second result corresponding to the second text.

[0036] Determining the training loss using the first and second results can be achieved by performing a dot product operation on the vectors of the first and second results. For example, performing a mutual exclusion operation on the first and second results can yield the training loss for the text recognition model.

[0037] S106, Train the target neural network based on the training loss.

[0038] Specifically, in this embodiment, the first training feature and the second training feature are predicted respectively to obtain the corresponding first result and second result. The training loss is obtained based on the first result and the second result. Given the preset loss of the target neural network, the training loss is added to train the target neural network until the target neural network converges.

[0039] It should be noted that, in this embodiment, a first training feature and a second training feature are obtained, wherein the second training feature and the first training feature are similar features; a training loss is determined based on a first result and a second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature; the target neural network is trained based on the training loss. This achieves the goal of incorporating a corresponding optimization objective into the training process of similar features, thereby improving the technical effect of the target neural network in recognizing similar features, and thus solving the technical problem of low prediction accuracy of text recognition models due to the inability of text recognition models to recognize similar features in related technologies.

[0040] Optionally, in this embodiment, the target neural network is a text recognition model, and the first training feature and the second training feature are text. Obtaining the first training feature and the second training feature includes at least one of the following: performing data augmentation on words in the first training feature to obtain the second training feature; performing data augmentation on phrases in the first training feature to obtain the second training feature; and performing data augmentation on sentences in the first training feature to obtain the second training feature.

[0041] In specific application scenarios, when the target neural network is a text recognition model, the predicted input feature is text, and the first training feature is text. Then, obtaining the corresponding second training feature through the first training feature includes, but is not limited to: data augmentation of words in the text, data augmentation of phrases in the text, and data augmentation of sentences in the text. For example, data augmentation of the words in the text "solve one problem" yields the text "solve two problems"; data augmentation of the phrases in the text "solve one problem" yields the text "solve one kind of problem"; and data augmentation of the sentences in the text "solve one problem" yields the text "solve one missing problem".

[0042] Through the above embodiments, when the target neural network is a text recognition model, it is possible to obtain different second training features corresponding to the first training feature, that is, to obtain similar features corresponding to the first training feature, and then to obtain text features such as synonyms, synonym phrases and synonym sentences.

[0043] Optionally, in this embodiment, the training loss is determined based on the first result and the second result, including but not limited to: when the first training feature and the second training feature are different words, the first result and the second result are mutually exclusive to obtain the word training loss.

[0044] Specifically, in this embodiment, the text recognition model is the BERT model, such as... Figure 2 As shown, the first training feature is "solving one problem". By performing word-based data augmentation (mask-augment) on the first training feature, the second training feature "solving three problems" is obtained. For example... Figure 3 As shown, the second training feature, which is enhanced by masking and augmenting the predicted data through the pre-trained BERT model, and the original first training feature that has been masked and augmented are mutually exclusive and repeled to obtain the word training loss.

[0045] It should be noted that, in this embodiment, mutual exclusion means that the distance between two vector representations increases as training progresses. Specifically, mutual exclusion is the dot product operation on vectors or matrices.

[0046] Through the above embodiments, when the first training feature and the second training feature are different words, the first result and the second result are mutually exclusive to obtain the word training loss, so that the target neural network incorporates an optimization objective when processing synonyms, thereby improving the accuracy of word classification.

[0047] Optionally, in this embodiment, the training loss is determined based on the first result and the second result, including but not limited to: when the first training feature and the second training feature are different phrases, the first result and the second result are mutually exclusive to obtain the phrase training loss.

[0048] Specifically, in this embodiment, the text recognition model is the BERT model, such as... Figure 4 As shown, the first training feature is "solving a problem". Phrase-based data augmentation (mask-augment) is applied to the first training feature to obtain the second training feature, "solving a type of problem". For example... Figure 5 As shown, the second training feature predicted by the pre-trained BERT model through masking and the original first training feature that was masked are mutually exclusive repeld to obtain the phrase training loss.

[0049] Through the above embodiments, when the first training feature and the second training feature are different phrases, the first result and the second result are mutually exclusive to obtain the phrase training loss, so that the target neural network incorporates an optimization objective when processing synonymous phrases, thereby improving the accuracy of the target neural network in phrase classification.

[0050] Optionally, in this embodiment, the training loss is determined based on the first result and the second result, including but not limited to: when the first training feature and the second training feature are different sentences, the first result and the second result are mutually exclusive to obtain the sentence training loss.

[0051] Specifically, in this embodiment, the text recognition model is the BERT model, and the first training feature is "solving a problem". The first training feature is subjected to data augmentation masking corresponding to the sentence to obtain the second training feature "solving a missing problem". The second training feature predicted by the pre-trained BERT model and the original masked first training feature are mutually exclusive repeled to obtain the sentence training loss.

[0052] Through the above embodiments, when the first training feature and the second training feature are different phrases, the first result and the second result are mutually exclusive to obtain the sentence training loss, so that the target neural network incorporates an optimization objective when processing synonymous sentences, thereby improving the accuracy of the target neural network in classifying sentences.

[0053] Optionally, in this embodiment, the training loss is determined based on the first result and the second result, including but not limited to: inputting the second result into a preset decoder to obtain the first decoding loss corresponding to the preset decoder.

[0054] In a specific application scenario, in this embodiment, during the processing of the first feature or the second training feature by the target neural network, in one example, after encoding the first training feature, the corresponding feature vector is decoded and input into a preset decoder. The decoder generates the original input feature, i.e., a feature similar to the first training feature, based on the input feature vector and the first decoding loss.

[0055] Specifically, after applying a data augmentation mask token to the training sentence, it is fed into the decoder. Based on the first decoding loss, a result sentence similar to the original input training sentence is generated to ensure the stability of the meaning of the result sentence obtained from the training sentence.

[0056] Optionally, in this embodiment, determining the training loss based on the first result and the second result may include, but is not limited to, attracting the first result and the decoding loss to obtain the second decoding loss.

[0057] Specifically, in the example above, after encoding the first training feature, the corresponding feature vector is decoded and input into a preset decoder. The decoder generates the original input feature, i.e., the output feature, which is similar to the first training feature, based on the input feature vector and the first decoding loss. A second decoding loss is obtained by performing an attraction operation between the output feature and the first training feature.

[0058] It should be noted that attraction refers to the reduction of the distance between two feature vectors as training progresses, which can be obtained through convolution dot product.

[0059] Specifically, such as Figure 6 As shown, after data augmentation mask tokens are applied to the training sentences, they are fed into the decoder to generate the original complete sentences. The generated sentences and the original sentences are then subjected to an attraction test to obtain the second decoding loss.

[0060] Through the embodiments of the present invention, a first training feature and a second training feature are obtained, wherein the second training feature and the first training feature are similar features; a training loss is determined based on a first result and a second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature; the target neural network is trained according to the training loss, thereby achieving the purpose of adding a corresponding optimization objective to the training process of similar features, thus realizing the technical effect of improving the accuracy of the target neural network in recognizing similar features, and thus solving the technical problem that the text recognition model has low prediction accuracy because the text recognition model in related technologies cannot recognize similar features.

[0061] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0063] Example 2

[0064] According to embodiments of the present invention, a neural network training apparatus for implementing the above-described neural network training method is also provided, such as... Figure 7 As shown, the device includes:

[0065] 1) Acquisition unit 70, used to acquire a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature;

[0066] 2) Determining unit 72, used to determine training loss based on the first result and the second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature;

[0067] 3) Training unit 74, used to train the target neural network according to the training loss.

[0068] Optionally, specific examples in this embodiment can refer to the examples described in Embodiments 1 and 2 above, and will not be repeated here.

[0069] Example 3

[0070] According to an embodiment of the present invention, an electronic device is also provided, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the neural network training method as described in Embodiment 1.

[0071] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0072] S1, obtain a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature;

[0073] S2, determine the training loss based on the first result and the second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature;

[0074] S3, train the target neural network according to the training loss.

[0075] Optionally, specific examples in this embodiment can refer to the examples described in Embodiment 1 above, and will not be repeated here.

[0076] Example 4

[0077] Embodiments of the present invention also provide a readable storage medium. Optionally, in this embodiment, the readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the neural network training method as described in Embodiment 1.

[0078] Optionally, in this embodiment, the readable storage medium is configured to store program code for performing the following steps:

[0079] S1, obtain a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature;

[0080] S2, determine the training loss based on the first result and the second result, wherein the first result is obtained by the target neural network predicting the first training feature, and the second result is obtained by the target neural network predicting the second training feature;

[0081] S3, train the target neural network according to the training loss.

[0082] Optionally, the storage medium is also configured to store program code for performing the steps included in the method of Embodiment 1 above, which will not be described again in this embodiment.

[0083] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0084] Optionally, specific examples in this embodiment can refer to the examples described in Embodiments 1 and 2 above, and will not be repeated here.

[0085] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0086] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0087] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A neural network training method, characterized by, The method comprises the following steps: obtaining a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature; determining a training loss according to a first result and a second result, wherein the first result is obtained by predicting the first training feature by a target neural network, and the second result is obtained by predicting the second training feature by the target neural network; training the target neural network according to the training loss; the target neural network is a text recognition model, and the first training feature and the second training feature are texts, wherein the first training feature and the second training feature are obtained by at least one of the following methods: performing data enhancement on a word in the first training feature to obtain the second training feature; performing data enhancement on a phrase in the first training feature to obtain the second training feature; performing data enhancement on a sentence in the first training feature to obtain the second training feature; determining the training loss according to the first result and the second result comprises: in the case that the first training feature and the second training feature are different words, performing mutual exclusion on the first result and the second result to obtain a word training loss, and the mutual exclusion is specifically a dot product operation on a vector or a matrix.

2. The method of claim 1, wherein, determining the training loss according to the first result and the second result comprises: in the case that the first training feature and the second training feature are different phrases, performing mutual exclusion on the first result and the second result to obtain a phrase training loss.

3. The method of claim 1, wherein, determining the training loss according to the first result and the second result comprises: in the case that the first training feature and the second training feature are different sentences, performing mutual exclusion on the first result and the second result to obtain a sentence training loss.

4. The method of claim 1, wherein, determining the training loss according to the first result and the second result comprises: inputting the second result into a preset decoder to obtain a first decoding loss corresponding to the preset decoder.

5. The method of claim 4, wherein, determining the training loss according to the first result and the second result further comprises: attracting the first result and the first decoding loss to obtain a second decoding loss, and the attracting is specifically obtained by convolution dot multiplication.

6. A neural network training apparatus characterized by comprising: The method comprises the following steps: an obtaining unit is configured to obtain a first training feature and a second training feature, wherein the second training feature is a similar feature to the first training feature; a determining unit is configured to determine a training loss according to a first result and a second result, wherein the first result is obtained by predicting the first training feature by a target neural network, and the second result is obtained by predicting the second training feature by the target neural network; a training unit is configured to train the target neural network according to the training loss; the target neural network is a text recognition model, and the first training feature and the second training feature are texts, wherein the first training feature and the second training feature are obtained by at least one of the following methods: performing data enhancement on a word in the first training feature to obtain the second training feature; performing data augmentation on a phrase in the first training feature to obtain the second training feature; performing data augmentation on a sentence in the first training feature to obtain the second training feature; determining a training loss according to the first result and the second result, including: in a case where the first training feature and the second training feature are different in words, performing mutual exclusion on the first result and the second result to obtain a word training loss, and the mutual exclusion is specifically a dot product operation on vectors or matrices.

7. An electronic device, comprising: The processor, the memory, and the program or instructions stored on the memory and executable on the processor, when executed by the processor, implement the steps of the neural network training method according to any one of claims 1-5.

8. A readable storage medium, characterized by, The program or instructions are stored on the readable storage medium, and when executed by the processor, implement the steps of the neural network training method according to any one of claims 1-5.

Images