Method and device for determining training data, training method, equipment, medium

Abstract

The present application relates to the field of data screening technology, and discloses a method and device for determining training data, a training method, equipment, and medium. The method for determining training data includes: obtaining a reference corpus in the field to which the model to be trained belongs, and sorting the first words in the order of the first word frequency inverse document frequency value from high to low, and screening the first words located in the front preset digit as the first keyword set; obtaining the pending corpus in the field to which the model to be trained belongs, and sorting the second words in the order of the second word frequency inverse document frequency value from high to low, and screening the second words located in the front preset digit as the second keyword set; when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than the preset number, determining the pending corpus as the training data for training the model to be trained. In this way, corpus with similar content to the reference corpus can be automatically screened out, thereby improving the efficiency of screening corpus.

Description

Technical Field

[0001] The present application relates to the technical field of data screening, and in particular to a method and device for determining training data, and a training method, equipment, and medium. Background Art

[0002] At present, with the continuous development of artificial intelligence technology, more and more users are using big models in their work and life, such as chatgpt. Big models have shown strong capabilities and potential in processing complex tasks and massive data. Before using the big model, it is necessary to pre-train the big model with massive text data. Relevant technical personnel have found that the strength of the big model's capabilities is directly affected by the training data. High-quality training data can help the big model better understand the task and improve the prediction accuracy, while low-quality or insufficient training data may cause the prediction accuracy of the big model to decrease. In related technologies, technical personnel usually manually screen the corpus used for big model training, but the efficiency of screening the corpus in this way is low.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present application provide a method and apparatus for determining training data, a training method, equipment, and medium to improve the efficiency of screening corpus.

[0006] An embodiment of the present application provides a method for determining training data, including: S11. obtaining a reference corpus in the field to which the model to be trained belongs, and counting the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, sorting the first words in descending order according to the first term frequency inverse document frequency values, and filtering the first words located in the front preset number of digits as the first keyword set; S12. obtaining a pending corpus in the field to which the model to be trained belongs, and counting the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus; sorting the second words in descending order according to the second term frequency inverse document frequency values, and filtering the second words located in the front preset number of digits as the second keyword set; the number of sentences in the pending corpus is greater than the number of sentences in the reference corpus; S13. when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than the preset number, determining that the pending corpus is training data for training the model to be trained.

[0007] In the above implementation, by comparing the word frequency inverse document frequency values ​​of the words in the reference corpus and the pending corpus, the words with the highest frequency in the pending corpus and the reference corpus can be reflected. By comparing the keywords in the reference corpus with the frequency of occurrence in the first preset digit and the keywords in the pending corpus, it can be reflected whether the content between the pending corpus and the reference corpus is similar, so as to automatically screen out the corpus with similar content to the reference corpus, and improve the efficiency of screening the corpus. At the same time, by screening through the word frequency inverse document frequency value, the pending corpus and the reference corpus are not only similar, but also the word order of the pending corpus can be more varied, which is more conducive to model training.

[0008] Furthermore, the first term frequency inverse document frequency value corresponding to each first word in the reference corpus is counted, including: segmenting the reference corpus to obtain the first words; and using a preset term frequency inverse document frequency algorithm to calculate the first term frequency inverse document frequency value for each of the first words.

[0009] In the above embodiment, by segmenting the reference corpus, the continuous text in the reference corpus can be divided into independent words or phrases, so as to more accurately calculate the frequency and distribution of each word in the reference corpus, so as to calculate the first term frequency inverse document frequency value of each first word.

[0010] Furthermore, the second term frequency inverse document frequency value corresponding to each second word in the pending corpus is counted, including: segmenting the pending corpus to obtain the second words; and calculating the second term frequency inverse document frequency value for each second word using a preset term frequency inverse document frequency algorithm.

[0011] In the above implementation, by performing word segmentation on the pending corpus, the continuous text in the pending corpus can be divided into independent words or phrases, so as to more accurately calculate the frequency and distribution of each word in the pending corpus, so as to calculate the second word frequency inverse document frequency value of each second word.

[0012] Furthermore, the method for determining training data also includes: determining the similarity between a first term frequency inverse document frequency value corresponding to the first keyword set and a second term frequency inverse document frequency value corresponding to the second keyword set; when the number of repetitions is greater than or equal to a preset number and the similarity is greater than or equal to a preset similarity, determining that the pending corpus is training data for training the model to be trained; when the number of repetitions is less than a preset number, or the similarity is less than a preset similarity, discarding the pending corpus and not using it for training the model to be trained.

[0013] In the above implementation, by determining the similarity between the first term frequency inverse document frequency value corresponding to the first keyword set and the second term frequency inverse document frequency value corresponding to the second keyword set, it is possible to more accurately reflect whether the occurrence probabilities of keywords in the two keyword sets are consistent, and thus to select a pending corpus that is more similar to the reference corpus.

[0014] Furthermore, determining the similarity between a first term frequency inverse document frequency value corresponding to the first keyword set and a second term frequency inverse document frequency value corresponding to the second keyword set includes: constructing a first frequency sequence for the first term frequency inverse document frequency value corresponding to the first word in the first keyword set according to the order of the number of digits of the first word in the first keyword set; constructing a second frequency sequence for the second term frequency inverse document frequency value corresponding to the second word in the order of the number of digits of the second word in the second keyword set; and calculating the similarity between the first frequency sequence and the second frequency sequence.

[0015] In the above implementation, a first frequency sequence is constructed according to the order of the first word, and a second frequency sequence is constructed according to the order of the digits of the second word, and the similarity between the two sequences is calculated. The calculated similarity can reflect the similarity between the same words in the two keyword sets. And by constructing a frequency sequence, the similarity between multiple words can be reflected at one time, which is more convenient.

[0016] Furthermore, the method for determining training data also includes: obtaining the number of corpora of the training data; when the number of corpora is less than a preset number of corpora, looping steps S12 to S13 until the number of corpora of the training data is greater than or equal to the preset number of corpora.

[0017] In the above implementation, by repeatedly obtaining the pending corpus and determining whether the pending corpus is used as training data for training the model to be trained, until the amount of the obtained corpus is greater than the preset amount of corpus, sufficient corpus for training the model to be trained can be automatically obtained.

[0018] An embodiment of the present application provides a method for training a model, including: obtaining a corpus to be trained in the field to which the model to be trained belongs; determining the pending corpus and reference corpus used to train the model to be trained by the corpus to be trained obtained by the above method; and training the model to be trained using the corpus to be trained.

[0019] In the above implementation, the efficiency of training the model to be trained can be improved by training the model to be trained using the automatically acquired corpus to be trained.

[0020] An embodiment of the present application provides a device for determining training data, including: a first acquisition module, used to acquire reference corpus in the field to which a model to be trained belongs, and to count first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, sort the first words in descending order according to the first term frequency inverse document frequency values, and filter the first words located in the front preset number of digits as a first keyword set; a second acquisition module, used to acquire pending corpus in the field to which the model to be trained belongs, and to count second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus, sort the second words in descending order according to the second term frequency inverse document frequency values, and filter the second words located in the front preset number of digits as a second keyword set; the number of sentences in the pending corpus is greater than the number of sentences in the reference corpus; a screening module, used to determine that the pending corpus is training data for training the model to be trained when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than a preset number.

[0021] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned method for determining training data or method for training a model.

[0022] An embodiment of the present application provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions enable the processor to implement the above-mentioned method for determining training data or method for training a model.

[0023] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] One or more embodiments are exemplarily described by corresponding drawings, and these exemplary descriptions and drawings do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are considered similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0025] Figure 1 is a schematic diagram of a method for determining training data provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of a method for training a model provided in an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a device for determining training data provided in an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a device for training a model provided in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of an electronic device provided in an embodiment of the present application.

[0030] Reference numerals:

[0031] 1: first acquisition module; 2: second acquisition module; 3: screening module; 4: third acquisition module; 5: training module; 6: processor; 7: memory; 8: communication interface; 9: bus. DETAILED DESCRIPTION

[0032] In order to be able to understand the features and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0033] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present application described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0034] Unless otherwise stated, the term "plurality" means two or more.

[0035] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0036] Embodiment 1

[0037] The present application provides a method for determining training data in an embodiment. Figure 1 As shown, Figure 1 This is a flow chart of a method for determining training data provided in an embodiment of the present application, the method comprising:

[0038] Step S11, obtain the reference corpus of the field to which the model to be trained belongs, and count the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, sort the first words in descending order according to the first term frequency inverse document frequency values, and select the first words in the front preset position as the first keyword set.

[0039] The field can refer to a certain professional direction, such as chemistry, preschool education, law, medicine, etc. The field can also be a subdivision of a certain professional direction, for example, the subdivisions of the medical field include anesthesia, medical imaging, dentistry, etc.

[0040] A reference corpus may include one or more corpora, and a corpus may include one or more sentences.

[0041] For example, a single piece of corpus can come from a book, a website, a knowledge base, a multimedia material, etc. The presentation form of a single piece of corpus can be an article, a web page, a recording, a speech, etc. Among them, in order to facilitate processing, non-text corpus can be converted into text form, for example: converting a recording into text.

[0042] In some embodiments, the reference corpus may be obtained from a set source range or from a set first database.

[0043] The engineer may set a source of relatively high-quality corpus as the set source range based on experience, such as a professional book selected by a user. For example, a preset number of corpus may be randomly selected from the set source range as the reference corpus.

[0044] The first database stores corpora in the field to which the model to be trained belongs that have been pre-screened by the user. Exemplarily, a preset number of corpora can be randomly selected from the first database as reference corpora.

[0045] Optionally, the number of corpora in the reference corpus may be set according to the experience of the engineer, that is, the preset number may be set according to the experience of the engineer, for example, 3 corpora, 5 corpora, etc.

[0046] In some embodiments, the first term frequency inverse document frequency value corresponding to each first word in the reference corpus is statistically calculated by: segmenting the reference corpus to obtain the first words; and calculating the first term frequency inverse document frequency value for each first word using a preset term frequency inverse document frequency algorithm.

[0047] In an optional manner of the above embodiment, the word segmentation of the reference corpus may be performed by: using a preset word segmentation model or a preset word segmentation algorithm to segment the reference corpus.

[0048] Among them, the preset word segmentation algorithms include: forward maximum matching method, reverse maximum matching method, bidirectional maximum matching method, etc.

[0049] The preset word segmentation model can be obtained in the following way: inputting sample corpus with word segmentation labels into a preset training model for training to obtain the word segmentation model.

[0050] Exemplary, preset training models, such as: hidden Markov model, recurrent neural network, etc.

[0051] In another optional manner of the above embodiment, the word segmentation of the reference corpus may be performed by using a preset word segmentation tool.

[0052] Among them, word segmentation tools, such as: jieba (jieba word segmentation), NLTK (Natural Language Toolkit), etc.

[0053] Exemplarily, the reference corpus is sent to a preset word segmentation tool for word segmentation, and the word segmentation result returned by the word segmentation tool is obtained.

[0054] In another optional manner of the above embodiment, the reference corpus includes multiple sentences. Segmenting the reference corpus may include: segmenting the reference corpus according to punctuation marks to obtain multiple first sentences; and segmenting each first sentence.

[0055] Among them, the method of performing word segmentation processing on the single first sentence is similar to the method of performing word segmentation processing on the above, and will not be repeated here.

[0056] In some embodiments, after the reference corpus is segmented to obtain a first segmentation result, stop words in the first segmentation result may be removed, and the first segmentation result without the stop words is determined as the first word. The first segmentation result includes: a plurality of words.

[0057] Exemplarily, stop words can be meaningless words, such as modal particles. Exemplarily, modal particles, such as de, ne, ah, etc.

[0058] In an optional manner of the above embodiment, the first term frequency inverse document frequency value is calculated for each first word using a preset term frequency inverse document frequency algorithm, which can be: for each first word: determine the term frequency of the first word in the reference corpus; determine the reverse document frequency of the first word in the reference corpus; calculate tfidf 1 = tf1 × idf1, and obtain the first term frequency inverse document frequency value of the first word. Wherein, tfidf 1 is the first term frequency inverse document frequency value of the first word, tf1 is the term frequency of the first word in the reference corpus, and idf1 is the reverse document frequency of the first word in the reference corpus.

[0059] In the above optional method, the word frequency of a single first word in the reference corpus can be determined in the following manner: obtain the number of occurrences of the first word and the total number of words in the reference corpus, calculate the number of occurrences of the first word divided by the total number of words in the reference corpus, and obtain the word frequency of the first word in the reference corpus. The total number of words in the reference corpus is the total number of words obtained after the reference corpus is segmented. The number of occurrences of the first word is the number of times the first word appears in the reference corpus.

[0060] Exemplarily, assume that the reference corpus consists of 5 first words, for example: A, A, B, A, C. The total number of words in the reference corpus is 5, the first word "A" appears 3 times, the first word "B" appears 1 time, and the first word "C" appears 1 time.

[0061] In the above optional method, the reverse document frequency of a single first word in the reference corpus can be determined in the following manner: obtain the number of corpora in the reference corpus and the number of corpora in which the first word exists in the reference corpus; calculate idf1=log2(n / k) to obtain the reverse document frequency of the first word in the reference corpus. Where n is the number of corpora in the reference corpus, and k is the number of corpora in which the first word exists in the reference corpus. The number of corpora in the reference corpus is the total number of corpora included in the reference corpus.

[0062] Optionally, the number of pre-set digits may be set based on the experience of engineers, for example, the number of pre-set digits may be set to 100, or the number of pre-set digits may be set to 1000.

[0063] Exemplarily, assuming that the first word is "hello", there are a total of 8 words after the reference corpus is segmented. Among them, the first word "hello" appears 2 times, and the word frequency of the first word "hello" in the reference corpus is 2 / 8. Assuming that the entire reference corpus includes 3 corpora, and the first word "hello" appears in 2 of them, the inverse document frequency of the first word "hello" in the reference corpus is: idf1=log2(3 / 2). Calculate tfidf 1=tf1×idf1, and obtain the first word "hello" The first term frequency inverse document frequency value is 2 / 8×log2(3 / 2). After obtaining the first term frequency inverse document frequency value of each first word, sort the first words in descending order according to the first term frequency inverse document frequency value, and select the first words whose first term frequency inverse document frequency values ​​are in the top 1000 to form the first keyword set.

[0064] Step S12, obtain the pending corpus in the field to which the model to be trained belongs, and count the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus, sort the second words in descending order according to the second term frequency inverse document frequency values, and select the second words in the front preset number as the second keyword set.

[0065] Among them, the number of sentences in the pending corpus is greater than the number of sentences in the reference corpus.

[0066] In some embodiments, the pending corpus may be corpus obtained by searching the network and related to the field to which the model to be trained belongs, or may be corpus collected in advance by a user and needs to be screened.

[0067] In some embodiments, the second term frequency inverse document frequency value corresponding to each second word in the pending corpus is counted, which can be: segmenting the pending corpus to obtain the second words; and using a preset term frequency inverse document frequency algorithm to calculate the second term frequency inverse document frequency value for each second word.

[0068] In an optional manner of the above embodiment, the word segmentation of the pending corpus may be performed by using a preset word segmentation model or a preset word segmentation algorithm.

[0069] In another optional manner of the above embodiment, the word segmentation of the subject corpus may be performed by using a preset word segmentation tool.

[0070] In another optional manner of the above embodiment, the pending corpus includes multiple sentences. The pending corpus is segmented by: segmenting the pending corpus according to punctuation marks to obtain multiple second sentences; and segmenting each second sentence.

[0071] The method of performing word segmentation processing on the single second sentence is similar to the method of performing word segmentation processing on the above sentence, and will not be described in detail here.

[0072] In some embodiments, after segmenting the subject corpus to be determined, a second segmentation result is obtained, and stop words in the second segmentation result may be removed, and the second segmentation result without the stop words is determined as the second word. The second segmentation result includes: a plurality of words.

[0073] In an optional manner of the above embodiment, the second term frequency inverse document frequency value is calculated for each second word using a preset term frequency inverse document frequency algorithm, which can be: for each second word: determine the term frequency of the second word in the pending corpus; determine the reverse document frequency of the second word in the pending corpus; calculate tfidf 2 = tf2 × idf2 to obtain the second term frequency inverse document frequency value of the second word. Wherein, tfidf 2 is the second term frequency inverse document frequency value of the second word, tf2 is the term frequency of the second word in the pending corpus, and idf2 is the reverse document frequency of the second word in the pending corpus.

[0074] Among them, determining the word frequency of the second word in the pending corpus is similar to determining the word frequency of the first word in the reference corpus; determining the reverse file frequency of the second word in the pending corpus is similar to determining the reverse file frequency of the second word in the reference corpus, which will not be repeated here.

[0075] Step S13, when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than a preset number, determining the to-be-determined corpus as training data for training the to-be-trained model.

[0076] In some embodiments, when the number of repetitions between a first word in the first keyword set and a second word in the second keyword set is less than a preset number, the pending corpus is discarded.

[0077] For example, suppose there are 5 first words in the first keyword set, such as A, B, C, D and E. Suppose there are 5 first words in the second keyword set, such as A, B, F, D and G. Then the number of repetitions between the first words in the first keyword set and the second words in the second keyword set is 3.

[0078] In some embodiments, the method for determining training data also includes: determining the similarity between a first term frequency inverse document frequency value corresponding to a first keyword set and a second term frequency inverse document frequency value corresponding to a second keyword set; when the number of repetitions is greater than or equal to a preset number and the similarity is greater than or equal to a preset similarity, determining the pending corpus as training data for training the model to be trained; when the number of repetitions is less than a preset number, or the similarity is less than a preset similarity, discarding the pending corpus and not using it for training the model to be trained.

[0079] In the above embodiment, determining the similarity between the first term frequency inverse document frequency value corresponding to the first keyword set and the second term frequency inverse document frequency value corresponding to the second keyword set can be: constructing a first frequency sequence for the first term frequency inverse document frequency value corresponding to the first word according to the order of the number of digits of the first word in the first keyword set; constructing a second frequency sequence for the second term frequency inverse document frequency value corresponding to the second word according to the order of the number of digits of the second word in the second keyword set; and calculating the similarity between the first frequency sequence and the second frequency sequence.

[0080] The similarity between the first frequency sequence and the second frequency sequence may be calculated using cosine similarity, Pearson correlation coefficient, Euclidean distance, and the like.

[0081] In some embodiments, the method for determining training data further includes: obtaining the amount of corpus of the training data; when the amount of corpus is less than a preset amount of corpus, looping steps S12 to S13 until the amount of corpus of the training data is greater than or equal to the preset amount of corpus.

[0082] In some embodiments, the method for determining training data further includes: storing the training data used to train the model to be trained in a preset storage space. In this way, the screened training data is stored to facilitate subsequent use of the training data for model training.

[0083] Exemplarily, assuming that the field to which the model to be trained belongs is the field of law, the electronic device can obtain 3 articles from the set legal professional books as reference corpus. The 3 articles are processed by sentence division according to punctuation marks to obtain multiple first sentences. All the first sentences are processed by word segmentation, and the words without stop words are used as the first words. The first word frequency inverse document frequency value is calculated for each first word by using the preset word frequency inverse document frequency algorithm, and the relationship between the first word and the first word frequency inverse document frequency value can be obtained. The first words are sorted in order from high to low according to the first word frequency inverse document frequency value, and the first words in the top n are selected as the first keyword set. Then the electronic device searches the network to obtain 20 articles related to the field to which the model to be trained belongs as the pending corpus. The 20 articles are processed by sentence division according to punctuation marks to obtain multiple second sentences. All the second sentences are processed by word segmentation, and the words without stop words are used as the second words. The second word frequency inverse document frequency value is calculated for each second word by using the preset word frequency inverse document frequency algorithm, and the relationship between the second word and the second word frequency inverse document frequency value can be obtained. Sort the second words in descending order of the second word frequency inverse document frequency value, and select the second words in the top n as the second keyword set. When the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than a preset number, construct a first frequency sequence for the first word frequency inverse document frequency value corresponding to the first word in the first keyword set according to the order of the digits of the first word in the first keyword set, and construct a second frequency sequence for the second word frequency inverse document frequency value corresponding to the second word in the order of the digits of the second word in the second keyword set. Assuming that the first frequency sequence is [A1, A2..., An] and the second frequency sequence is [B1, B2..., Bn], through cosine similarity Calculation is performed to obtain the cosine between the first frequency sequence and the second frequency sequence. When the value of cosθ is greater than or equal to the preset similarity, the undetermined corpus is determined as training data for training the model to be trained. When the value of cosθ is lower than the preset similarity, the undetermined corpus is discarded.

[0084] Embodiment 2

[0085] The present application provides a method for training a model. Figure 2 As shown, Figure 2 A flow chart of a method for training a model provided in an embodiment of the present application includes:

[0086] Step S21, obtaining the corpus to be trained in the field to which the model to be trained belongs.

[0087] The corpus to be trained is the undetermined corpus and reference corpus obtained by the method of the first embodiment and used to train the model to be trained.

[0088] Step S22: training the model to be trained using the corpus to be trained.

[0089] For example, assuming that the field to which the model to be trained belongs is the field of network security, a model capable of identifying vulnerabilities can be obtained by collecting training corpus related to vulnerabilities to train the model to be trained.

[0090] As another example, assuming that the field to which the model to be trained belongs is the field of law, a model capable of identifying legal clauses related to an event can be obtained by collecting legal-related training corpus to be trained for training the model to be trained.

[0091] As another example, assuming that the field to which the model to be trained belongs is the field of plants, a model capable of identifying plant types can be obtained by collecting plant-related training corpus to be trained for the model to be trained.

[0092] Embodiment 3

[0093] Based on the same inventive concept, the embodiment of the present application provides a device for determining training data, such as Figure 3 As shown, the device for determining training data includes: a first acquisition module 1, a second acquisition module 2 and a screening module 3. Among them, the first acquisition module 1 is used to obtain the reference corpus of the field to which the model to be trained belongs, and to count the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, sort the first words in the order of the first term frequency inverse document frequency values ​​from high to low, and screen the first words located in the front preset digit as the first keyword set; the second acquisition module 2 is used to obtain the pending corpus of the field to which the model to be trained belongs, and to count the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus; sort the second words in the order of the second term frequency inverse document frequency values ​​from high to low, and screen the second words located in the front preset digit as the second keyword set; the number of sentences in the pending corpus is greater than the number of sentences in the reference corpus; the screening module 3 is used to determine that the pending corpus is the training data for training the model to be trained when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than the preset number.

[0094] In some embodiments, the first acquisition module 1 can count the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus in the following way: segment the reference corpus to obtain the first words; and use a preset term frequency inverse document frequency algorithm to calculate the first term frequency inverse document frequency value for each first word.

[0095] In some embodiments, the second acquisition module 2 can count the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus in the following manner: segment the pending corpus to obtain the second words; and use a preset term frequency inverse document frequency algorithm to calculate the second term frequency inverse document frequency value for each second word.

[0096] In some embodiments, the screening module 3 is also used to determine the similarity between the first term frequency inverse document frequency value corresponding to the first keyword set and the second term frequency inverse document frequency value corresponding to the second keyword set; when the number of repetitions is greater than or equal to a preset number, and the similarity is greater than or equal to a preset similarity, the pending corpus is determined as training data for training the model to be trained; when the number of repetitions is less than a preset number, or the similarity is less than a preset similarity, the pending corpus is discarded and not used for training the model to be trained.

[0097] In some embodiments, the screening module 3 is used to determine the similarity between the first term frequency inverse document frequency value corresponding to the first keyword set and the second term frequency inverse document frequency value corresponding to the second keyword set in the following manner: construct a first frequency sequence for the first term frequency inverse document frequency value corresponding to the first word according to the order of the number of digits of the first word in the first keyword set; construct a second frequency sequence for the second term frequency inverse document frequency value corresponding to the second word according to the order of the number of digits of the second word in the second keyword set; and calculate the similarity between the first frequency sequence and the second frequency sequence.

[0098] In some embodiments, the screening module 3 is further used to obtain the number of corpora of the training data; when the number of corpora is less than the preset number of corpora, steps S12 to S13 are looped until the number of corpora of the training data is greater than or equal to the preset number of corpora.

[0099] It can be understood that the various embodiments described in the first embodiment are also applicable to the second embodiment if there is no conflict. Based on the consideration of brevity of description, they will not be repeated here.

[0100] Embodiment 4

[0101] Based on the same inventive concept, the present application embodiment provides a device for training a model, such as Figure 4 As shown, the device for training the model includes: a third acquisition module 4 and a training module 5. The third acquisition module 4 is used to acquire the training corpus of the field to which the model to be trained belongs. The training module 5 is used to train the model to be trained using the training corpus.

[0102] Combination Figure 5As shown, an embodiment of the present application provides an electronic device, including a processor 6 and a memory 7. Optionally, the device may also include a communication interface 8 and a bus 9. The processor 6, the communication interface 8, and the memory 7 may communicate with each other through the bus 9. The communication interface 8 may be used for information transmission. The processor 6 may call the logic instructions in the memory 7 to execute the method for determining training data in the above embodiment.

[0103] In addition, the logic instructions in the above-mentioned memory 7 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0104] The memory 7 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present application. The processor 6 executes the program instructions / modules stored in the memory 7 to perform functional applications and data processing, that is, to implement the method for determining training data in the above embodiment.

[0105] The memory 7 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 7 may include a high-speed random access memory and may also include a non-volatile memory.

[0106] The electronic device may be a computer or a server, etc.

[0107] An embodiment of the present application provides a storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above-mentioned method for determining training data.

[0108] An embodiment of the present application provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned method for determining training data.

[0109] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0110] The technical solution of the embodiment of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium can be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., which can store program codes, or a transient storage medium.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0112] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application. At the same time, the above embodiments can be combined with each other to form new embodiments without conflict.

Claims

1. A method for determining training data, characterized in that: include: S11. Obtain reference corpus in the field to which the model to be trained belongs, and count the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, sort the first words in descending order according to the first term frequency inverse document frequency values, and select the first words located in the front preset position as the first keyword set; S12. Obtain a pending corpus in the field to which the model to be trained belongs, and count the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus, sort the second words in descending order according to the second term frequency inverse document frequency values, and select the second words located in the front preset position as the second keyword set; The number of sentences in the pending corpus is greater than the number of sentences in the reference corpus; S13. When the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than a preset number, determine the to-be-determined corpus as training data for training the to-be-trained model.

2. The method according to claim 1, characterized in that Counting the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, including: Performing word segmentation on the reference corpus to obtain the first word; A preset word frequency inverse document frequency algorithm is used to calculate a first word frequency inverse document frequency value for each of the first words.

3. The method according to claim 1, characterized in that Counting the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus, including: Performing word segmentation on the pending corpus to obtain the second word; A preset word frequency inverse document frequency algorithm is used to calculate a second word frequency inverse document frequency value for each of the second words.

4. The method according to claim 1, characterized in that The method further comprises: Determine a similarity between a first term frequency inverse document frequency value corresponding to the first keyword set and a second term frequency inverse document frequency value corresponding to the second keyword set; When the number of repetitions is greater than or equal to a preset number and the similarity is greater than or equal to a preset similarity, determining the undetermined corpus as training data for training the model to be trained; When the number of repetitions is less than a preset number, or the similarity is less than a preset similarity, the to-be-determined corpus is discarded and is not used for training the to-be-trained model.

5. The method according to claim 4, characterized in that Determining the similarity between a first term frequency inverse document frequency value corresponding to the first keyword set and a second term frequency inverse document frequency value corresponding to the second keyword set includes: Constructing a first frequency sequence based on the first word frequency inverse document frequency value corresponding to the first word in the first keyword set according to the order of the number of digits of the first word in the first keyword set; Constructing a second frequency sequence based on the second word frequency inverse document frequency values ​​corresponding to the second words according to the order of the digits of the second words in the second keyword set; The similarity between the first frequency sequence and the second frequency sequence is calculated.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining the corpus quantity of the training data; When the number of corpora is less than the preset number of corpora, steps S12 to S13 are repeated until the number of corpora of the training data is greater than or equal to the preset number of corpora.

7. A method for training a model, characterized in that: include: Obtain the training corpus in the field to which the model to be trained belongs; The to-be-trained corpus is obtained by the method according to any one of claims 1 to 6 to determine the undetermined corpus and the reference corpus used to train the to-be-trained model; The model to be trained is trained using the corpus to be trained.

8. A device for determining training data, characterized in that: include: A first acquisition module is used to acquire a reference corpus in the field to which the model to be trained belongs, and to count the first term frequency inverse document frequency values ​​corresponding to each first word in the reference corpus, to sort the first words in descending order according to the first term frequency inverse document frequency values, and to select the first words located in the front preset position as the first keyword set; A second acquisition module is used to acquire a pending corpus in the field to which the to-be-trained model belongs, and to count the second term frequency inverse document frequency values ​​corresponding to each second word in the pending corpus; The second words are sorted in descending order according to the second word frequency inverse document frequency value, and the second words located in the front preset position are selected as the second keyword set; The number of sentences in the pending corpus is greater than the number of sentences in the reference corpus; A screening module is used to determine that the pending corpus is training data for training the model to be trained when the number of repetitions between the first word in the first keyword set and the second word in the second keyword set is greater than a preset number.

9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method for determining training data as described in any one of claims 1 to 6 or the method for training a model as described in claim 7.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by the processor, the computer-executable instructions enable the processor to implement the method for determining training data as described in any one of claims 1 to 6 or the method for training a model as described in claim 7.

Images