Model training data processing method and device and electronic equipment

Abstract

The invention provides a model training data processing method and device and electronic equipment. The model training data processing method comprises the steps that word segmentation preprocessing is carried out on a training text in model training data to form a first word segmentation list, vocabulary de-duplication is carried out on the first word segmentation list to obtain a second word segmentation list, and the second word segmentation list comprises a plurality of non-repeated vocabularies to be processed; forming a text vector matrix of the training text according to a word vector matrix corresponding to the first word segmentation list; the similarity between the word vectors of the to-be-processed words and the text vector matrix is determined, K to-be-processed words with the maximum similarity are determined as K subject words of the training text, and K is larger than or equal to 1; and dividing the K training texts with the same subject terms into one group in the model training data, and performing duplicate removal on the training texts with the similarity of the text vector matrixes greater than a preset threshold value in the same group. The embodiment of the invention can improve the model training efficiency.

Description

Background Art

[0002] A large model (LM) refers to a machine learning model with large-scale parameters and complex computing structure. It has a huge number of parameters, can process massive amounts of data, and complete a variety of complex tasks. When a large model is mainly used to process language tasks, it is also called a large language model (LLM).

[0003] Large language models usually require massive amounts of training texts to train, and these training texts are all unlabeled. Therefore, there are a large number of long texts in the existing training datasets, and there are many nearly repeated examples and long repeated substrings between these long texts. Using such training datasets to train large language models is difficult to improve the perplexity of the model, and it is easy to cause the trained model to show memory (generated content is copied from the training data), resulting in privacy issues caused by the model memorizing its training data.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a model training data processing method, device and electronic device for deduplicating long texts in a training data set of a large language model to solve the problem of model memorizing training data and showing memory, while improving model training efficiency.

[0006] According to a first aspect of an embodiment of the present disclosure, a model training data processing method is provided, comprising: performing word segmentation preprocessing on training text in the model training data to form a first word segmentation list, performing vocabulary deduplication on the first word segmentation list to obtain a second word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating to-be-processed words; forming a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list; determining the similarity between the word vectors of the to-be-processed words and the text vector matrix, determining the K to-be-processed words with the largest similarity as K subject words of the training text, K≥1; grouping the training texts with the same K subject words in the model training data into a group, and deduplicating the training texts in the same group whose similarity of the text vector matrix is ​​greater than a preset threshold.

[0007] According to a second aspect of an embodiment of the present disclosure, a model training data processing device is provided, comprising: a word segmentation processing module, configured to perform word segmentation preprocessing on training texts in model training data to form a first word segmentation list, and to obtain a second word segmentation list after deduplication of vocabulary on the first word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating to-be-processed vocabulary; a text feature extraction module, configured to form a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list; a keyword extraction module, configured to determine the similarity between the word vectors of the to-be-processed vocabulary and the text vector matrix, and determine the K to-be-processed vocabulary with the largest similarity as K subject words of the training text, K≥1; and a text deduplication module, configured to group the training texts with the same K subject words in the model training data into one group, and to dedupe the training texts in the same group whose similarity of the text vector matrix is ​​greater than a preset threshold.

[0008] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0009] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the model training data processing method as described in any one of the above items is implemented.

[0010] The disclosed embodiment obtains a first word segmentation list based on word segmentation preprocessing of a training text, and forms a text vector matrix of the training text based on the first word segmentation list, thereby being able to better extract text features; by calculating the similarity between the word vectors of non-repeated to-be-processed vocabulary in the training text and the text vector matrix, K keywords that best express the semantics of the long text can be extracted; by grouping the training texts according to keywords and deduplicating the training texts within the groups, a coarse classification of massive training texts based on the K keywords that best express the semantics can be performed, thereby reducing the amount of calculation for text similarity comparison, improving data processing efficiency, and thereby improving model training efficiency, making the trained model have better perplexity, and preventing the model from showing memory of the training data.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0013] Figure 1 Detailed description of the invention The figure is a flowchart of a method for processing model training data in an exemplary embodiment of the present invention.

[0014] Figure 2 It is a sub-flowchart of step S3 in the exemplary embodiment of the present disclosure.

[0015] Figure 3 It is a sub-flowchart of step S3 in some other embodiments of the present disclosure.

[0016] Figure 4 is a flowchart of one embodiment of method 100 in an exemplary embodiment of the present disclosure.

[0017] Figure 5 It is a block diagram of a model training data processing device in an exemplary embodiment of the present disclosure.

[0018] Figure 6 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0020] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0021] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 Detailed description of the invention The figure is a flowchart of a method for processing model training data in an exemplary embodiment of the present invention.

[0023] refer to Figure 1 , the model training data processing method 100 may include:

[0024] Step S1, performing word segmentation preprocessing on the training text in the model training data to form a first word segmentation list, and removing duplicate words from the first word segmentation list to obtain a second word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating words to be processed;

[0025] Step S2, forming a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list;

[0026] Step S3, determining the similarity between the word vector of the to-be-processed vocabulary and the text vector matrix, and determining the K to-be-processed vocabulary with the largest similarity as the K subject words of the training text, K≥1;

[0027] Step S4, grouping the training texts with the same K subject words in the model training data into one group, and removing duplicates from the training texts in the same group whose similarity of the text vector matrix is ​​greater than a preset threshold.

[0028] The disclosed embodiment obtains a first word segmentation list based on word segmentation preprocessing of a training text, and forms a text vector matrix of the training text based on the first word segmentation list, thereby being able to better extract text features; by calculating the similarity between the word vectors of non-repeated to-be-processed vocabulary in the training text and the text vector matrix, K keywords that best express the semantics of the long text can be extracted; by grouping the training texts according to keywords and deduplicating the training texts within the groups, a coarse classification of massive training texts based on the K keywords that best express the semantics can be performed, thereby reducing the amount of calculation for text similarity comparison, improving data processing efficiency, and thereby improving model training efficiency, making the trained model have better perplexity, and preventing the model from showing memory of the training data.

[0029] Next, each step of the model training data processing method 100 is described in detail.

[0030] In step S1, the training text in the model training data is preprocessed with word segmentation to form a first word segmentation list, and the first word segmentation list is deduplicated to obtain a second word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating words to be processed.

[0031] The method of the disclosed embodiment can be applied to a large language model, and the model training data to be processed is the training data of the large language model, including text data collected for training the large language model, and these text data include long texts. After being processed by the method 100, the model training data to be processed can be used to train the large language model, which not only has a higher training efficiency, but also can overcome the problem of memory exhibited by the large language model.

[0032] In an exemplary embodiment, the model training data to be processed may include, but is not limited to, web page data, social media texts, news articles, academic papers, conversation records, comments, novels, etc. In some application scenarios, some models may focus on learning knowledge in a specific field, and the model training data to be processed may include, for example, professional texts in the fields of science and technology, law, or medicine. In addition, the model training data to be processed may also include a large number of general corpora for the model to learn the basic laws and patterns of language through training.

[0033] In step S1, the model training data to be processed can be first obtained. In this process, the file format, character style, etc. of data of different types and sources can be adjusted, and finally a plurality of training texts with fixed file formats that can be used as qualified data processing objects are formed. It should be noted that in order to ensure the effective deduplication of the training text, the text length of the training text obtained in step S1 can be limited to training texts with a length greater than a preset length, that is, they are all long texts, containing multiple sentences or paragraphs, rather than phrases and words.

[0034] Next, each training text is preprocessed by word segmentation.

[0035] In an exemplary embodiment, word segmentation preprocessing of a training text includes: word segmenting the training text to obtain a word segmentation result including a plurality of words, and then removing punctuation characters and stop words in the word segmentation result to obtain a first word segmentation list corresponding to each training text, wherein the punctuation characters and stop words can be determined according to a preset punctuation character data table and a stop word table.

[0036] It is understandable that the first word segmentation list may include multiple repeated words. Each word in the first word segmentation list is arranged according to the original word order in the training text.

[0037] After saving the first word segmentation list, the repeated words in the first word segmentation list can be removed to form a second word segmentation list. The words included in the second word segmentation list are non-repeated words in the training text A. The words in the second word segmentation list are called to-be-processed words. The to-be-processed words are all the word segmentations involved in the training text A, and there is no repetition between the to-be-processed words. Due to the past repetition processing, the order between the to-be-processed words may be inconsistent with the order in which the to-be-processed words appear in the training text A. It can be understood that if the number of words in the first word segmentation list is L and the number of to-be-processed words in the second word segmentation list is N, then N≤L, where N and L are both natural numbers.

[0038] After the second word segmentation list is saved, the first word segmentation list may be processed first.

[0039] In step S2, a text vector matrix of the training text is formed according to the word vector matrix corresponding to the first word segmentation list.

[0040] In the disclosed embodiment, a text vector matrix corresponding to the training text is formed according to the word segmentation result of the training text to better represent the features of the training text.

[0041] In an exemplary embodiment, first, according to the first word segmentation list corresponding to the training text A, a word vector for each word ci in the training text can be generated to form a word vector matrix MA corresponding to the first word segmentation list, where MA = [m1, m2, m3, ..., mL], where i is the sequence number of the word in the first word segmentation list of the training text A, mi is the word vector of the word ci, and L is the number of words in the first word segmentation list, which can be regarded as the number of words in the training text A.

[0042] Among them, word vector refers to a numerical vector converted from the vocabulary in the text. There are many methods for generating word vectors, including traditional methods based on matrix decomposition and learning methods based on neural networks. In the embodiment of the present disclosure, word vectors can be generated by Word2vec. Word2vec is a word vector generation model based on neural networks. Through training, each word is mapped to a continuous vector space, so that words with similar semantics are also adjacent in the vector space.

[0043] Word vectors usually use values ​​of multiple dimensions to express a word. The higher the dimension, the higher the expression accuracy. The dimensions of word vectors for different words are not necessarily the same. The dimension of the word vector matrix MA is L*D, where D is the dimension value of the word vector with the largest dimension among all word vectors, which is a natural number.

[0044] Next, the text vector matrix WA corresponding to the training text A can be determined based on the product of the word vector matrix MA and the transpose of the word vector matrix MA, as follows:

[0045] WA=MA×MA T (1)

[0046] The present disclosure sets a text vector matrix WA to perform feature description on the training text A.

[0047] In an exemplary embodiment, as in the calculation method of WA described above, the text vector matrix WA may be the text vector matrix WA of the training text A or a Gram matrix.

[0048] The text vector matrix WA can be obtained by multiplying the word vector matrix and its transpose so that each element in the matrix represents the inner product between the corresponding word vectors, and the inner product can measure the similarity between words. The higher the value, the closer the semantics of the two words are. Therefore, the text vector matrix WA can capture the similarity between words, identify semantically similar words through the values ​​of the elements of the text vector matrix WA, and intuitively represent the relationship between words, helping to understand the structure between words.

[0049] In addition, the text vector matrix WA can be used for dimensionality reduction to help identify important features, thereby reducing the complexity of the data. For example, a general word vector is 1*D dimensional, where the value of D is between 300 and 500, and the vocabulary L of a long text is usually greater than 1000, that is, in the dimension L*D of the word vector matrix, L>1000. If the word vector matrix is ​​used directly for calculation, the calculation dimension will increase as the length of the text increases, and the calculation speed is uncontrollable. When processing ultra-long texts, the processing speed will be greatly reduced. In the disclosed embodiment, by forming a text vector matrix WA of the training text, the dimension of the text vector matrix WA that characterizes the features of the training text A can be made D*D, the dimension is controllable, the calculation speed is controllable, and no matter how many words there are in the long text, the processing speed will not be greatly affected.

[0050] Therefore, subsequent processing of the training text based on the text vector matrix can not only improve the processing speed, but also accurately represent the semantics and word meanings of the training text.

[0051] Next, based on the second word segmentation list and the text vector matrix, an importance assessment is performed on each non-repeated word that appears in the training text A, ie, the word to be processed, so as to select the word that best expresses the semantics of the training text A.

[0052] In step S3, the similarity between the word vector of the to-be-processed vocabulary and the text vector matrix is ​​determined, and the K to-be-processed vocabulary with the largest similarity is determined as the K subject words of the training text, K≥1.

[0053] Let the word to be processed be aj, j is the serial number of the word to be processed in the second word segmentation list. Next, according to the word vector mi corresponding to at least one word ci corresponding to the word to be processed aj, the word vector mj of the word to be processed aj is obtained. Alternatively, the word vector of the word to be processed aj can be directly calculated to obtain the word vector mj.

[0054] Figure 2 It is a sub-flowchart of step S3 in the exemplary embodiment of the present disclosure.

[0055] refer to Figure 2 In some embodiments, step S3 may include:

[0056] Step S31, multiplying the word vector of the vocabulary to be processed by the text vector matrix to obtain an evaluation vector corresponding to the vocabulary to be processed;

[0057] Step S32, determining the similarity between the word vector of the vocabulary to be processed and the text vector matrix according to the evaluation vector.

[0058] In an exemplary embodiment, the word vector mj of the word to be processed aj may be multiplied by the text vector matrix WA to obtain an evaluation vector sj, and the similarity between the word to be processed aj and the text vector matrix WA may be determined based on the evaluation vector sj.

[0059] In some embodiments, the evaluation vector sj may be directly set to the similarity between the to-be-processed vocabulary aj and the text vector matrix WA.

[0060] The calculation process of the evaluation vector sj belongs to mapping the original word vector space to a new space. The evaluation vector sj is the projection of the word vector mj in the new space defined by the text vector matrix WA. It belongs to the word vector after a specific linear transformation and can express the position of the word vector mj in the semantic space WA. Since the text vector matrix WA encodes the semantic information of the text, the evaluation vector sj can be regarded as the representation of the word aj in the text semantic space, providing information about the semantic role of the word aj in the training text A.

[0061] The evaluation vector sj obtained in this way can reflect the contextual relationship of the word aj to be processed in the entire text, taking into account the similarity and correlation between words, and better capturing contextual information. Among them, each element in the evaluation vector si represents the weighted importance of the word ci in the training text A relative to other words. This weighting not only takes into account the characteristics of the word itself, but also its relationship with other words. It can more reliably express the importance of the word ci in the training text A, identify more critical words in a specific context, and enable the model to better understand the semantic structure of the text and enhance semantic understanding. In addition, by multiplying the word vector mj with the text vector matrix WA, the influence of some low-frequency words or irrelevant words on the importance evaluation can be reduced, thereby highlighting more representative words, achieving a noise reduction effect, and making text analysis and processing more efficient and accurate.

[0062] Next, the evaluation vectors sj corresponding to each word aj to be processed are compared to select K keywords that best express the semantics of the training text A.

[0063] In an exemplary embodiment, the K value can be set according to the text length of the training text (eg, the average length of the training text processed in this batch). The longer the text length, the larger the K value. It can also be set according to the needs of subsequent grouping.

[0064] When comparing multiple evaluation vectors sj, it can be achieved in many ways.

[0065] In some embodiments, the similarity between the to-be-processed vocabulary aj and the text vector matrix WA may be determined based on the sum of the components of the evaluation vector sj.

[0066] For example, suppose the evaluation vector sj = [a, b, c, d], then the similarity corresponding to the evaluation vector sj is a+b+c+d.

[0067] By directly summing up the elements of the evaluation vector and then comparing the sums between the evaluation vectors, the calculation speed can be improved. The evaluation vector sj is a 1*D-dimensional vector, and the value of D is usually between 300 and 500. Each dimension may contain different information. The amount of computation required to directly compare different evaluation vectors sj is large, and the computation speed needs to be improved. By calculating the sum of the elements in the evaluation vector, the multi-dimensional information can be compressed into a single value, which simplifies the representation, makes the results easier to understand and interpret, and can also intuitively reflect the overall importance of vocabulary B in text A. This quantitative representation method facilitates the comparison of the importance of different vocabulary, and is accurate and efficient when multiple vocabulary need to be sorted or screened. In addition, in some cases, some dimensions in the evaluation vector may contain more noise or less relevant information. By summing, the impact of these noises can be reduced to a certain extent, making the final importance score more stable and reliable, and improving the interpretability and application flexibility of the model.

[0068] In other embodiments, the evaluation vectors sj may be compared in other ways.

[0069] For example, the modulus (length) of each evaluation vector, that is, the square root of the sum of the squares of the vector elements, may be calculated to obtain the modulus (length) of each evaluation vector, and the moduli of each evaluation vector may be compared.

[0070] Alternatively, corresponding weights may be set for elements at different positions in the evaluation vector sj, and the weighted sum of each element of the evaluation vector sj may be calculated.

[0071] In some embodiments, the maximum element values ​​of the evaluation vectors may be compared, or the minimum element values ​​of the evaluation vectors may be compared.

[0072] In other embodiments, the medians of the element values ​​of each evaluation vector may be compared, or the averages of the element values ​​of each evaluation vector may be compared, or the geometric mean (square root of the product of the elements, the number of square roots is equal to the number of elements) of the element values ​​of each evaluation vector may be compared.

[0073] In some further embodiments, the Lp norms of the evaluation vectors may be compared. Setting the p value in the Lp norm calculation formula to 2 is equivalent to comparing the Euclidean distances of the evaluation vectors; setting the p value in the Lp norm calculation formula to 1 is equivalent to comparing the Manhattan distances of the evaluation vectors. Those skilled in the art may set the p value according to actual conditions.

[0074] There may be many methods for comparing the evaluation vectors sj, and the present disclosure does not impose any particular limitation thereto.

[0075] By comparing the evaluation vectors sj corresponding to each word to be processed aj, the largest K evaluation vectors are selected, and then the words to be processed corresponding to the selected evaluation vectors are determined to be the keywords of the training text A. This can speed up the calculation while effectively extracting the words that best express the meaning of the training text A as the keywords of the training text A.

[0076] Figure 3 It is a sub-flowchart of step S3 in some other embodiments of the present disclosure.

[0077] refer to Figure 3 In some other exemplary embodiments of the present disclosure, step S3 may include:

[0078] Step S33, forming a weighted word vector of the word to be processed according to the word vector of the word to be processed and the word frequency of the word to be processed in the training text;

[0079] Step S34, multiplying the weighted word vector of the vocabulary to be processed by the text vector matrix to obtain an evaluation vector corresponding to the vocabulary to be processed;

[0080] Step S35, determining the similarity between the word vector of the vocabulary to be processed and the text vector matrix according to the evaluation vector.

[0081] exist Figure 3 In the illustrated embodiment, a weighted word vector of the vocabulary to be processed may be formed based on the word vector of the vocabulary to be processed and the word frequency of the vocabulary to be processed in the training text, and then the similarity between the weighted word vector of the vocabulary to be processed and the text vector matrix may be determined.

[0082] In an exemplary embodiment, the product of the weighted word vector of the vocabulary to be processed and the text vector matrix can be set as the similarity corresponding to the vocabulary to be processed.

[0083] Term frequency (TF) refers to the frequency of a particular word appearing in a document. The word vector of the word to be processed aj is multiplied by the word frequency of the word to be processed aj to form a weighted word vector, and then multiplied with the text vector matrix WA to obtain an evaluation vector tj. Compared with the evaluation vector sj formed in the above embodiment, the frequency of occurrence of the word to be processed in the text is taken into account. The introduction of word frequency information can more accurately reflect the importance of a word in the text. For example, if a word appears frequently in a text, its contribution and importance to the text should be higher. In addition, the weighted word vector combines the semantic information and word frequency information of the word, which can more comprehensively reflect the importance of the word in the text.

[0084] Finally, high-frequency words tend to be more representative of the topic or content of the text. By forming a weighted word vector, the weight of high-frequency words increases, which has a greater impact on the results during the calculation process, while enhancing the influence of important words helps to more accurately capture the main features and topics of the text.

[0085] In short, by multiplying the word vector by the word frequency to form a weighted word vector, and then multiplying it by the text vector matrix WA to obtain the evaluation vector tj, we can better reflect the importance of the word aj to be processed in the text, thereby improving the effect of text analysis and processing. Compared with the evaluation vector sj obtained by directly multiplying the word vector by the text vector matrix WA, the evaluation vector tj can better reflect the impact of word frequency on the importance of the word, thereby providing a more accurate evaluation.

[0086] After obtaining the evaluation vector tj, the evaluation vectors tj corresponding to each word to be processed aj can also be compared to obtain the largest K evaluation vectors. The comparison method is referred to the above embodiment and will not be repeated here. Next, the K words to be processed corresponding to the K evaluation vectors are set as the K keywords of the training text A.

[0087] After extracting the keywords, the training texts can be grouped according to the K keywords.

[0088] In step S4, the training texts with the same K subject words in the model training data are grouped into one group, and within the same group, the training texts with similarity of the text vector matrix greater than a preset threshold are deduplicated.

[0089] In related technologies, the traditional algorithm used for deduplication of text is to calculate the similarity of different texts - matching the text similarity of each text with other texts. When the number of texts is large, it takes a huge amount of time and space resources to calculate the training texts in the entire database. In addition, this calculation method stays at the text level of text matching and cannot consider the semantic level. It will identify texts with similar words but different semantics as duplicate texts.

[0090] In order to solve these problems, the embodiment of the present disclosure extracts K keywords from each training text, and then groups the training texts according to the K keywords, so as to quickly classify the training texts with similar semantics, so as to further deduplicate the texts within the group by comparing the text vectors that express the semantics, thereby reducing the amount of calculation and computing resources required for comparing the texts.

[0091] In an exemplary embodiment, the training texts grouped into the same group have the same K keywords. In some embodiments, the K value can be adjusted to limit the accuracy of the grouping of the training texts. The larger the K value, the more accurate the grouping, the fewer the number of training texts in the same group, and the greater the semantic similarity of the training texts in the same group; the smaller the K value, the fewer the number of groups, and the faster the grouping speed.

[0092] The above-mentioned “K keywords are the same” does not restrict the order of the K keywords to be the same.

[0093] In an exemplary embodiment, considering that two or more texts with a large length difference may have exactly the same K keywords, but the possibility that two or more texts with a large length difference are duplicate texts is not high, after the training texts with exactly the same K keywords are divided into the same group and the training texts with not exactly the same K keywords are divided into different groups, the number of words corresponding to the training texts in the same group can also be determined, and the training texts in the same group can be divided into different groups again according to the number of words, so that the difference in the number of words of the training texts in the same group is not greater than a preset number, and the K keywords of the training texts in the same group are the same.

[0094] By grouping the training texts according to the two limiting conditions of the number of words and K keywords, it is possible to most efficiently screen out training texts that may have semantic duplication and content duplication, making it easier to remove duplicates within the group, thereby increasing the success rate of removing duplicates within the group and improving computing efficiency.

[0095] Next, the similarity between the text vector matrix of each training text in the same group and the text vector matrix of other training texts is calculated. In the process of calculating the similarity, each text vector matrix can be flattened first.

[0096] Flattening a text vector matrix refers to converting a multi-dimensional text vector matrix into a one-dimensional array. This process is also called "flattening" or "straightening" to simplify the data structure and facilitate subsequent similarity comparison. In different programming languages, the flattening method may be different, and the present disclosure does not impose any special restrictions on this. The flattened text vector matrix can be called a text vector.

[0097] Next, we can use cosine similarity to calculate the similarity between text vectors in the same group. The calculation process can be expressed as follows:

[0098]

[0099] Where α and β are the serial numbers of the two text vectors being compared (i.e., the serial numbers of the training texts corresponding to the corresponding text vector matrix in all the training texts of the model training data), Sα and Sβ represent the two text vectors being compared in the same group, and the value range of formula (2) is [-1, 1], where -1 is completely dissimilar and 1 is completely similar. Set a preset threshold T. When Similarity(Sα, Sβ)>T, the training text corresponding to Sβ can be removed from the group, and the above process is repeated. Sα is used to calculate the similarity with the remaining text vectors in the same group. After Sα calculates the similarity with each text vector, other text vectors are used to compare the similarity with each remaining text vector until the similarity between the remaining text vectors in the group is less than or equal to T.

[0100] In an exemplary embodiment, in addition to the pairwise comparison, multiple text vectors can also be directly compared for similarity. For example, the text vectors are applied to a hierarchical clustering algorithm, and the text vectors are gradually grouped according to the similarity to generate a hierarchical structure to show the similarity relationship between different text vectors; or, the k-nearest neighbor algorithm is used to calculate the distance or similarity between a text vector and all other text vectors, and the closest k text vectors are selected as the measure of similarity to find the k text vectors that are most similar to the text vector.

[0101] There are many methods for calculating the similarity between text vectors, such as Euclidean distance, Manhattan distance, etc., which are not limited to the above formula, and those skilled in the art can set it according to actual conditions.

[0102] Figure 4 is a flowchart of one embodiment of method 100 in an exemplary embodiment of the present disclosure.

[0103] refer to Figure 4 In an exemplary embodiment, the implementation process of method 100 may include:

[0104] Step S401, obtaining training text.

[0105] Step S402: perform word segmentation preprocessing to obtain a first word segmentation list.

[0106] Step S403, deduplication of words in the first word segmentation list is performed to obtain a second word segmentation list, wherein the second word segmentation list includes a plurality of words to be processed.

[0107] Step S404, calculating the word vector of each word in the first word segmentation list to form a word vector matrix corresponding to the first word segmentation list.

[0108] Step S405, forming a text vector matrix according to the word vector matrix corresponding to the first word segmentation list, and proceeding to step S406 or step S407.

[0109] Step S406, calculating the word frequency of each word to be processed in the training text.

[0110] Step S407: determine the word vector or weighted word vector corresponding to the to-be-processed vocabulary in the second word segmentation list.

[0111] Step S408, determining the similarity between the word vector or weighted word vector corresponding to the vocabulary to be processed and the text vector matrix.

[0112] Step S409: Select K words to be processed with the largest similarity to form K subject words of the training text.

[0113] Step S410, dividing the training texts and their corresponding text vector matrices into P groups according to the keywords.

[0114] Step S411, flatten the text vector matrix to obtain the text vector.

[0115] Step S412, calculating the similarity of the text vectors in the kth group, deduplicating the training texts corresponding to the text vectors whose similarity is greater than a preset threshold, and setting the initial value of k to 1.

[0116] Step S413, determine whether the current group number k is equal to the total number of groups P. If so, add one to k and return to step S412, otherwise, end the long text deduplication process.

[0117] The disclosed embodiment can better express the semantic information of long texts through word vector matrices and text vector matrices, and the subject word classification can further improve the performance of the algorithm when classifying long texts. It is different from the previous invention patents based on the text vector matrix. During the generation of the text vector matrix, the word vector matrix is ​​generated according to the sequence of long text sentences, and the text vector matrix is ​​generated using the word vector matrix. The word vector weight is calculated in conjunction with the word frequency weight, and the subject word classification can effectively reduce the problem of system resource consumption in long text classification.

[0118] By generating a text vector matrix based on the word vector matrix, and using the word frequency weight to weight and generate the word vector matrix in sequence, the word order relationship of the long text can be fully considered to obtain a text vector matrix with good semantic information, which can improve the deduplication accuracy and efficiency when performing similarity deduplication. In addition, based on the long text deduplication method of vector matrix and subject word classification, the text vector matrix can fully express the semantic information of the long text, and based on subject word classification and similarity deduplication, it greatly reduces the time consumption problem in text similarity matching, and the deduplication accuracy is improved.

[0119] Using the training text after deduplication by method 100 to train the model has a faster training speed. After experiments, the model has significantly better perplexity than the model trained by using the training data without deduplication, and can avoid the model from showing memory.

[0120] In addition to providing good data preprocessing for the training of general large models or domain large models and implementing semantic-based long text deduplication, the method of the embodiments of the present invention can also meet the data processing requirements for deleting duplicate texts in all other large model training corpora, and has high universality.

[0121] Corresponding to the above method embodiments, the present disclosure also provides a model training data processing device, which can be used to execute the above method embodiments.

[0122] Figure 5 It is a block diagram of a model training data processing device in an exemplary embodiment of the present disclosure.

[0123] refer to Figure 5 , the model training data processing device 500 may include:

[0124] A word segmentation processing module 51 is configured to perform word segmentation preprocessing on the training text in the model training data to form a first word segmentation list, and to obtain a second word segmentation list after deduplication of words in the first word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating words to be processed;

[0125] A text feature extraction module 52, configured to form a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list;

[0126] A keyword extraction module 53 is configured to determine the similarity between the word vector of the to-be-processed vocabulary and the text vector matrix, and determine the K to-be-processed vocabulary with the largest similarity as the K subject words of the training text, K≥1;

[0127] A text deduplication module 54 is configured to group the training texts with the same K subject words in the model training data into one group, and to deduplication the training texts with the similarity of the text vector matrix greater than a preset threshold in the same group;

[0128] Since the functions of the device 500 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0129] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0130] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0131] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0132] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present invention is described. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).

[0134] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the method shown in the embodiment of the present disclosure.

[0135] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0136] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0137] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0138] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0139] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0140] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0141] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0142] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0145] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0146] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0147] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A model training data processing method, characterized in that: include: Performing word segmentation preprocessing on the training text in the model training data to form a first word segmentation list, performing word deduplication on the first word segmentation list to obtain a second word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating words to be processed; Forming a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list; Determine the similarity between the word vector of the to-be-processed vocabulary and the text vector matrix, and determine the K to-be-processed vocabulary with the largest similarity as the K subject words of the training text, K≥1; In the model training data, the training texts with the same K subject words are grouped into one group, and within the same group, the training texts with similarity of the text vector matrix greater than a preset threshold are deduplicated.

2. The model training data processing method according to claim 1, characterized in that: The forming of the text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list includes: The text vector matrix is ​​determined according to the product of the word vector matrix and the transpose of the word vector matrix.

3. The model training data processing method according to claim 1, characterized in that: Determining the similarity between the word vector of each of the to-be-processed words and the text vector matrix includes: Multiplying the word vector of the vocabulary to be processed by the text vector matrix to obtain an evaluation vector corresponding to the vocabulary to be processed; The similarity between the word vector of the vocabulary to be processed and the text vector matrix is ​​determined according to the evaluation vector.

4. The model training data processing method according to claim 1, characterized in that: Determining the similarity between the word vector of the vocabulary to be processed and the text vector matrix includes: Forming a weighted word vector of the vocabulary to be processed according to the word vector of the vocabulary to be processed and the word frequency of the vocabulary to be processed in the training text; Multiplying the weighted word vector of the vocabulary to be processed by the text vector matrix to obtain an evaluation vector corresponding to the vocabulary to be processed; The similarity between the word vector of the vocabulary to be processed and the text vector matrix is ​​determined according to the evaluation vector.

5. The model training data processing method according to claim 3 or 4, characterized in that: Determining the similarity between the word vector of the vocabulary to be processed and the text vector matrix according to the evaluation vector includes: The sum of the elements of the evaluation vector is set as the similarity between the word vector of the vocabulary to be processed and the text vector matrix.

6. The model training data processing method according to claim 1, characterized in that: In the model training data, the training texts having the same K subject words are grouped into one group, including: The training texts with the K completely identical keywords are divided into the same group, and the training texts with the K non-completely identical keywords are divided into different groups; Determine the number of words in the first word segmentation list corresponding to the training text in the same group; The training texts in the same group are divided into different groups again according to the number of words, so that the difference in the number of words of the training texts in the same group is not greater than a preset number, and the K keywords of the training texts in the same group are the same.

7. A model training data processing device, characterized in that: include: A word segmentation processing module, configured to perform word segmentation preprocessing on the training text in the model training data to form a first word segmentation list, and to obtain a second word segmentation list after deduplication of words in the first word segmentation list, wherein the second word segmentation list includes a plurality of non-repeating words to be processed; A text feature extraction module, configured to form a text vector matrix of the training text according to the word vector matrix corresponding to the first word segmentation list; A keyword extraction module is configured to determine the similarity between the word vector of the to-be-processed vocabulary and the text vector matrix, and determine the K to-be-processed vocabulary with the largest similarity as the K subject words of the training text, K≥1; The text deduplication module is configured to group the training texts with the same K subject words in the model training data into one group, and to dedupe the training texts whose similarity of the text vector matrix is ​​greater than a preset threshold within the same group.

8. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images