Data processing methods, apparatus, electronic devices and computer-readable storage media

Abstract

This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium. The method includes: obtaining multiple frequency calculation tasks, and for each training corpus data in each frequency calculation task, calculating the frequency of tuples in the training corpus data; statistically analyzing the frequencies of tuples in all training corpus data in each frequency calculation task to obtain the pre-cumulative frequency of tuples in each frequency calculation task; statistically analyzing the pre-cumulative frequencies of tuples in all frequency calculation tasks to obtain the cumulative frequency of tuples, and calculating the probability of each tuple based on the cumulative frequency of tuples. The method of this application optimizes data processing complexity by adding pre-cumulative frequency processing and calculating the probability of each tuple based on the cumulative frequency of tuples, thus solving the problem that the vocabulary size supported by existing language models is limited by single-machine memory and that ultra-large-scale corpora cannot be stored and computed on a single machine.

Description

Technical Field

[0001] This application relates to natural language processing technology, and more particularly to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Language models are widely used in applications such as natural language processing, machine translation, input methods, and speech recognition. N-gram (N-gram language model) language models are the most common and effective type of language model. They can be trained using various language modeling tools to obtain standard Arpa-formatted language models.

[0003] Common existing language model training tools include SRILM (Stanford Research Institute Language Modeling Toolkit) and IRSTLM (IRST language modeling toolkit). Both of these language modeling tools operate on a single computer. SRILM is based on a trie (prefix tree) and supports various smoothing methods and entropy pruning, which are applied in some decoders; however, it has a large memory footprint. IRSTLM is based on a sorted trie and is designed for low memory consumption, but some smoothing methods are not implemented, such as the improved Kneser-Ney method.

[0004] However, the vocabulary size supported by existing language models is limited by single-machine memory, and ultra-large-scale corpora cannot be stored and computed on a single machine. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium to solve the problem that the vocabulary size supported by existing language models is limited by single-machine memory and that ultra-large-scale corpora cannot be stored and computed on a single machine.

[0006] One embodiment of this application provides a data processing method, including:

[0007] Multiple frequency calculation tasks are obtained, and for each training corpus data in each frequency calculation task, the frequency of tuples in the training corpus data is calculated;

[0008] Count the frequency of tuples in all training corpus data for each frequency calculation task to obtain the pre-cumulative frequency of tuples in each frequency calculation task;

[0009] The cumulative frequency of each tuple is obtained by counting the pre-cumulative frequency of all tuples in the frequency calculation task, and the probability of each tuple is calculated based on the cumulative frequency of the tuples.

[0010] In one embodiment, the probability of each tuple is calculated based on the cumulative frequency of the tuples, specifically including:

[0011] Multiple frequency calculation tasks are obtained, each of which includes a pre-set discount factor for smoothing.

[0012] For each frequency calculation task, a smoothing function and a discount factor are used to process the cumulative frequency of the tuples to obtain the intermediate probability of each tuple;

[0013] For each frequency calculation task, the intermediate probabilities of each tuple are backed up, and the probabilities of the tuples are output.

[0014] In one embodiment, for each frequency calculation task, the intermediate probabilities of each tuple are backed up, and the probabilities of the tuples are output, specifically including:

[0015] For each frequency calculation task, the first low-order tuple is obtained by dividing each tuple in each frequency calculation task into different orders, and the probability of the first low-order tuple is calculated based on the intermediate probability of the tuple.

[0016] For each tuple in each frequency calculation task, the joint probability of the tuple is obtained based on the probability of the first low-order tuple and the intermediate probability of the tuple.

[0017] For each frequency calculation task, each tuple in each frequency calculation task is further subdivided to obtain a second low-order tuple, and the probability of the second low-order tuple is calculated based on the intermediate probability of the tuple.

[0018] For each tuple in each frequency calculation task, the backoff probability of the second lower-order tuple is obtained based on the probability of the second lower-order tuple and the joint probability of the tuple.

[0019] In one embodiment, multiple frequency calculation tasks are obtained, specifically including

[0020] Multiple training corpora are segmented into words to obtain multiple segmented training corpora.

[0021] The training corpus data after multiple word segmentation is processed into multiple training tasks.

[0022] In one embodiment, the method further includes:

[0023] Obtain the corpus to be processed, sort the corpus using the probabilities of the tuple, and output the sentences;

[0024] or

[0025] Obtain the corpus to be processed, use the probabilities of the tuple to predict the corpus, and output the sentences.

[0026] Another embodiment of this application provides a data processing apparatus, including:

[0027] The acquisition module is used to acquire multiple frequency calculation tasks and, for each training corpus data in each frequency calculation task, calculate the frequency of tuples in the training corpus data;

[0028] The processing module is used to count the frequency of tuples in all training corpus data in each frequency calculation task, and obtain the pre-cumulative frequency of tuples in each frequency calculation task.

[0029] The processing module is also used to calculate the cumulative frequency of each tuple in all frequency calculation tasks by counting the pre-cumulative frequency of the tuples, and to calculate the probability of each tuple based on the cumulative frequency of the tuples.

[0030] In one embodiment, the processing module is specifically used for:

[0031] Multiple frequency calculation tasks are obtained, each of which includes a pre-set discount factor for smoothing.

[0032] For each frequency calculation task, a smoothing function and a discount factor are used to process the cumulative frequency of the tuples to obtain the intermediate probability of each tuple;

[0033] For each frequency calculation task, the intermediate probabilities of each tuple are backed up, and the probabilities of the tuples are output.

[0034] Another embodiment of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0035] The memory stores the instructions that the computer executes;

[0036] The processor executes computer-executable instructions stored in memory to implement any of the methods described in 1 to 5 above.

[0037] In another embodiment of this application, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, are used to implement the methods described in any one of 1 to 5 above.

[0038] Another embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements any one of the methods described in 1 to 5 above.

[0039] The data processing method, apparatus, electronic device, and computer-readable storage medium provided in this application obtain multiple frequency calculation tasks and, for each training corpus data in each frequency calculation task, calculate the frequency of tuples in the training corpus data; statistically analyze the frequency of tuples in all training corpus data in each frequency calculation task to obtain the pre-cumulative frequency of tuples in each frequency calculation task, thereby solving the data skew problem caused by large-scale vocabularys during language model training and reducing the burden on individual nodes; statistically analyze the pre-cumulative frequency of tuples in all frequency calculation tasks to obtain the cumulative frequency of tuples, and calculate the probability of each tuple based on the cumulative frequency of tuples, optimizing complexity in steps such as calculation backoff, ensuring that the number of N-tuples reduced by a single node in the worst case is less than or equal to the number of tuples, and can handle large-scale corpora and dictionaries, solving the problem that the vocabulary size supported by existing language models is limited by single-machine memory and that ultra-large-scale corpora cannot be stored and calculated on a single machine. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 A flowchart of a data processing method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of a data counting method provided in an embodiment of this application;

[0043] Figure 3 A flowchart of a method for calculating the probability of each tuple based on the cumulative frequency of tuples is provided in another embodiment of this application;

[0044] Figure 4 A flowchart of a probability method for outputting tuples is provided in another embodiment of this application;

[0045] Figure 5A A schematic diagram illustrating a backoff value calculation method provided in another embodiment of this application;

[0046] Figure 5B A schematic diagram illustrating a backoff value calculation method provided in another embodiment of this application;

[0047] Figure 6 A flowchart of a method for obtaining multiple frequency calculation tasks is provided in another embodiment of this application;

[0048] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application;

[0049] Figure 8This is a schematic diagram of an electronic device structure provided in another embodiment of this application.

[0050] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] Language models are widely used in applications such as natural language processing, machine translation, input methods, and speech recognition. N-gram language models are the most common and effective type of language model. They can be trained using various language modeling tools to obtain language models in the standard Arpa format.

[0053] Common language model training tools include SRILM and IRSTLM. Both of these tools operate on a single computer. SRILM, based on a Trie tree, supports various smoothing methods and entropy pruning, and is applied in some decoders, but it has a large memory footprint. IRSTLM, based on a sorted trie, is designed for low memory consumption, but some smoothing methods are not implemented, such as the improved Kneser-Ney method.

[0054] However, the vocabulary size supported by existing language models is limited by single-machine memory, and ultra-large-scale corpora cannot be stored and computed on a single machine.

[0055] To address the aforementioned issues, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable storage medium. These methods, apparatus, electronic devices, and computer-readable storage media aim to solve the problem that the vocabulary size supported by existing language models is limited by single-machine memory, and that ultra-large-scale corpora cannot be stored and computed on a single machine. The technical concept of this application is as follows: By obtaining multiple frequency calculation tasks, and for each training corpus data in each frequency calculation task, the frequency of tuples in the training corpus data is calculated; the frequency of tuples in all training corpus data in each frequency calculation task is statistically analyzed to obtain the pre-cumulative frequency of tuples in each frequency calculation task, thereby solving the data skew problem caused by large-scale vocabularys during language model training and reducing the burden on individual nodes; the cumulative frequency of tuples is obtained by statistically analyzing the pre-cumulative frequency of tuples in all frequency calculation tasks, and the probability of each tuple is calculated based on the cumulative frequency of tuples. Complexity optimization is performed in steps such as calculation backoff to ensure that the number of N-tuples reduced by a single node in the worst case is less than or equal to the number of words in the dictionary, which can handle large-scale corpora and dictionaries, and solves the problem that the vocabulary size supported by existing language models is limited by single-machine memory and that ultra-large-scale corpora cannot be stored and calculated on a single machine.

[0056] like Figure 1 As shown, one embodiment of this application provides a data processing method, which includes the following steps:

[0057] S101. Obtain multiple frequency calculation tasks, and for each training corpus data in each frequency calculation task, calculate the frequency of tuples in the training corpus data.

[0058] In this step, considering the massive corpus data, a language model training tool was developed based on the Hadoop big data platform. The platform can be selected according to specific needs. Streaming tasks are applied on the Hadoop platform for processing, resulting in low memory consumption per node, easy parallelization, and fast training speed. The core design of the Hadoop framework consists of HDFS (Hadoop Distributed File System) and MapReduce. HDFS provides storage for massive amounts of data, while MapReduce provides computation for massive amounts of data.

[0059] Based on the Hadoop platform, the MapReduce framework first automatically splits the massive corpus data into multiple splits, and then divides the splits by row.<key,value> Yes, that is, to obtain multiple frequency calculation tasks. The segmented...<key,value> The data is processed by the map method to generate new data.<key,value> Yes, the key is the split dictionary or word, and the value is the frequency of occurrence corresponding to the key. That is, for each frequency calculation task, the frequency of each tuple in the training corpus data is calculated.

[0060] S102. Count the frequency of tuples in all training corpus data in each frequency calculation task to obtain the pre-cumulative frequency of tuples in each frequency calculation task.

[0061] In this step, to address the issue that when dealing with extremely large datasets, each map operation may generate a large amount of output, severely consuming network bandwidth and reducing program efficiency, and causing single nodes to be overloaded and leading to task failure, the following approach is taken: First, the local pre-cumulative frequency of the tuples in each frequency calculation task is performed. The keys generated and output by each map are locally sorted, and the values ​​corresponding to the same key are locally iterated, thus obtaining the pre-cumulative frequency of the tuples in each frequency calculation task.

[0062] S103. Calculate the cumulative frequency of each tuple in the frequency calculation task by counting the pre-cumulative frequency of all tuples, and calculate the probability of each tuple based on the cumulative frequency of the tuples.

[0063] In this step, the pre-cumulative frequency of the tuples in each frequency calculation task is obtained.<key,value> Then the mapper calculates the pre-cumulative frequency of tuples in the task for different frequencies.<key,value> Data with the same key is globally sorted and merged to obtain the final output of the shuffle. Reduce processes the data received from the shuffle to obtain new data.<key,value> right.

[0064] For example, such as Figure 2This diagram illustrates a data counting method according to an embodiment of this application. There are three frequency calculation tasks, each including: W1, W2, and W3; W7, W8, and W9; W4, W5, and W6. After three map operations, a...<key,value> Yes, the output of the first map is: <1, 1>, <2, 1>, <1, 1>; the output of the second map is: <1, 1>, <1, 1>, <3, 1>; and the output of the third map is: <1, 1>, <1, 1>, <4, 1>. After combining, the same keys are locally sorted on a single map, and the values ​​corresponding to the same keys are iterated to obtain new values.<key,value> Yes, <1, 2>, <2, 1>; <1, 2>, <3, 1>; <1, 2>, <4, 1>. After shuffling, the maps with the same key are globally merged and sorted to obtain a new map.<key,value> Yes, <1, 2>, <1, 2>, <1, 2>; <2, 1>; <3, 1>; <4, 1>; Finally, the reduce method performs global iteration processing on the values ​​corresponding to the same key to obtain the final word count result, specifically: <1, 6>; <2, 1>; <3, 1>; <4, 1>.

[0065] Based on the cumulative frequency of the tuples, unreliable (low-frequency) N-gram frequencies appearing in the training set are pruned and allocated to unseen N-grams. Different "pruning + allocation" methods correspond to different smoothing methods, adjusting the probability distribution to ensure that each probability parameter in the N-gram statistical language model is non-zero and to make the probability distribution more reasonable and uniform. When querying higher-order N-gram probabilities, due to data sparsity and pruning, the model often does not observe N-order probabilities. Therefore, an (n-1)-order probability is used to replace the unobserved probability, and a penalty is applied to this probability by calculating a backoff probability. Finally, the n-grams are separated by order, and the n-grams of each order are sorted, the probability precision is trimmed, and the output is in Arpa format for application in natural language processing, machine translation, input methods, and speech recognition.

[0066] In one embodiment, the method further includes:

[0067] S901. Obtain the corpus to be processed, sort the corpus using the probabilities of the tuple, and output the statements.

[0068] In this step, the N-gram language model is trained using the aforementioned language modeling tools. These tools include: corpus segmentation, N-gram extraction, N-gram counting, count adjustment, probability calculation, backoff probability calculation, and outputting the language model in Arpa format. This invention is based on Hadoop and optimizes the training tools to obtain a model capable of training on large-scale data. For example, when a user uses the model to sort sentences, such as translating Chinese into French, the model performs a subject-verb-object grammatical sorting process on the French sentences based on the probability of using tuples, and then outputs the desired French sentences.

[0069] In another embodiment, the method further includes:

[0070] S902. Obtain the corpus to be processed, use the probability of the tuple to predict the corpus to be processed, and output the sentence.

[0071] In this step, an N-gram language model is trained using the aforementioned language modeling tools. These tools include: corpus segmentation, N-gram extraction, N-gram counting, count adjustment, probability calculation, backoff probability calculation, and outputting the language model in Arpa format. This invention is based on Hadoop and optimizes the training tools to obtain a model capable of training on large-scale data. For example, when a user uses the model to search for information, such as inputting the sentence "I want to understand mathematical modeling methods," the trained model, after acquiring the corpus, can use the probability of N-grams to predict and output the desired sentence, "I want to understand mathematical modeling methods."

[0072] In the above technical solution, multiple frequency calculation tasks are obtained, and for each training corpus data in each frequency calculation task, the frequency of tuples in the training corpus data is calculated; the frequency of tuples in all training corpus data in each frequency calculation task is counted to obtain the pre-cumulative frequency of tuples in each frequency calculation task, which solves the data skew problem caused by large-scale vocabulary (tens of millions of words) during language model training and reduces the burden on a single node; the pre-cumulative frequency of tuples in all frequency calculation tasks is counted to obtain the cumulative frequency of tuples, and the probability of each tuple is calculated based on the cumulative frequency of tuples. Complexity optimization is performed in steps such as calculation backoff to ensure that the number of N-tuples reduced by a single node in the worst case is less than or equal to the number of words in the dictionary, which can handle large-scale corpora and dictionaries, and solves the problem that the vocabulary size supported by existing language models is limited by single-machine memory and that ultra-large-scale corpora cannot be stored and calculated on a single machine.

[0073] like Figure 3 As shown, another embodiment of this application provides a method for calculating the probability of each tuple based on the cumulative frequency of the tuples. The method includes the following steps:

[0074] S201. Obtain multiple frequency calculation tasks, wherein each frequency calculation task includes a pre-set discount factor for smoothing processing.

[0075] In this step, in order to better estimate the model, the frequencies of unreliable (low-frequency) N-grams that appear in the training set must first be reduced and allocated to N-grams that do not appear.

[0076] For example, the discount factor is obtained through Katz smoothing. It should be noted that the calculation of the discount factor can be customized according to requirements. The Katz smoothing method extends the Good-Turing estimation method by incorporating a combination of high-order and low-order models. The implementation formula in Katz smoothing is as follows:

[0077]

[0078] Among them, n consecutive words w i-n , ..., w i It is called an n-order tuple, p katz (w i-n , ..., w i The probability of a tuple of order n occurring given a tuple of order n-1 is w. i The probability of f(w) i |w i-n w i-n+1 , ..., w i-1 Let be the probability of an n-th tuple, and w be the probability of an (n-1)-th tuple. i Frequency; c(w) i-n , ..., w i ) represents the frequency of occurrence of n-order tuples, also known as counting; d is the discount factor; α is the backtracking probability.

[0079] To facilitate the calculation of the discount factor d, let r represent c(w) i-n , ..., w i );

[0080]

[0081] Where, n k This represents the frequency of the k-frequency.

[0082] S202. For each frequency calculation task, use a smoothing function and discount factor to process the cumulative frequency of the tuples to obtain the intermediate probability of each tuple.

[0083] In this step, for each frequency calculation task, the cumulative frequency of the tuple is processed according to the smoothing function and the discount factor. The smoothing function is specifically as follows:

[0084]

[0085] Among them, n consecutive words w i-n , ..., w i It is called an n-order tuple, p(w i |w i-n w i-n+1 , ..., w i-1 ) represents the middle probability of an n-th order tuple, and w occurs given an n-1 order tuple. i The probability of f(w) i |w i-n w i-n+1 , ..., w i-1 Let be the probability of an n-th tuple, and w be the probability of an (n-1)-th tuple. i Frequency; c(w) i-n , ..., w i The frequency of occurrence of n-order tuples can also be called counting; d r α is the discount factor; α is the rollback probability.

[0086] S203. For each frequency calculation task, backtrack the intermediate probabilities of each tuple and output the probabilities of the tuple.

[0087] In this step, the intermediate probabilities of each tuple are backtracked, and the probabilities of the tuples are output. The specific backtracking process is as follows:

[0088]

[0089] Wherein, p(w i |w i-n w i-n+1 , ..., w i-1 ) is c(w i-n , ..., w i The probability of an n-tuple being greater than 0.

[0090] In the above technical solution, by first calculating the deduction coefficient and then substituting it into the probability calculation, the frequency of unreliable (low-frequency) N-grams appearing in the training set is reduced; and by calculating the backoff value, the problem that when querying higher-order N-gram probabilities is solved, due to data sparsity and pruning, an n-1 order probability is used to replace the unobserved probability, and the model often does not observe the N-order probability.

[0091] like Figure 4As shown, another embodiment of this application provides a probability method for outputting tuples, the method comprising the following steps:

[0092] S301. For each frequency calculation task, divide each tuple in each frequency calculation task into first low-order tuples and calculate the probability of the first low-order tuples based on the intermediate probability of the tuples.

[0093] In this step, for each tuple in each frequency calculation task, a custom map method is used to obtain the lower-order probabilities.<key,value> Yes, the key is the first low-order tuple of the tuple, and the value contains two parts: the first part is the higher-order prefix, and the second part is the higher-order probability, which is the middle probability of the tuple. Then, a shuffle operation is performed to globally sort and merge tuples with the same key.

[0094] S302. For each tuple in each frequency calculation task, obtain the joint probability of the tuple based on the probability of the first low-order tuple and the intermediate probability of the tuple.

[0095] In this step, for each tuple in each frequency calculation task, a custom reduce method is used to broadcast the lower-order probability to the higher-order probability, and the higher and lower-order probabilities are combined to obtain the joint probability of the tuple. The joint probability includes the lower-order probability and the higher-order probability (intermediate probability).

[0096] For example, such as Figure 5AThis diagram illustrates a backoff value calculation method provided in another embodiment of this application. The probabilities of each tuple at the three nodes are: ABC 0.8, ABD 0.15, ABE 0.05; BC 0.6, BD 0.3, BE 0.1; FBC 0.3, FBD 0.5, FBE 0.2. After processing using the map method, the following is obtained:<key,value> Yes, specifically: Key: BC val: (A, 0.8), Key: BD val: (A, 0.15), Key: BE val: (A, 0.05); Key: BC val: (#, 0.6), Key: BD val: (#, 0.3), Key: BE val: (#, 0.1); Key: BC val: (F, 0.3), Key: BD val: (F, 0.5), Key: BE val: (F, 0.2). After shuffling, identical keys are sorted and reduced to obtain new...<key,value> Yes, specifically: Key: BC val: (A, 0.8), Key: BC val: (F, 0.3), Key: BC val: (#, 0.6); Key: BD val: (A, 0.15), Key: BD val: (F, 0.5), Key: BD val: (#, 0.3); Key: BE val: (A, 0.05), Key: BE val: (F, 0.2), Key: BE val: (#, 0.1). Using a custom reduce method, the joint probabilities of the higher and lower order probabilities are obtained: ABC 0.8, 0.6, FBC 0.3, 0.6; ABD 0.15, 0.3, FBD 0.5, 0.3; ABE 0.05, 0.1, FBE 0.2, 0.1.

[0097] S303. For each frequency calculation task, each tuple in each frequency calculation task is further subdivided to obtain a second low-order tuple, and the probability of the second low-order tuple is calculated based on the intermediate probability of the tuple.

[0098] In this step, for each tuple in each frequency calculation task, the lower-order probabilities are again obtained using a custom map method.<key,value> Yes, the key is the prefix of the second lowest-order tuple of the tuple, and the value contains two parts: the first part is the higher-order suffix, and the second part is the combined higher and lower-order probabilities. The probability of the second lowest-order tuple is calculated using the intermediate probabilities of the tuples according to the formula. Then, a shuffle operation is performed to globally sort and merge tuples with the same key.

[0099] S304. For each tuple in each frequency calculation task, obtain the backoff probability of the second low-order tuple based on the probability of the second low-order tuple and the joint probability of the tuple.

[0100] In this step, for each tuple in each frequency calculation task, a backoff coefficient is calculated using a custom reduce method to obtain the backoff probability of the second low-order tuple prefix.

[0101] For example, such as Figure 5B This diagram illustrates a backoff value calculation method provided in another embodiment of this application. The high- and low-order fusion probabilities of each tuple at the three nodes are: ABC 0.8, 0.6, FBC 0.3, 0.6; ABD 0.15, 0.3, FBD 0.5, 0.3; ABE 0.05, 0.1, FBE 0.2, 0.1. After processing using the map method, the following is obtained:<key,value> Yes, specifically: Key: AB val: (C, 0.8, 0.6), Key: FB val: (C, 0.3, 0.6); Key: AB val: (D, 0.15, 0.3), Key: FB val: (D, 0.5, 0.3); Key: AB val: (E, 0.05, 0.1), Key: FB val: (E, 0.2, 0.1). After a shuffle operation, identical keys are globally sorted and merged to obtain new...<key,value> Yes, specifically: Key: AB val: (C, 0.8, 0.6), Key: AB val: (D, 0.15, 0.3), Key: AB val: (E, 0.05, 0.1); Key: FB val: (C, 0.3, 0.6), Key: FB val: (D, 0.5, 0.3), Key: FB val: E, 0.2, 0.1. Using a custom reduce method, the backoff probabilities of the tuple prefixes AB and FB are obtained by combining the higher and lower order probabilities.

[0102] In the above technical solution, a two-step MapReduce method is designed. First, low-order probabilities are broadcast to high-order probabilities. Then, the high-order prefix is ​​reduced using the fused high- and low-order probabilities obtained by combining the probabilities, and the backoff probability of the tuple prefix is ​​obtained, thus optimizing the computational complexity. This addresses the problem that when dealing with large vocabularies and large corpora, node timeouts can be severe, even exceeding the memory capacity of a single node, causing the task to fail directly and preventing training from being completed. In the worst case, each node only processes a number of data points equal to the size of the vocabulary.

[0103] like Figure 6 As shown, another embodiment of this application provides a method for obtaining multiple frequency calculation tasks, the method including the following steps:

[0104] S401. Perform word segmentation on multiple training corpora to obtain multiple segmented training corpora.

[0105] In this step, multiple training corpora are obtained through text cleaning, primarily to remove misspellings, profanities, rare characters, punctuation marks, etc., from commonly used inputs, thereby reducing the modeling scope and improving modeling quality. Furthermore, the text is counted during the Reduce phase to reduce computational load.

[0106] After obtaining multiple training corpora, the training corpora are divided from continuous character sequences into continuous word sequences to achieve word segmentation.

[0107] For example, word segmentation can be forward segmentation, also known as forward maximum matching algorithm, which searches from left to right in the sentence to be segmented to find the maximum match for the word. First, a maximum word length needs to be defined. During each scan, a word of the current length is searched to match with words in the dictionary. If no match is found, the length is shortened and the search continues until a word in the dictionary is found or a single character is formed.

[0108] For example, word segmentation can be done using Viterbi segmentation, which is the application of the Viterbi algorithm to word segmentation. The Viterbi algorithm solves the shortest path problem in a fence-like graph. The nodes of the graph are organized by columns, and the number of nodes in each column can be different. Nodes in each column can only be connected to nodes in adjacent columns, not across columns, and there are different distances between nodes. For Chinese character segmentation, we treat characters as fences, and the intervals between characters are abstracted as nodes.

[0109] For example, a word segmentation method could be self-supervised N-gram segmentation. This method uses an N-gram model to score all possible paths in a sentence and selects the path with the highest probability as the final segmentation result. First, forward random segmentation is used to cover all possible paths in a sentence, preventing errors in subsequent results due to errors in a single segmentation. Then, word frequency statistics are used to distribute data across nodes within a single machine, addressing the issue of data skew in individual nodes. The N-gram model is initialized using Viterbi segmentation with an N-gram array. To ensure accuracy, model estimation and backoff value calculations are used, followed by self-supervised iteration to determine whether adjacent segmentation results converge, thus ensuring the accuracy of the segmentation probability.

[0110] It should be noted that you can choose the word segmentation method according to your needs.

[0111] S402. The training corpus data after multiple word segmentation is processed into multiple training tasks.

[0112] In this step, based on the Hadoop platform, the platform can be selected according to its own needs. First, the MapReduce framework automatically splits the massive corpus data into multiple splits, and then divides the splits by row.<key,value> Yes, the language model receives multiple training tasks.

[0113] In the above technical solution, self-supervised N-gram segmentation is used to segment multiple training corpora to obtain multiple segmented training corpora, ensuring the accuracy of the segmentation probability; then, the MapReduce framework is used to automatically split the multiple segmented training corpora into multiple training tasks, solving the problem of data skew on a single node.

[0114] like Figure 7 As shown, one embodiment of this application provides a data processing apparatus 700, the apparatus comprising:

[0115] The acquisition module 701 is used to acquire multiple frequency calculation tasks and, for each training corpus data in each frequency calculation task, calculate the frequency of tuples in the training corpus data.

[0116] The processing module 702 is used to count the frequency of tuples in all training corpus data in each frequency calculation task, and obtain the pre-cumulative frequency of tuples in each frequency calculation task.

[0117] The processing module 702 is also used to obtain the cumulative frequency of the tuples by counting the pre-cumulative frequency of all frequency calculation tasks, and to calculate the probability of each tuple based on the cumulative frequency of the tuples.

[0118] In one embodiment, the processing module 702 is specifically used for:

[0119] The probability of each tuple is calculated based on the cumulative frequency of the tuples, specifically including:

[0120] Multiple frequency calculation tasks are obtained, each of which includes a pre-set discount factor for smoothing.

[0121] For each frequency calculation task, a smoothing function and a discount factor are used to process the cumulative frequency of the tuples to obtain the intermediate probability of each tuple;

[0122] For each frequency calculation task, the intermediate probabilities of each tuple are backed up, and the probabilities of the tuples are output.

[0123] In one embodiment, the processing module 702 is further specifically used for:

[0124] For each frequency calculation task, the intermediate probabilities of each tuple are backed up, and the probabilities of the tuples are output, specifically including:

[0125] For each frequency calculation task, the first low-order tuple is obtained by dividing each tuple in each frequency calculation task into different orders, and the probability of the first low-order tuple is calculated based on the intermediate probability of the tuple.

[0126] For each tuple in each frequency calculation task, the joint probability of the tuple is obtained based on the probability of the first low-order tuple and the intermediate probability of the tuple.

[0127] For each frequency calculation task, each tuple in each frequency calculation task is further subdivided to obtain a second low-order tuple, and the probability of the second low-order tuple is calculated based on the intermediate probability of the tuple.

[0128] For each tuple in each frequency calculation task, the backoff probability of the second lower-order tuple is obtained based on the probability of the second lower-order tuple and the joint probability of the tuple.

[0129] In one embodiment, the acquisition module 701 is specifically used for:

[0130] Multiple training corpora are segmented into words to obtain multiple segmented training corpora.

[0131] The training corpus data after multiple word segmentation is processed into multiple training tasks.

[0132] In one embodiment, the acquisition module 701 is further specifically used for:

[0133] Obtain the corpus to be processed, sort the corpus using the probabilities of the tuple, and output the sentences;

[0134] or

[0135] Obtain the corpus to be processed, use the probabilities of the tuple to predict the corpus, and output the sentences.

[0136] like Figure 8 As shown, one embodiment of this application provides an electronic device 800, which includes a memory 801 and a processor 802.

[0137] Among them, memory 801 is used to store computer instructions that can be executed by the processor;

[0138] The processor 802 implements the various steps of the method in the above embodiments when executing computer instructions. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0139] Optionally, the memory 801 can be either independent or integrated with the processor 802. When the memory 801 is configured independently, the electronic device also includes a bus for connecting the memory 801 and the processor 802.

[0140] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the methods described above.

[0141] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the various steps in the methods described above.

[0142] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0143] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining a plurality of frequency calculation tasks, and calculating the frequency of a plurality of tuples in each training corpus data in each of the frequency calculation tasks; counting the frequency of the plurality of tuples in all training corpus data in each of the frequency calculation tasks to obtain the pre-accumulation frequency of the plurality of tuples in each frequency calculation task; counting the pre-accumulation frequency of the plurality of tuples in all frequency calculation tasks to obtain the accumulation frequency of the plurality of tuples, and calculating the probability of each plurality of tuples according to the accumulation frequency of the plurality of tuples; the probability of the plurality of tuples is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing, and is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing.

2. The method of claim 1, wherein, and according to the accumulation frequency of the plurality of tuples, the probability of each plurality of tuples is obtained, which specifically comprises: obtaining a plurality of frequency calculation tasks, wherein each frequency calculation task includes a discount factor during pre-set smoothing processing; for each frequency calculation task, using a smoothing processing function and the discount factor to process the accumulation frequency of the plurality of tuples to obtain the intermediate probability of each plurality of tuples; for each frequency calculation task, the intermediate probability of each plurality of tuples is back-off processed, and the probability of the plurality of tuples is output.

3. The method of claim 2, wherein, for each frequency calculation task, the intermediate probability of each plurality of tuples is back-off processed, and the probability of the plurality of tuples is output, which specifically comprises: for each frequency calculation task, each plurality of tuples in each frequency calculation task is ranked to obtain a first low-order plurality of tuples, and the probability of the first low-order plurality of tuples is calculated according to the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, the joint probability of the plurality of tuples is obtained according to the probability of the first low-order plurality of tuples and the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, a second low-order plurality of tuples is obtained by ranking again, and the probability of the second low-order plurality of tuples is calculated according to the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, the back-off probability of the second low-order plurality of tuples is obtained according to the probability of the second low-order plurality of tuples and the joint probability of the plurality of tuples.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining a plurality of frequency calculation tasks, and calculating the frequency of a plurality of tuples in each training corpus data in each of the frequency calculation tasks; counting the frequency of the plurality of tuples in all training corpus data in each of the frequency calculation tasks to obtain the pre-accumulation frequency of the plurality of tuples in each frequency calculation task; 5. The method according to any one of claims 1 to 3, characterized in that, counting the pre-accumulation frequency of the plurality of tuples in all frequency calculation tasks to obtain the accumulation frequency of the plurality of tuples, and calculating the probability of each plurality of tuples according to the accumulation frequency of the plurality of tuples; the probability of the plurality of tuples is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing, and is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing. and according to the accumulation frequency of the plurality of tuples, the probability of each plurality of tuples is obtained, which specifically comprises: obtaining a plurality of frequency calculation tasks, wherein each frequency calculation task includes a discount factor during pre-set smoothing processing; for each frequency calculation task, using a smoothing processing function and the discount factor to process the accumulation frequency of the plurality of tuples to obtain the intermediate probability of each plurality of tuples; 6. A data processing apparatus, characterized by, for each frequency calculation task, the intermediate probability of each plurality of tuples is back-off processed, and the probability of the plurality of tuples is output. for each frequency calculation task, the intermediate probability of each plurality of tuples is back-off processed, and the probability of the plurality of tuples is output, which specifically comprises: for each frequency calculation task, each plurality of tuples in each frequency calculation task is ranked to obtain a first low-order plurality of tuples, and the probability of the first low-order plurality of tuples is calculated according to the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, the joint probability of the plurality of tuples is obtained according to the probability of the first low-order plurality of tuples and the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, a second low-order plurality of tuples is obtained by ranking again, and the probability of the second low-order plurality of tuples is calculated according to the intermediate probability of the plurality of tuples; for each plurality of tuples in each frequency calculation task, the back-off probability of the second low-order plurality of tuples is obtained according to the probability of the second low-order plurality of tuples and the joint probability of the plurality of tuples. The method further comprises: obtaining a plurality of frequency calculation tasks, and calculating the frequency of a plurality of tuples in each training corpus data in each of the frequency calculation tasks; counting the frequency of the plurality of tuples in all training corpus data in each of the frequency calculation tasks to obtain the pre-accumulation frequency of the plurality of tuples in each frequency calculation task; counting the pre-accumulation frequency of the plurality of tuples in all frequency calculation tasks to obtain the accumulation frequency of the plurality of tuples, and calculating the probability of each plurality of tuples according to the accumulation frequency of the plurality of tuples; the probability of the plurality of tuples is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing, and is obtained by back-off processing of the intermediate probability of each plurality of tuples obtained by the accumulation frequency of the plurality of tuples and the discount factor during the pre-set smoothing processing. and according to the accumulation frequency of the plurality of tuples, the probability of each plurality of tuples is obtained, which specifically comprises: obtaining a plurality of frequency calculation tasks, wherein each frequency calculation task includes a discount factor during pre-set smoothing processing; for each frequency calculation task, using a smoothing processing function and the discount factor to process the accumulation frequency of the plurality of tuples to obtain the intermediate probability of each plurality of tuples; for each frequency calculation task, the intermediate probability of each plurality of tuples is back-off processed, and the probability of the plurality of tuples is output. The processing module is further configured to: count the pre-accumulation frequencies of the multiple tuples in all frequency calculation tasks to obtain accumulation frequencies of the multiple tuples, and calculate probabilities of each multiple tuple according to the accumulation frequencies of the multiple tuples; and the probability of each multiple tuple is obtained by performing back-off processing on intermediate probabilities of each multiple tuple obtained by multiplying the accumulation frequencies of the multiple tuples by discount coefficients in the pre-set smoothing processing, and then obtaining low-order multiple tuple probabilities in stages and calculating joint probabilities.

7. The apparatus of claim 6, wherein, The processing module is specifically configured to: obtain a plurality of frequency calculation tasks, wherein each frequency calculation task includes discount coefficients in pre-set smoothing processing; for each frequency calculation task, use a smoothing processing function and the discount coefficients to process the accumulation frequencies of the multiple tuples to obtain intermediate probabilities of each multiple tuple; for each frequency calculation task, perform back-off processing on the intermediate probabilities of each multiple tuple to output the probabilities of the multiple tuples.

8. An electronic device comprising: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 5.

Images