Data processing methods and apparatus

Abstract

This application provides a data processing method and apparatus, relating to the field of artificial intelligence technology, including: segmenting the acquired text to be predicted to obtain a sequence of words to be predicted; iteratively processing each segment in the sequence of words to be predicted, performing the following steps respectively: obtaining the semantic adjacency probability of the first segment in the sequence of words to be predicted with each preset segment in a preset dictionary, and the original adjacency probability of the first segment with the adjacent second segment; if the original adjacency probability is less than a probability threshold, selecting multiple candidate segments from a preset dictionary based on preset filtering conditions and the second segment; using the semantic adjacency probability of the preset segment that matches the candidate segment as the candidate adjacency probability, and using the candidate segment corresponding to the highest candidate adjacency probability as the target segment, replacing the second segment in the sequence of words to be predicted with the target segment; to obtain the final target text of the text to be predicted. This improves the accuracy and efficiency of data cleaning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus. Background Technology

[0002] Data cleaning refers to the process of identifying and correcting identifiable errors in data files. It is typically done by computers rather than humans and includes checking data consistency and handling invalid and missing values. Generally, data in a data warehouse is a collection of historical data extracted from multiple business systems and focused on a specific theme. Therefore, it's unavoidable to have duplicate, erroneous, or conflicting data that doesn't meet requirements. The task of data cleaning is to remove this non-compliant data to obtain valuable data, which can be used for data statistics, model training, and other purposes.

[0003] In related technologies, Cantonese translation applications typically rely on Cantonese language models to convert Cantonese into standard Mandarin, or vice versa. Therefore, a large amount of high-quality Cantonese text data is often needed to train these models and improve their Cantonese processing performance. In other words, the development of Cantonese translation applications often involves a significant demand for high-quality Cantonese text data. There are two main methods for obtaining high-quality Cantonese text: one is manual data cleaning, which is labor-intensive and time-consuming; the other uses rule-based correction methods, which are prone to errors and cannot remove all "dirty" data.

[0004] Therefore, there is an urgent need to redesign a data processing method and overcome the aforementioned shortcomings. Summary of the Invention

[0005] This application provides a data processing method and apparatus to improve the accuracy and efficiency of data cleaning.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0007] The obtained text to be predicted is segmented into words to obtain a sequence of words to be predicted;

[0008] For each word segmentation iteration in the sequence of words to be predicted, the following steps are performed:

[0009] Obtain the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset dictionary, and the original adjacency probability of the first word segment with the second word segment, wherein the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted;

[0010] If the original adjacency probability is less than the probability threshold, at least one candidate word is selected from the preset dictionary based on the preset filtering conditions and the second word segmentation.

[0011] The semantic adjacency probability of the preset word that matches the candidate word is taken as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is taken as the target word. The second word in the word sequence to be predicted is replaced by the target word.

[0012] The text corresponding to the processed word sequence to be predicted is used as the target text.

[0013] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0014] The word segmentation unit is used to segment the acquired text to be predicted to obtain a sequence of words to be predicted.

[0015] The processing unit is used to iteratively process each word segment in the word sequence to be predicted, and performs the following steps respectively:

[0016] Obtain the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset dictionary, and the original adjacency probability of the first word segment with the second word segment, wherein the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted;

[0017] If the original adjacency probability is less than the probability threshold, at least one candidate word is selected from the preset dictionary based on the preset filtering conditions and the second word segmentation.

[0018] The semantic adjacency probability of the preset word that matches the candidate word is taken as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is taken as the target word. The second word in the word sequence to be predicted is replaced by the target word.

[0019] The processing unit is further configured to use the text corresponding to the processed word sequence to be predicted as the target text.

[0020] Optionally, the processing unit is specifically used for:

[0021] Obtain the first intermediate word sequence and the second intermediate word sequence from the word sequence to be predicted;

[0022] The first intermediate word sequence is the word sequence of each word in the word sequence to be predicted that precedes the first word segment and contains the first word segment; the second intermediate word sequence is the word sequence in the word sequence to be predicted that contains the first intermediate word sequence and the second word segment.

[0023] The first intermediate word sequence is concatenated with each preset word segment in the preset word library to obtain the concatenated word sequence corresponding to each preset word segment.

[0024] Based on the attention mechanism, the first semantic association between the first segment and the preset segment in the concatenated word sequence, and the second semantic association between the first segment and the second segment in the second intermediate word sequence are extracted. The semantic adjacency probability is obtained based on the first semantic association, and the original adjacency probability is obtained based on the second semantic association.

[0025] Optionally, the processing unit is specifically used for:

[0026] Based on a Markov process, the largest candidate adjacency probability is determined from the at least one candidate adjacency probability.

[0027] The candidate word corresponding to the largest candidate adjacency probability is taken as the target word.

[0028] Optionally, the processing unit is further configured to:

[0029] If the original adjacency probability is not less than the probability threshold, the second word segment in the word sequence to be predicted is retained.

[0030] Optionally, the processing unit is further configured to:

[0031] If the second word does not belong to a fixed language, then based on the translation tool, the second word is translated into a second word that belongs to a fixed language.

[0032] Optionally, the method is performed by a target language model, and the processing unit is further configured to: during the processing of each word segment in the sequence of words to be predicted:

[0033] If the original adjacency probability is not less than the probability threshold, the parameters of the target language model are adjusted based on the first reward function;

[0034] If the original adjacency probability is less than the probability threshold, and the second word segmentation belongs to a fixed language, the parameters of the target language model are adjusted based on the second reward function;

[0035] If the original adjacency probability is less than the probability threshold, and the second word segmentation does not belong to a fixed language, the parameters of the target language model are adjusted based on the third reward function.

[0036] Optionally, the method is performed through a target language model, and the processing unit is further configured to: perform unsupervised training on the self-attention mechanism in the target language model using an unlabeled text set, including:

[0037] Each unlabeled text in the unlabeled text set is segmented to obtain an unlabeled word sequence, and the unlabeled word sequence is then segmented and corrupted to obtain a corrupted word sequence.

[0038] Based on the self-attention mechanism, the unlabeled semantic association between each word segment in the unlabeled word sequence and the damaged semantic association between each word segment in the damaged word sequence are obtained respectively. The unlabeled adjacency probability is obtained based on the unlabeled semantic association and the damaged adjacency probability is obtained based on the damaged semantic association.

[0039] The parameters of the self-attention mechanism are adjusted based on the difference between the unlabeled adjacency probability and the damaged adjacency probability.

[0040] Optionally, the word segmentation unit is further configured to: segment each unlabeled text in the unlabeled text set into words to obtain each preset word segmentation;

[0041] The processing unit is also used to: establish the preset word library based on the preset word segments.

[0042] Optional, also includes:

[0043] The text to be predicted and the unlabeled text are Cantonese texts.

[0044] Thirdly, an embodiment of this application provides a computer device including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs any of the data processing methods described in the first aspect above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium comprising a computer program, wherein when the computer program is run on a computer device, the computer program is used to cause the computer device to perform any of the data processing methods described in the first aspect above.

[0046] Fifthly, an embodiment of this application provides a computer program product, the computer program product including a computer program stored in a computer-readable storage medium; when a processor of a computer device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the computer device to perform any of the data processing methods described in the first aspect above.

[0047] The beneficial effects of this application are as follows:

[0048] In the data processing method provided in this application embodiment, each segmentation in the word sequence to be predicted is iteratively processed to obtain the original adjacency probability between the first segmentation and the adjacent second segmentation in the word sequence to be predicted, as well as the semantic adjacency probability between the first segmentation and each preset segmentation in the preset dictionary. That is, based on the semantic information of the first segmentation and the second segmentation, the probability that the first segmentation and the second segmentation are adjacent in a sentence text is obtained, and based on the semantic information of the first segmentation and the preset segmentation, the probability that the first segmentation and the preset segmentation are adjacent in a sentence text is obtained.

[0049] In this way, the relationship between the original adjacency probability and the probability threshold can be determined. If the original adjacency probability is less than the probability threshold, it indicates that the semantic information connection between the first and second segmented words is not smooth, and the second segmented word may be an incorrect word adjacent to the first segmented word. Then, based on the second segmented word and preset filtering conditions, preset segmented words that are homophones, synonyms, or similar to the second segmented word can be selected from a preset dictionary as candidate segmented words. Since the semantic adjacency probabilities between the first segmented word and each preset segmented word have already been obtained, the semantic adjacency probability of the preset segmented word that is the same as the candidate segmented word is the probability that the first segmented word and the candidate segmented word are adjacent in a sentence. Therefore, the semantic adjacency probability of the preset segmented word that matches the candidate segmented word is used as the candidate adjacency probability, and the candidate segmented word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target segmented word. Thus, the target segmented word is most likely the correct segmented word adjacent to the first segmented word, and the target segmented word is used to replace the second segmented word in the word sequence to be predicted.

[0050] The target text obtained after performing the above processing on each word in the word sequence to be predicted is the correct text. It can be seen that, compared to related technologies that use manual error correction or rule-based correction for data cleaning, the natural language processing method used in this application can automatically and quickly perform data cleaning, saving labor costs, shortening the data cleaning cycle, and improving the accuracy of data cleaning. Because erroneous text to be detected is corrected to obtain correct text, the amount of data obtained after data cleaning is also guaranteed.

[0051] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 An optional schematic diagram illustrating an application scenario provided in this application embodiment;

[0054] Figure 2 An optional schematic diagram illustrating an application scenario provided in this application embodiment;

[0055] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0056] Figure 4 This application provides a schematic flowchart of a method for obtaining the original adjacency probability and semantic understanding probability in an embodiment of the present application.

[0057] Figure 5 This is a schematic diagram of a self-attention mechanism structure provided in an embodiment of this application;

[0058] Figure 6 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0061] It is understood that, in the following specific embodiments of this application, the data related to the text to be detected and the unlabeled text, etc., require relevant licenses or consents when the various embodiments of this application are applied to specific products or technologies, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, when relevant data is needed, relevant volunteers can be recruited and relevant agreements authorizing the volunteers to authorize data can be signed, and then the data of these volunteers can be used for implementation; or, implementation can be carried out within the authorized scope of an organization, and the following implementation methods can be implemented using the data of the organization's internal members to make relevant recommendations to internal members; or, the relevant data used in the specific implementation are all simulated data, such as simulated data generated in a virtual scene.

[0062] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0063] A Markov process is a class of stochastic processes and an important method for studying the state space of discrete-event dynamic systems. It is a stochastic process in the state space that transitions from one state to another. This process requires "memorylessness," meaning that the probability distribution of the next state is determined solely by the current state; previous events in the time series are irrelevant. A Markov process assumes that all past information is preserved in the current state. For example, suppose the existing sequence states are...X t-2 X t-1 X t X t+1 ..., then in X t+1 The conditional probability of a state at a given moment depends only on the state X at the previous moment. t That is: P(X) t+1 |…X t-2 X t-1 X t )=P(X t+1 |X t )

[0064] If the probability of a state transition at a certain moment depends only on its previous state, then as long as the transition probability between any two states in the system can be calculated, the Markov process can be determined. It can be seen that its mathematical basis is the theory of stochastic processes.

[0065] BERT (Transformer-based Bidirectional Encoder Representation): A pre-trained model where "bidirectional" means that when processing a word, the model can utilize information from both the preceding and following words. It randomly masks some words and uses all the unmasked words for prediction.

[0066] A feedforward neural network (FNN) is a type of artificial neural network. It employs a unidirectional, multi-layered structure. Each layer contains several neurons. In this type of neural network, each neuron receives signals from the neurons in the previous layer and generates an output for the next layer. Layer 0 is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. Hidden layers can be single or multiple. There is no feedback throughout the network; signals propagate unidirectionally from the input layer to the output layer, extracting different features from each layer before finally outputting the result.

[0067] An activation function is a function that runs on neurons in an artificial neural network. It maps the input of a neuron to its output, determining what is emitted from the previous neuron to the next. Activation functions include the Softmax function, the Tanh function, and the ReLU function.

[0068] The Tanh function is one of the hyperbolic functions. Tanh() represents the hyperbolic tangent and is a saturating activation function in neural networks, which can alleviate the vanishing gradient phenomenon. In mathematics, the hyperbolic tangent "Tanh" is derived from the basic hyperbolic functions hyperbolic sine and hyperbolic cosine.

[0069] The Softmax function is a common sigmoid function in biology, also known as the sigmoid growth curve. It is a saturation activation function in neural networks. In information science, due to its monotonically increasing and inverse monotonically increasing properties, the sigmoid function is often used as an activation function for neural networks, mapping variables to the range of 0 and 1.

[0070] The ReLU function (Rectified Linear Unit) is a commonly used neural activation function. It's essentially a piecewise linear function that sets all negative values ​​to 0 while leaving positive values ​​unchanged; this operation is known as one-sided inhibition. The ReLU function has the advantage of sparsity, allowing sparse models to better extract relevant features and fit training data. In the x>0 region, it avoids gradient saturation and vanishing gradient problems. It has low computational complexity, requiring no exponential operations; a threshold is sufficient to obtain the activation value. The disadvantage is that the output is not symmetric to 0. Since the ReLU activation function value is 0 and the gradient is 0 when the value is less than 0, some neurons will never be updated.

[0071] Self-attention, a core component of BERT, enables the machine to notice the correlations between different parts of the input. It addresses situations where a neural network receives many vectors of varying sizes as input, with relationships between them. However, during training, the network often fails to fully leverage these relationships, resulting in poor model performance. Examples include machine translation, part-of-speech tagging, and semantic analysis. Self-attention addresses the problem of fully connected neural networks failing to establish correlations between multiple related inputs, allowing the machine to notice the relationships between different parts of the input. One simple implementation involves multiple input vectors, and the following steps are performed on each of these vectors:

[0072] The self-attention mechanism includes a weight matrix W. q Weight matrix W k Weight matrix W v For an input vector X1, calculate X1*W respectively. q =Q1, X1*W k =K1,X1*W v =V1, obtain this input vector and compare it with other input vectors (X2, X3...X). n The correlation between the input vectors is calculated to obtain the correlation matrix QK1. A Softmax or ReLU operation is then performed on this correlation matrix to obtain the correlation representation matrix QK1'. Further, based on the meaning of the input vectors, Z1 = QK1' * V1 is obtained. Thus, the output Z1 contains both the meaning of the input vector X1 itself and the meaning of X1 with other input vectors (X2, X3…X…). n The correlation between the input vectors (X1, X2…X) in the original data space. Based on the foregoing, it can be understood that the essence of the self-attention mechanism is to transform the input vectors (X1, X2…X…X) in the original data space through nonlinear transformation. n The position of point X in the input matrix is ​​mapped to point Z in the output matrix of the output result in the new space.

[0073] The technical solutions in this application relate to artificial intelligence, natural language processing, and machine learning technologies. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.

[0074] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0075] Big data technology is a solution for processing and storing massive amounts of data, encompassing multiple stages including data acquisition, preprocessing, storage, computation, and analysis. "Big data" refers to massive amounts of data, containing vast and complex information, the scale of which is too large for mainstream software tools to capture within a reasonable timeframe. Big data acquisition typically involves obtaining data from various data sources, such as social media, enterprise databases, and log files. Data preprocessing is a crucial step in big data processing, including data cleaning, integration, and transformation to ensure data quality and accuracy. Data storage generally involves traditional relational database technologies and newer database technologies, such as NoSQL databases, columnar storage databases, and distributed file systems. Data computation and analysis involve various technologies such as distributed computing and stream processing. Among the aforementioned stages of big data technology, data mining and machine learning are also involved, enabling the discovery of valuable data and patterns from massive datasets, as well as the prediction of future trends. Commonly used machine learning algorithms include classification, clustering, regression, and prediction, applicable to various scenarios such as recommendation systems, anomaly detection, and predictive models.

[0076] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs. Based on big data and NLP technologies, it can perform data cleaning and correction on large amounts of data. For example, it can correct batches of input text to be predicted by replacing incorrect second-order words in the text with the correct target word segmentation to obtain the correct target text, and then obtain batches of correct target text. In areas with limited open-source data, this processing method can yield rich data information.

[0077] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0078] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0079] The solutions provided in this application involve artificial intelligence technologies such as natural language processing, data processing, and machine learning.

[0080] In this embodiment, machine learning techniques are applied to train the target language model. During training, the self-attention mechanism in the target language model is iteratively trained in unsupervised mode based on batches of unlabeled text from a collection of unlabeled texts. Big data technology is used to analyze the unlabeled texts to obtain the unlabeled semantic relationships between adjacent word segments. Based on these semantic relationships, the parameters of the self-attention mechanism are adjusted. This results in a self-attention mechanism capable of initially and autonomously analyzing the probabilities of adjacency between adjacent word segments. This ensures that the probabilities of adjacency between adjacent word segments obtained during subsequent unsupervised training of the target language model have a preliminary basis for accuracy, guaranteeing its applicability.

[0081] In addition, it should be noted that the artificial neural network model in the embodiments of this application can be trained online or offline, and no specific limitation is made here. This article uses offline training as an example for illustration.

[0082] The design concept of the embodiments of this application is briefly introduced below:

[0083] In the field of artificial intelligence technology, to meet the needs of life and work, there are more and more applications with text recognition and translation, such as multilingual translation applications, Cantonese and common standard Chinese translation applications, etc. Among them, in Cantonese and common standard Chinese translation applications, a Cantonese language model is usually used for text analysis and processing to perform translation processing between Cantonese and common standard Chinese. For example, when inputting the Cantonese text "呢個天时话唔埋㗎,话变就变" or the common standard Chinese text "这种天气很难捉摸,说变就变", the Chinese text "这种天气很难捉摸,说变就变" or "呢個天时话唔埋㗎,话变就变" can be accurately output.

[0084] In related technologies, in order to improve the Cantonese processing performance of the Cantonese language model, a large amount of high-quality data needs to be collected for training so that the Cantonese language model can accurately translate Cantonese and Chinese. However, during the data collection process, there is a lack of open-source resources related to Cantonese, resulting in insufficient data to support model training. And even though social media can provide a large amount of data, there are also problems such as uneven data quality, mixed Chinese and English, and homophone substitution. Therefore, it is necessary to clean the social media data. Data cleaning usually adopts two methods. One is the manual correction method, which requires a large amount of manpower and time. The other is the rule correction method, which requires predefined rules and may not cover all text error types.

[0085] In view of this, the embodiments of the present application provide a data processing method, device, computer device, and storage medium. In the data processing method, for each word segment arranged in sequence in the text to be predicted, it is sequentially detected whether the semantic information connection between adjacent word segments is smooth. If there is a non-smooth semantic information connection between adjacent word segments (the first word segment sorted in the front and the second word segment sorted in the back), a target word segment that is smoothly semantically connected to the first word segment is selected from the preset dictionary, and the non-smooth second word segment is replaced with the target word segment. In this way, both data cleaning can be achieved, and "dirty data" can be transformed into valuable data.

[0086] The following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0087] As Figure 1 shown, it is a schematic diagram of the application scenario of the embodiments of the present application. The application scenario diagram includes as Figure 1The diagram illustrates an application scenario of this application. The scenario includes any one of multiple servers, 110 and 120. Server 110 can collect data from business systems, social media, or other related systems or platforms and send the collected data to server 120. After receiving the collected data, server 120 processes each piece of text to be detected within the collected data to obtain the target text, thereby acquiring a large amount of high-value data. Alternatively, either server 110 or server 120 can both collect and process the data.

[0088] Based on this application scenario, this application provides another application scenario illustration, such as... Figure 2 As shown, any one of the multiple terminal devices 210 and any one of the multiple servers 220. Terminal device 210 can be used to upload the text to be detected, and server 220 receives the text to be detected, processes the data, and returns the target text of the text to be detected to terminal device 210.

[0089] In the above Figure 1 and Figure 2 In the application scenarios, terminal devices 210 include, but are not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. These terminal devices may have clients related to data processing services installed. These clients can be software (such as browsers, communication software, etc.), web pages, or mini-programs. Servers 110, 120, and 220 are backend servers corresponding to the software, web pages, or mini-programs, or backend servers specifically used to return processing results of data processing services to the client. This application does not impose specific limitations on these servers. Servers 110, 120, and 220 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0090] It should be noted that, Figure 1 and Figure 2 The examples shown are merely illustrative; in reality, the number of terminal devices and servers, as well as the communication methods, are not limited and are not specifically restricted in the embodiments of this application.

[0091] In addition, the data processing method of the embodiments of the present application can be applied to various scenarios, such as translation applications, document correction applications, cross-language conversation applications, etc., or for cleaning massive data to obtain training data for relevant language models in these applications, etc. The application scenarios of the data processing method are not specifically limited here.

[0092] Next, in combination with the above-described application scenarios, the data processing method provided by the exemplary embodiments of the present application will be described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not limited in this regard.

[0093] See Figure 3 As shown, it is a flowchart of the data processing method provided by the embodiments of the present application. Here, an example is given with the server as the execution subject. The specific implementation process of this method is as follows:

[0094] Step 301: Perform word segmentation on the obtained text to be predicted to obtain a sequence of words to be predicted.

[0095] In one embodiment, the text to be predicted can be text data in literature or social media.

[0096] In one embodiment, the text to be predicted can be Cantonese text. For example, a Cantonese dictionary can be collected through legal channels, covering relevant information such as standard writing forms and pronunciation phonetic notations. For example, various data released by news media can be collected, including the text content of articles. For example, social media data can be collected, including main post content, comments, replies, and other relevant information. Correspondingly, the word segmentation in the preset dictionary and the preset word segmentation in the preset word library can also be Cantonese word segmentation.

[0097] In one embodiment, if the massive data is obtained from platforms such as news media and social media, in order to ensure the effectiveness and accuracy of subsequent data processing and analysis, the massive data can be screened by language type based on a language type classification algorithm to obtain a batch of Cantonese texts to be predicted.

[0098] In one embodiment, when the text to be predicted is "你食咗饭未?", performing word segmentation on this text to be predicted can obtain a sequence of words to be predicted "你,食咗,饭,未,?". The example here is only used to clearly illustrate the word segmentation of the text to be predicted, and does not limit the text to be predicted and its word segmentation situation.

[0099] Step 302: Iteratively process each word segmentation in the sequence of words to be predicted, and respectively perform the following steps:

[0100] Step 3021: Obtain the semantic adjacency probabilities between the first word segment in the word sequence to be predicted and each preset word segment in the preset word library, as well as the original adjacency probability between the first word segment and the second word segment, where the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted.

[0101] In one embodiment, based on the above embodiment, if the first word segment is "你" (you), the second word segment is "食咗" (have eaten); if the first word segment is "食咗" (have eaten), the second word segment is "饭" (meal), and so on. The second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted.

[0102] In one embodiment, the semantic adjacency probability refers to the probability that the first word segment is followed by a preset word segment, and the original adjacency probability refers to the probability that the first word segment is followed by the second word segment. For example, if the first word segment is "天气" (weather), the probabilities of the following word segments being "好" (good), "坏" (bad), "晴朗" (sunny) are greater than the probabilities of "吃" (eat), "睡" (sleep).

[0103] In one embodiment, methods such as support vector machines, Word2Vec, etc. can be used to obtain the semantic information between the first word segment and the second word segment, calculate the original adjacency probability between the first word segment and the second word segment according to the semantic information, and obtain the semantic information between the first word segment and the preset word segment, and calculate the semantic adjacency probability between the first word segment and the preset word segment according to the semantic relationship. Among them, there are various methods for obtaining the original adjacency probability and semantic adjacency probability. For example, neural network methods can also be used. The related technologies involved in these methods belong to relatively mature technologies and will not be elaborated here.

[0104] In one embodiment, the preset word library can be composed of a large number of preset word segments collected in advance, and the preset word segments can be obtained from relevant data sources such as literature materials and social media.

[0105] Step 3022: If the original adjacency probability is less than the probability threshold, select at least one candidate word segment from the preset dictionary based on the preset screening conditions and the second word segment.

[0106] In one embodiment, the original adjacency probability is the adjacency probability between the first word segment and the second word segment. If the original adjacency probability is less than the probability threshold, it can be considered that the connection between the first word segment and the second word segment is not smooth. Then, at least one candidate word segment that can replace the second word segment needs to be reselected from the preset dictionary to find the target word segment that is smoothly adjacent to the first word segment from the at least one candidate word segment.

[0107] In one embodiment, the preset filtering conditions may include: selecting word segments from a preset dictionary that are homophones of the second word as candidate word segments, selecting word segments from a preset dictionary that have similar stroke counts to the second word as candidate word segments, etc. In other words, the preset filtering conditions may include at least one filtering condition among selecting word segments from a preset dictionary that are homophones of the second word as candidate word segments and selecting word segments from a preset dictionary that have similar forms to the second word as candidate word segments.

[0108] Step 3023: Take the semantic adjacency probability of the preset word that matches the candidate word as the candidate adjacency probability, and take the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target word, and replace the second word in the word sequence to be predicted with the target word.

[0109] In one embodiment, after obtaining at least one candidate word that may replace the second word, to obtain the candidate adjacency probability between the first word and the candidate word, since the semantic adjacency probability between the first word and each preset word is known in the previous steps, the candidate word can be matched with the preset words to find the preset word that is the same as the candidate word. Then, the semantic adjacency probability of the preset word that is the same as the candidate word can be used as the candidate adjacency probability. Subsequently, the candidate word corresponding to the highest candidate adjacency probability can be selected as the target word, and the target word is used to replace the second word in the word sequence to be predicted.

[0110] In one embodiment, if the maximum candidate adjacency probability is still less than the probability threshold, the second word can be directly deleted and the next loop can be entered. The original adjacency probability between the first word and the third word adjacent to the second word is re-determined. If the original adjacency probability between the first word and the third word is not less than the probability threshold, the first word and the third word are adjacent; otherwise, the text to be predicted is discarded.

[0111] In one embodiment, if the original adjacency probability is not less than the probability threshold, the second segment of the word sequence to be predicted is retained. Then, the next loop is entered, and the second segment of the current loop is used as the first segment, and the text corresponding to the processed word sequence to be predicted in step 303 of the current loop is used as the target text.

[0112] In one embodiment, after the processing of steps 301 and 302 above, the adjacency probabilities between adjacent words in the processed word sequence to be predicted are all greater than the probability threshold. In other words, the semantic connections between adjacent words in the word sequence to be predicted are fluent, and it can be used as the target text output.

[0113] In the above method, the original adjacency probability between the first segment and the adjacent second segment in the word sequence to be predicted, as well as the semantic adjacency probability between the first segment and each preset segment in the preset dictionary, are obtained. The relationship between the original adjacency probability and the probability threshold is then determined. If the original adjacency probability is less than the probability threshold, it indicates that the semantic information connection between the first and second segments is not smooth, and the second segment may be an incorrect word adjacent to the first segment. Subsequently, preset segments that are homophones, similar in form, or otherwise related to the second segment can be selected from the preset dictionary as candidate segments based on the second segment and preset filtering conditions. Since the semantic adjacency probability between the first segment and each preset segment has already been obtained, the semantic adjacency probability of the preset segment that is the same as the candidate segment is the probability that the first segment and the candidate segment are adjacent in a sentence. Therefore, the semantic adjacency probability of the preset segment that matches the candidate segment is used as the candidate adjacency probability, and the candidate segment corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target segment. Thus, the target segment is most likely the correct segment immediately following the first segment, and the target segment replaces the second segment in the word sequence to be predicted. Compared to related technologies that use manual error correction or rule-based correction for data cleaning, the natural language processing method used in this application can automatically and quickly clean the data, saving labor costs, shortening the data cleaning cycle, eliminating the need for preset rules, flexibly responding to different data anomalies, and improving the accuracy of data cleaning. Because it corrects erroneous text to be detected to obtain correct text, it also ensures the amount of data obtained after data cleaning.

[0114] Based on the above Figure 3 The data processing method in this application provides a method for obtaining the original adjacency probability and semantic understanding probability, such as... Figure 4 As shown, in step 3021 above, obtaining the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset vocabulary, and the original adjacency probability of the first word segment with the second word segment, includes:

[0115] Step 401: Obtain the first intermediate word sequence and the second intermediate word sequence in the word sequence to be predicted; the first intermediate word sequence is the word sequence of each word before and containing the first word in the word sequence to be predicted, and the second intermediate word sequence is the word sequence in the word sequence to be predicted that contains the first intermediate word sequence and the second word.

[0116] In one embodiment, based on the word sequence to be predicted in the above embodiment: "you, eat, rice, not yet, ?", if the first word segment is "eat", then the second word segment is "rice", then the first intermediate word sequence is "you, eat", and the second intermediate word sequence is "you, eat, rice".

[0117] Step 402: Concatenate the first intermediate word sequence with each preset word segment in the preset word library to obtain the concatenated word sequences corresponding to each preset word segment respectively.

[0118] In one embodiment, based on the first intermediate word sequence in the above embodiment being "你,食咗" (which might be something like "you, have eaten" in a rough approximation), assuming the preset word library contains preset word segments such as "铁" (iron), "金属" (metal), "水果" (fruit), "包子" (steamed bun), etc., then the first intermediate word sequence is concatenated with each preset word segment in the preset word library to obtain multiple concatenated word sequences. For example, the concatenated word sequences: "你,食咗,铁" (you, have eaten, iron), "你,食咗,金属" (you, have eaten, metal), "你,食咗,水果" (you, have eaten, fruit), "你,食咗,金属" (you, have eaten, metal), "你,食咗,包子" (you, have eaten, steamed bun), etc.

[0119] Step 403: Based on the attention mechanism, extract the first semantic association relationship between the first word segment and the preset word segment in the concatenated word sequence, and the second semantic association relationship between the first word segment and the second word segment in the second intermediate word sequence, and obtain the semantic adjacency probability according to the first semantic association relationship, and obtain the original adjacency probability according to the second semantic association relationship.

[0120] In one embodiment, based on the above embodiment, the obtained second intermediate word sequence and each concatenated word sequence are respectively input into the self-attention mechanism to obtain the first semantic association relationship between the first word segment and the adjacent preset word segment in the concatenated word sequence, and the second semantic association relationship between the first word segment and the second word segment in the second intermediate word sequence. As introduced above about the self-attention mechanism, the self-attention mechanism can retain the meaning expressed by the vectors and can also obtain the correlation between the vectors. That is to say, the self-attention mechanism can retain the meaning of each word segment in the second intermediate word sequence and can also obtain the second semantic association relationship between each word segment. Similarly, the self-attention mechanism can retain the meaning of each word segment in the concatenated word sequence and can also obtain the first semantic association relationship between each word segment. Thus, based on the first semantic association relationship, the semantic adjacency probability between the first word segment and the preset word segment can be obtained to express the semantic fluency between the first word segment and the preset word segment. Similarly, based on the second semantic association relationship, the original adjacency probability between the first word segment and the second word segment can be obtained to express the semantic fluency between the first word segment and the second word segment.

[0121] In one embodiment, the self-attention mechanism here can be multi-layer, such as Figure 5 As shown, it is a schematic diagram of a self-attention mechanism structure provided by an embodiment of the present application. The self-attention mechanism can adopt a feed-forward neural network and an activation function to improve the characteristic of the semantic features of the extracted word segments, optimize the correlation between the semantic features, and ensure the accuracy of the output result. Among them, the number of layers of the mechanism can be set as needed and is not specifically limited here.

[0122] Based on the above Figure 3 or Figure 4 The method in, or Figure 5 The self-attention mechanism structure in the above step 3023 involves selecting the candidate word segment corresponding to the candidate with the highest candidate adjacency probability from at least one candidate adjacency probability as the target word segment, including:

[0123] Based on Markov processes, the candidate with the largest adjacency probability is determined from at least one candidate adjacency probability; the candidate word corresponding to the largest candidate adjacency probability is taken as the target word.

[0124] In one embodiment, during the iterative processing, for each segment in the word sequence to be predicted, if the original adjacency probability between the first segment in the current loop and its adjacent second segment is less than the probability threshold, then the candidate segment with the highest candidate adjacency probability can be selected from the candidate segments and replaced with the target segment. In other words, the Markov process only focuses on the semantic adjacency probability (state transition probability) between the current segment (current state) and the next segment (next state), and only seeks to obtain the maximum semantic adjacency probability (state transition probability).

[0125] In one embodiment, a Markov process is combined with the aforementioned self-attention mechanism to obtain a target language model. When the input text to be predicted is used, the semantic relationships between segments in the word sequence to be predicted are obtained iteratively based on the self-attention mechanism. During this iterative process, while preserving the semantic relationships of the preceding ordered segments, the fluency of semantic connections between segments is obtained, i.e., the fluency of the text to be predicted. According to the Markov process, segments with non-fluid connections in the text to be predicted are corrected to segments with fluent semantic connections, thus outputting the correct target text. This method can be called a reinforcement learning-based data cleaning strategy. It models natural language text based on a Markov decision process, focusing on long-term rewards rather than short-term gains. That is, text correction focuses on the entire text rather than simple word correction, thereby better handling complex datasets.

[0126] In one embodiment, when the data processing method is applied to data cleaning, it may further include deleting the text to be predicted. For example, if more than a set percentage of the segments in the word sequence to be predicted have an original adjacency probability with adjacent segments that is less than a probability threshold, then the word sequence to be predicted can be deleted. Similarly, if the text to be predicted is semantically incomplete, it can be deleted, and so on. The specific rules for deleting text are not limited here and can be set as needed.

[0127] Based on the above Figure 3The method flow in this application embodiment provides a data processing method. In step 3022, before selecting at least one candidate word from a preset dictionary based on preset filtering conditions and the second word, the method further includes: if the second word does not belong to a fixed language, then translating the second word into a second word belonging to a fixed language based on a translation tool.

[0128] In one embodiment, if the original adjacency probability between the first and second word segments is less than a probability threshold, there is another possibility: the second word segment may be a non-fixed language, resulting in very low fluency between cross-language word segments. Therefore, if the second word segment does not belong to a fixed language, its language can be determined based on a language classification algorithm, and then translated into the fixed language using a translation tool. For example, if the fixed language is Cantonese, the second word segment may be English, Japanese, Korean, etc., and it can be translated into Cantonese using a translation tool.

[0129] In one embodiment, the words in the preset dictionary belong to a fixed language. The fixed language can be Standard Mandarin or Cantonese.

[0130] Based on the above methods, processes, and embodiments, this application provides a data processing method. This method is executed by a target language model, and in the processing of each word segment in the word sequence to be predicted:

[0131] If the original adjacency probability is not less than the probability threshold, adjust the parameters of the target language model based on the first reward function;

[0132] If the original adjacency probability is less than the probability threshold, and the second word segmentation belongs to a fixed language, the parameters of the target language model are adjusted based on the second reward function;

[0133] If the original adjacency probability is less than the probability threshold and the second word segment does not belong to a fixed language, the parameters of the target language model are adjusted based on the third reward function.

[0134] In one embodiment, a Markov process is combined with a self-attention mechanism to obtain the target language model. Unsupervised training is performed simultaneously during data processing; thus, the more the target language model is used, the higher the accuracy of data processing. It should be noted that the self-attention mechanism can also be replaced by support vector machines, neural networks, etc.; the self-attention mechanism used here is only to clearly illustrate the data processing method of this scheme.

[0135] In one embodiment, a target language model is obtained based on the combination of the above-described Markov process and self-attention mechanism, and then data processing is performed:

[0136] Step a: Segment the text to be predicted into words, resulting in a sequence of words x1, x2, x3, ..., x of length N.N We also obtain the state space S, which contains the strings: [x1], [x1,x2], [x1,x2,x3]...[x1,x2,x3…,x N-1 [x1,x2,x3…,x] N For each word segment in the sequence to be predicted, iterative processing is performed.

[0137] Step b: In the t-th loop (assuming the target language model does not have an initial character x0, then the t-th loop processes the t-th segment of the word sequence to be predicted; otherwise, if the target language model has an initial character x0, then the (t+1)-th loop processes the t-th segment of the word sequence to be predicted), the target language model obtains the t-th string x1, x2, x3…, x… T (state)s t ∈S, T≤F.

[0138] Step c, the action space of the target language model is denoted as A, for s t =x1,x2,x3…,x T x t The first segmentation, x t+1 For the second word segmentation, a preset dictionary is set in the action space A, P(x t+1 |s t ) represents the first word segmentation x obtained from the target language model. t Second participle x t+1 The original adjacency probability between them, where φ is the probability threshold;

[0139] First case: If P(x) t+1 |s t When )≥φ, the first segmentation x t Second participle x t+1 The semantic flow between them is smooth, and the intelligent system can then take action. t =x t+1 That is, after confirming the first word segmentation is adjacent to the second word segmentation, the environmental state of the target language model transitions to s. t+1 ∈S, confirm s t+1 = [x1,x2,x3…,x] t ,x t+1 A larger reward is obtained, and the parameters of the target language model are adjusted based on the first reward function, which is as follows:

[0140]

[0141] Where F is the length of the text to be predicted.

[0142] The second case: If P(x) t+1|s t When ) < φ, the first segmentation x t Second participle x t+1 The semantics between them are not fluent, and x t+1 When the language is a fixed language (assuming the fixed language is Cantonese), the AI ​​will select at least one candidate word from a preset dictionary (Cantonese dictionary) Z based on preset filtering conditions. The semantic adjacency probability of the preset word segment that is the same as the candidate word segment is used as the candidate adjacency probability, and the candidate word segment corresponding to the highest candidate adjacency probability is selected. As target word segmentation And replace the second segment x t+1 The environmental state transition of the obtained target language model is s t+1 ∈S, confirm A reward is then obtained, and the parameters of the target language model are adjusted based on the second reward function, which is as follows:

[0143]

[0144] Where F is the length of the text to be predicted.

[0145] The third case: If P(x) t+1 |s t When ) < φ, the first segmentation x t Second participle x t+1 The semantics between them are not fluent, and x t+1 Not a fixed language (x) t+1 When the language is English, Japanese, Korean, etc., the intelligent system will use translation tool C to translate the second participle x, which does not belong to a fixed language. t+1 Translated into the second participle of a fixed language Confirm x again t and The original adjacency probability of translation between them, if Then the first participle x t and the second participle belonging to fixed languages The semantics between them are fluent, As target word segmentation And replace the second segment x t+1 The environmental state transition of the obtained target language model is s t+1 ∈S, confirm And a reward is obtained, and the parameters of the target language model are adjusted based on the third reward function, which is as follows:

[0146]

[0147] Where F is the length of the text to be predicted.

[0148] Additionally, if P(c i |s t If ) < φ, then as in the second case, synonyms are selected from the preset dictionary as candidate segmentation words. The candidate segmentation word with the highest candidate adjacency probability is then selected as the target segmentation word and replaced with the second segmentation word. (Z) And in C All of these are words.

[0149] Step d: Adjust the parameters of the target language model based on the processing results of each of the above loops.

[0150] Specifically, during the data processing of the target language model, unsupervised training is performed simultaneously. The parameters of the target language model are adjusted based on the aforementioned first, second, and third reward functions. γ represents the weight of the impact of the decision action on the overall value in different cycles, that is, the importance used to balance the current reward and the future reward. γ∈[0,1] represents the decay coefficient or discount factor. T is the string length of the text to be predicted and the word sequence to be predicted, and t is the t-th word in the word sequence to be predicted.

[0151] In this model, a Markov process combined with a self-attention mechanism generates a string of length M based on the text to be predicted. During the iterative processing of the text to be predicted, the Markov process aims to maximize the probability P(w1,w2,…,w) of the word sequence. M According to the chain rule, the joint probability can be decomposed into a product of a series of conditional probabilities, such as:

[0152]

[0153] To optimize the objective function described above, the negative log-likelihood is typically minimized, transforming the product of joint probabilities into an addition operation, such as:

[0154]

[0155] We can set γ=1, so that the design of the reward model of the target language model takes into account the influence of the selected words in the current state on the selection of words in the future state.

[0156] In one implementation, the methods described in steps a to d above can be implemented using the Q-learning method.

[0157] Based on the above methods, processes, and embodiments, this application provides a model training method. The data processing method is executed through a target language model. Before segmenting the acquired text to be predicted to obtain the word sequence, an unlabeled text set is used to perform unsupervised training of the self-attention mechanism in the target language model, including:

[0158] Step 1: Perform word segmentation on each unlabeled text in the unlabeled text set to obtain an unlabeled word sequence, and then perform word segmentation corruption processing on the unlabeled word sequence to obtain a corrupted word sequence;

[0159] Step 2: Based on the self-attention mechanism, obtain the unlabeled semantic association between each word segment in the unlabeled word sequence and the damaged semantic association between each word segment in the damaged word sequence. Obtain the unlabeled adjacency probability based on the unlabeled semantic association and the damaged adjacency probability based on the damaged semantic association.

[0160] Step 3: Adjust the parameters of the self-attention mechanism based on the difference between the unlabeled adjacency probability and the damaged adjacency probability.

[0161] In one embodiment, a certain amount of unlabeled text can be collected in advance to obtain an unlabeled text set. Based on the unlabeled text set, a self-attention mechanism can initially learn the semantic correlation between word segments. Then, in the application process, the relatively accurate adjacency probability between each word segment of the text to be predicted can be given.

[0162] Based on the above model training method, this application provides a method for obtaining a preset lexicon, including: segmenting each unlabeled text in an unlabeled text set to obtain preset word segments; and establishing a preset lexicon based on each preset word segment. In other words, the unlabeled text in a high-quality unlabeled text set can be segmented, and the resulting words can be used as preset word segments to form a preset lexicon.

[0163] In one embodiment, the text to be predicted and the unlabeled text involved in the foregoing methods and embodiments are Cantonese text.

[0164] Based on the above methods, processes, and embodiments, this application provides a data processing method, such as... Figure 6 As shown, it includes:

[0165] Step 601: Collect news media text data and use a language classification algorithm to classify the news media text data by language. Error correction can be performed on the language-classified news media text data by combining rule correction and manual correction to obtain a high-quality unlabeled text set.

[0166] Step 602: Perform preliminary training on the self-attention mechanism in the target language model based on the unlabeled text set.

[0167] Step 603: Perform word segmentation on each unlabeled text in the unlabeled text set to obtain each preset word segmentation and establish a preset word library.

[0168] Step 604: Obtain new news media text data again, use a language classification algorithm to classify the new news media text data by language, obtain a batch of texts to be predicted, and perform word segmentation on each text to be predicted to obtain a sequence of words to be predicted.

[0169] Step 605: Process each word segment in the word sequence to be predicted in order. For each word segment in the word sequence to be predicted, perform the following steps:

[0170] Step 6051: Obtain the first intermediate word sequence and the second intermediate word sequence in the word sequence to be predicted; wherein, the first intermediate word sequence is the word sequence of each word in the word sequence to be predicted that precedes the first word segment and contains the first word segment, and the second intermediate word sequence is the word sequence in the word sequence to be predicted that contains both the first intermediate word sequence and the second word segment. The second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted.

[0171] Step 6052: Concatenate the first intermediate word sequence with each preset word segment in the preset word library to obtain the concatenated word sequence corresponding to each preset word segment.

[0172] Step 6053: Based on the attention mechanism, extract the first semantic association between the first segment and the preset segment in the concatenated word sequence, and the second semantic association between the first segment and the second segment in the second intermediate word sequence. Obtain the semantic adjacency probability based on the first semantic association and the original adjacency probability based on the second semantic association.

[0173] Step 6054: Is the original adjacency probability less than the probability threshold? If yes, proceed to step 6055; otherwise, proceed to step 6058.

[0174] Step 6055: Determine whether the second word does not belong to a fixed language. If so, proceed to step 6056; otherwise, proceed to step 6057.

[0175] Step 6056: Using a translation tool, translate the second word into a second word belonging to a fixed language. The translated second word can be used as the target word, replacing any second word not belonging to a fixed language. Alternatively, step 6057 can be executed: based on preset filtering conditions, select word segments from a preset dictionary that are semantically similar to the translated second word as candidate word segments. The semantic adjacency probability of preset word segments matching the candidate word is used as the candidate adjacency probability. The candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is selected as the target word, and the target word is used to replace the second word in the word sequence to be predicted.

[0176] Step 6057: Based on the preset filtering conditions and the second word segmentation, select at least one candidate word segmentation from the preset dictionary, take the semantic adjacency probability of the preset word segmentation that matches the candidate word segmentation as the candidate adjacency probability, and take the candidate word segmentation corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target word segmentation, and replace the second word segmentation in the word sequence to be predicted with the target word segmentation.

[0177] Step 6058: Retain the second segment of the text to be predicted.

[0178] Step 6059: Adjust the parameters of the target language model based on the processing results of the text to be predicted.

[0179] Step 606: Use the text corresponding to the processed word sequence to be predicted as the target text.

[0180] It should be noted that the above method flow steps are not unique. For example, step 602 can also be executed before step 603.

[0181] Based on the same concept, such as Figure 7 As shown, this application embodiment provides a data processing apparatus 700, which includes:

[0182] The word segmentation unit 701 is used to segment the acquired text to be predicted to obtain a sequence of words to be predicted.

[0183] Processing unit 702 is configured to perform the following steps for each word segment in the word sequence to be predicted:

[0184] Obtain the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset dictionary, and the original adjacency probability of the first word segment with the second word segment, wherein the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted;

[0185] If the original adjacency probability is less than the probability threshold, at least one candidate word is selected from the preset dictionary based on the preset filtering conditions and the second word segmentation.

[0186] The semantic adjacency probability of the preset word that matches the candidate word is taken as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is taken as the target word. The second word in the word sequence to be predicted is replaced by the target word.

[0187] The processing unit 702 is further configured to use the text corresponding to the processed word sequence to be predicted as the target text.

[0188] Optionally, the processing unit 702 is specifically used for:

[0189] Obtain the first intermediate word sequence and the second intermediate word sequence from the word sequence to be predicted;

[0190] The first intermediate word sequence is the word sequence of each word in the word sequence to be predicted that precedes the first word segment and contains the first word segment; the second intermediate word sequence is the word sequence in the word sequence to be predicted that contains the first intermediate word sequence and the second word segment.

[0191] The first intermediate word sequence is concatenated with each preset word segment in the preset word library to obtain the concatenated word sequence corresponding to each preset word segment.

[0192] Based on the attention mechanism, the first semantic association between the first segment and the preset segment in the concatenated word sequence, and the second semantic association between the first segment and the second segment in the second intermediate word sequence are extracted. The semantic adjacency probability is obtained based on the first semantic association, and the original adjacency probability is obtained based on the second semantic association.

[0193] Optionally, the processing unit 702 is specifically used for:

[0194] Based on the Markov decision process, the largest candidate adjacency probability is determined from the at least one candidate adjacency probability;

[0195] The candidate word corresponding to the largest candidate adjacency probability is taken as the target word.

[0196] Optionally, the processing unit 702 is further configured to:

[0197] If the original adjacency probability is not less than the probability threshold, the second word segment in the word sequence to be predicted is retained.

[0198] Optionally, the processing unit 702 is further configured to:

[0199] If the second word does not belong to a fixed language, then based on the translation tool, the second word is translated into a second word that belongs to a fixed language.

[0200] Optionally, the method is performed by a target language model, and the processing unit 702 is further configured to: during the processing of each word segment in the sequence of words to be predicted:

[0201] If the original adjacency probability is not less than the probability threshold, the parameters of the target language model are adjusted based on the first reward function;

[0202] If the original adjacency probability is less than the probability threshold, and the second word segmentation belongs to a fixed language, the parameters of the target language model are adjusted based on the second reward function;

[0203] If the original adjacency probability is less than the probability threshold, and the second word segmentation does not belong to a fixed language, the parameters of the target language model are adjusted based on the third reward function.

[0204] Optionally, the method is performed through a target language model, and the processing unit 702 is further configured to: perform unsupervised training on the self-attention mechanism in the target language model using an unlabeled text set, including:

[0205] Each unlabeled text in the unlabeled text set is segmented to obtain an unlabeled word sequence, and the unlabeled word sequence is then segmented and corrupted to obtain a corrupted word sequence.

[0206] Based on the self-attention mechanism, the unlabeled semantic association between each word segment in the unlabeled word sequence and the damaged semantic association between each word segment in the damaged word sequence are obtained respectively. The unlabeled adjacency probability is obtained based on the unlabeled semantic association and the damaged adjacency probability is obtained based on the damaged semantic association.

[0207] The parameters of the self-attention mechanism are adjusted based on the difference between the unlabeled adjacency probability and the damaged adjacency probability.

[0208] Optionally, the word segmentation unit 701 is further configured to: segment each unlabeled text in the unlabeled text set into words to obtain each preset word segmentation;

[0209] The processing unit 702 is further configured to: establish the preset word library based on the preset word segments.

[0210] Optionally, it may also include: the text to be predicted and the unlabeled text are Cantonese text.

[0211] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: The obtained text to be predicted is segmented into words to obtain a sequence of words to be predicted; For each word segmentation iteration in the sequence of words to be predicted, the following steps are performed: Obtain the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset dictionary, and the original adjacency probability of the first word segment with the second word segment, wherein the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted; If the original adjacency probability is less than the probability threshold, at least one candidate word is selected from the preset dictionary based on the preset filtering conditions and the second word segmentation. The semantic adjacency probability of the preset word that matches the candidate word is taken as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is taken as the target word. The second word in the word sequence to be predicted is replaced by the target word. The text corresponding to the processed word sequence to be predicted is used as the target text. The step of obtaining the semantic adjacency probabilities of the first word segment in the word sequence to be predicted with each preset word segment in the preset vocabulary, and the original adjacency probabilities of the first word segment with the second word segment, includes: Obtain the first intermediate word sequence and the second intermediate word sequence from the word sequence to be predicted; The first intermediate word sequence is the word sequence of each word in the word sequence to be predicted that precedes the first word segment and contains the first word segment; the second intermediate word sequence is the word sequence in the word sequence to be predicted that contains the first intermediate word sequence and the second word segment. The first intermediate word sequence is concatenated with each preset word segment in the preset word library to obtain the concatenated word sequence corresponding to each preset word segment. Based on the attention mechanism, the first semantic association between the first segment and the preset segment in the concatenated word sequence, and the second semantic association between the first segment and the second segment in the second intermediate word sequence are extracted. The semantic adjacency probability is obtained based on the first semantic association, and the original adjacency probability is obtained based on the second semantic association.

2. The method as described in claim 1, characterized in that, The step of selecting the candidate word corresponding to the candidate with the highest candidate adjacency probability among at least one candidate adjacency probability as the target word includes: Based on a Markov process, the largest candidate adjacency probability is determined from the at least one candidate adjacency probability. The candidate word corresponding to the largest candidate adjacency probability is taken as the target word.

3. The method as described in claim 1, characterized in that, Also includes: If the original adjacency probability is not less than the probability threshold, the second word segment in the word sequence to be predicted is retained.

4. The method as described in claim 1, characterized in that, Before selecting at least one candidate word from the preset dictionary based on preset filtering conditions and the second word segmentation, the method further includes: If the second word does not belong to a fixed language, then based on the translation tool, the second word is translated into a second word that belongs to a fixed language.

5. The method according to any one of claims 1-4, characterized in that, The method is executed through the target language model, during the processing of each word segment in the sequence of words to be predicted: If the original adjacency probability is not less than the probability threshold, the parameters of the target language model are adjusted based on the first reward function; If the original adjacency probability is less than the probability threshold, and the second word segmentation belongs to a fixed language, the parameters of the target language model are adjusted based on the second reward function; If the original adjacency probability is less than the probability threshold, and the second word segmentation does not belong to a fixed language, the parameters of the target language model are adjusted based on the third reward function.

6. The method according to any one of claims 1-4, characterized in that, The method is executed through a target language model. Before segmenting the acquired text to be predicted to obtain the word sequence, an unlabeled text set is used to perform unsupervised training on the self-attention mechanism in the target language model, including: Each unlabeled text in the unlabeled text set is segmented to obtain an unlabeled word sequence, and the unlabeled word sequence is then segmented and corrupted to obtain a corrupted word sequence. Based on the self-attention mechanism, the unlabeled semantic association between each word segment in the unlabeled word sequence and the damaged semantic association between each word segment in the damaged word sequence are obtained respectively. The unlabeled adjacency probability is obtained based on the unlabeled semantic association and the damaged adjacency probability is obtained based on the damaged semantic association. The parameters of the self-attention mechanism are adjusted based on the difference between the unlabeled adjacency probability and the damaged adjacency probability.

7. The method as described in claim 6, characterized in that, Also includes: Each unlabeled text in the unlabeled text set is segmented into words to obtain each preset word segmentation; Based on the aforementioned preset word segments, the preset word library is established.

8. The method as described in claim 6, characterized in that, Also includes: The text to be predicted and the unlabeled text are Cantonese texts.

9. A data processing apparatus for implementing the method as described in any one of claims 1-4, characterized in that, The device includes: The word segmentation unit is used to segment the acquired text to be predicted to obtain a sequence of words to be predicted. The processing unit is used to iteratively process each word segment in the word sequence to be predicted, and performs the following steps respectively: Obtain the semantic adjacency probability of the first word segment in the word sequence to be predicted with each preset word segment in the preset dictionary, and the original adjacency probability of the first word segment with the second word segment, wherein the second word segment is the word segment adjacent to the first word segment in the word sequence to be predicted; If the original adjacency probability is less than the probability threshold, at least one candidate word is selected from the preset dictionary based on the preset filtering conditions and the second word segmentation. The semantic adjacency probability of a preset word that matches the candidate word is used as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target word. The target word is used to replace the second word in the word sequence to be predicted. The processing unit is also used to use the text corresponding to the processed word sequence to be predicted as the target text.

Images