Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the word segmentation of an obtained to-be-predicted text, and obtaining a to-be-predicted word sequence; iterative processing is carried out on each segmented word in the to-be-predicted word sequence, and the steps of obtaining the semantic adjacency probability of a first segmented word in the to-be-predicted word sequence and each preset segmented word in a preset word bank and the original adjacency probability of the first segmented word and an adjacent second segmented word are respectively executed, and if the original adjacency probability is smaller than a probability threshold value, the to-be-predicted word sequence is obtained. Based on a preset screening condition and the second segmented word, selecting a plurality of candidate segmented words from a preset dictionary; taking the semantic adjacency probability of a preset segmented word matched with the candidate segmented word as a candidate adjacency probability, taking the candidate segmented word corresponding to the maximum candidate adjacency probability as a target segmented word, and replacing a second segmented word in the to-be-predicted word sequence with the target segmented word; and obtaining a final target text of the to-be-predicted text. The data cleaning accuracy and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device. Background Art

[0002] Data cleaning refers to a process of discovering and correcting identifiable errors in data files. It is usually done by computers rather than humans, including checking data consistency, handling invalid values ​​and missing values, etc. Generally speaking, the data in the data warehouse is extracted from multiple business systems and is a collection of historical data on a certain topic. Therefore, it is inevitable that there are duplicate, erroneous, conflicting and other non-compliant data. The task of data cleaning is to clean up these non-compliant data to obtain valuable data, which can be used for data statistics, model training, etc.

[0003] In the related art, Cantonese translation applications usually convert Cantonese into universal standard Chinese or convert Chinese into Cantonese based on the Cantonese language model. Therefore, a large amount of high-quality Cantonese text data is often required to train the Cantonese language model to improve its Cantonese processing performance. In other words, in the development process of Cantonese translation applications, there is often a very large demand for high-quality Cantonese text data. Among them, there are two ways to obtain high-quality Cantonese text. One is to manually clean the collected Cantonese text data. This method requires a lot of manpower costs and a long data cleaning cycle. The other method uses a rule correction method to clean the collected Cantonese text data. This method is prone to errors and omissions and cannot clean all "dirty data".

[0004] Therefore, there is an urgent need to redesign a data processing method and overcome the above-mentioned defects. Summary of the invention

[0005] The embodiments of the present application provide a data processing method and device to improve the accuracy and efficiency of data cleaning.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] Perform word segmentation on the acquired text to be predicted to obtain a word sequence to be predicted;

[0008] For each word segmentation in the to-be-predicted word sequence, the following steps are performed respectively:

[0009] Obtaining semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, where the second word is the word adjacent to the first word in the word sequence to be predicted;

[0010] If the original adjacency probability is less than the probability threshold, selecting at least one candidate word segmentation from a preset dictionary based on a preset screening condition and the second word segmentation;

[0011] Taking the semantic adjacency probability of the preset participle that matches the candidate participle as the candidate adjacency probability, taking the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target participle, and replacing the second participle in the to-be-predicted word sequence with the target participle;

[0012] The text corresponding to the processed word sequence to be predicted is used as the target text.

[0013] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0014] A word segmentation unit is used to perform word segmentation processing on the acquired text to be predicted to obtain a word sequence to be predicted;

[0015] The processing unit is used to iteratively process each word segment in the word sequence to be predicted, and respectively perform the following steps:

[0016] Obtaining semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, where the second word is the word adjacent to the first word in the word sequence to be predicted;

[0017] If the original adjacency probability is less than the probability threshold, selecting at least one candidate word segmentation from a preset dictionary based on a preset screening condition and the second word segmentation;

[0018] Taking the semantic adjacency probability of the preset participle that matches the candidate participle as the candidate adjacency probability, taking the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target participle, and replacing the second participle in the to-be-predicted word sequence with the target participle;

[0019] The processing unit is further configured to use the processed text corresponding to the to-be-predicted word sequence as the target text.

[0020] Optionally, the processing unit is specifically used for:

[0021] Obtaining a first intermediate word sequence and a second intermediate word sequence in the to-be-predicted word sequence;

[0022] The first intermediate word sequence is a word sequence of words before the first word in the word sequence to be predicted and including the first word, and the second intermediate word sequence is a word sequence including the first intermediate word sequence and the second word in the word sequence to be predicted;

[0023] splicing the first intermediate word sequence with each preset word in the preset word library to obtain a spliced ​​word sequence corresponding to each preset word;

[0024] Based on the attention mechanism, a first semantic association relationship between the first participle and the preset participle in the concatenated word sequence and a second semantic association relationship between the first participle and the second participle in the second intermediate word sequence are extracted, and the semantic adjacency probability is obtained according to the first semantic association relationship, and the original adjacency probability is obtained according to the second semantic association relationship.

[0025] Optionally, the processing unit is specifically used for:

[0026] Determining a maximum candidate adjacency probability from the at least one candidate adjacency probability based on a Markov process;

[0027] The candidate participle corresponding to the maximum candidate adjacency probability is used as the target participle.

[0028] Optionally, the processing unit is further used for:

[0029] If the original adjacency probability is not less than the probability threshold, the second word segmentation in the to-be-predicted word sequence is retained.

[0030] Optionally, the processing unit is further used for:

[0031] If the second participle does not belong to a fixed language, the second participle is translated into a second participle belonging to a fixed language based on a translation tool.

[0032] Optionally, the method is performed by a target language model, and the processing unit is further configured to: in the process of processing each word segment in the to-be-predicted word sequence:

[0033] If the original adjacency probability is not less than the probability threshold, adjusting the parameters of the target language model based on a first reward function;

[0034] If the original adjacency probability is less than the probability threshold, and the second word segment belongs to a fixed language, adjusting the parameters of the target language model based on a second reward function;

[0035] If the original adjacency probability is less than the probability threshold, and the second word segment does not belong to a fixed language, adjusting the parameters of the target language model based on a third reward function.

[0036] Optionally, the method is performed by a target language model, and the processing unit is further used to: use an unlabeled text set to perform unsupervised training on a self-attention mechanism in the target language model, including:

[0037] Performing word segmentation processing on each unlabeled text in the unlabeled text set to obtain an unlabeled word sequence, and performing word segmentation damage processing on the unlabeled word sequence to obtain a damaged word sequence;

[0038] Based on the self-attention mechanism, the unlabeled semantic association relationship between each word in the unlabeled word sequence and the damaged semantic association relationship between each word in the damaged word sequence are respectively obtained, and the unlabeled adjacency probability is obtained according to the unlabeled semantic association relationship, and the damaged adjacency probability is obtained according to the damaged semantic association relationship;

[0039] Based on the difference between the unlabeled neighbor probability and the damaged neighbor probability, a parameter of the self-attention mechanism is adjusted.

[0040] Optionally, the word segmentation unit is further used to: segment each unlabeled text in the unlabeled text set to obtain each preset word segmentation;

[0041] The processing unit is further configured to: establish the preset word library based on the preset word segmentations.

[0042] Optionally, also include:

[0043] The to-be-predicted text and the unlabeled text are Cantonese texts.

[0044] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any one of the data processing methods in the first aspect.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on a computer device, the computer program is used to enable the computer device to execute any one of the data processing methods in the first aspect above.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; when a processor of a computer device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the computer device executes any one of the data processing methods in the above-mentioned first aspect.

[0047] The beneficial effects of this application are as follows:

[0048] In the data processing method provided in the embodiment of the present application, each word segment in the word sequence to be predicted is iteratively processed to obtain the original adjacency probability between the first word segment and the adjacent second word segment in the word sequence to be predicted, and the semantic adjacency probability of the first word segment and each preset word segment in the preset word library being adjacent to each other. In other words, based on the semantic information of the first word segment and the second word segment, the probability of the first word segment and the second word segment being adjacent in a sentence text is obtained, and based on the semantic information of the first word segment and the preset word segment, the probability of the first word segment and the preset word segment being adjacent in a sentence text is obtained.

[0049] In this way, the relationship between the original adjacency probability and the probability threshold can be determined. If the original adjacency probability is less than the probability threshold, it means that the semantic information connection between the first participle and the second participle is not smooth, and the second participle may be an erroneous word adjacent to the first participle. After that, according to the second participle and the preset screening condition, the preset participles that are homophonic, synonymous, etc. with the second participle can be screened from the preset dictionary as candidate participles. Since the semantic adjacency probability between the first participle and each preset participle has been obtained before, the semantic adjacency probability corresponding to the preset participle that is the same as the candidate participle is the probability that the first participle and the candidate participle are adjacent in a sentence text. Therefore, the semantic adjacency probability of the preset participle that matches the candidate participle is used as the candidate adjacency probability, and the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target participle. In this way, the target participle is most likely to be the correct participle adjacent to the first participle, and the target participle is used to replace the second participle in the word sequence to be predicted.

[0050] After the above processing is performed on each word segment in the word sequence to be predicted, the target text obtained is the correct text. It can be seen that compared with the related art, which uses manual error correction or rule correction to perform data cleaning, the natural language processing method used in this application can automatically and quickly perform data cleaning, save labor costs, shorten the data cleaning cycle, and improve the accuracy of data cleaning. Since the erroneous text to be detected is corrected to obtain the correct text, the amount of data obtained after data cleaning is also guaranteed.

[0051] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 An optional schematic diagram of an application scenario schematic diagram provided in an embodiment of the present application;

[0054] Figure 2 An optional schematic diagram of an application scenario provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0056] Figure 4 A flowchart of a method for obtaining original adjacency probability and semantic understanding probability provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of a self-attention mechanism structure provided in an embodiment of the present application;

[0058] Figure 6 A flowchart of a data processing method provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0061] It is understandable that in the following specific implementations of the present application, the data related to the text to be detected and the unlabeled text involved, when the various embodiments of the present application are applied to specific products or technologies, need to obtain relevant permissions or consents, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, when it is necessary to obtain relevant data, you can recruit relevant volunteers and sign relevant agreements for volunteer authorization data, and then use the data of these volunteers for implementation; or, by implementing it within the authorized organization, use the data of internal members of the organization to implement the following implementation methods to make relevant recommendations to internal members; or, the relevant data used in the specific implementation are all simulated data, such as simulated data generated in a virtual scene.

[0062] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0063] Markov process is a kind of random process and an important method for studying the state space of discrete event dynamic systems. It is a random process that changes from one state to another in the state space. The process requires "no memory", that is, the probability distribution of the next state can only be determined by the current state. The previous events in the time series are irrelevant. The Markov process believes that all past information is stored in the current state. For example, suppose the existing sequence state is...X t-2 , X t-1 , X t , X t+1 ..., then in X t+1 The conditional probability of the state at a certain moment depends only on the state X at the previous moment. t , that is: P(X t+1 ∣…X t-2 , X t-1 , X t )=P(X t+1 ∣X t )

[0064] It is known that the probability of state transition at a certain moment only depends on its previous state. Therefore, as long as the transition probability between any two states in the system can be calculated, the Markov process can be determined. It can be seen that its mathematical basis is the theory of random processes.

[0065] BERT (Bidirectional Encoder Representation Based on Transformer): A pre-trained model. "Bidirectional" means that when the model processes a word, it can use the information of both the previous word and the following word. Randomly mask some words and use all the unmasked words for prediction.

[0066] Feedforward neural network (FNN), referred to as feedforward network, is a type of artificial neural network. Feedforward neural network adopts a unidirectional multi-layer structure. Each layer contains a number of neurons. In this kind of neural network, each neuron can receive signals from the neurons in the previous layer and generate outputs to the next layer. The 0th layer is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers (or hidden layers, hidden layers). The hidden layer can be one layer or multiple layers. There is no feedback in the entire network. The signal propagates unidirectionally from the input layer to the output layer. Different features are extracted through each layer of the network, and the final output is the result.

[0067] Activation function is a function that runs on the neurons of artificial neural networks. It is responsible for mapping the input of the neuron to the output and determines the content of transmitting the previous layer of neurons to the next neuron. It includes Softmax function, Tanh function and ReLU function.

[0068] Tanh function is one of the hyperbolic functions. Tanh() is the hyperbolic tangent, which is a saturation activation function in neural networks and can alleviate the gradient disappearance phenomenon. In mathematics, the hyperbolic tangent "Tanh" is derived from the basic hyperbolic functions hyperbolic sine and hyperbolic cosine.

[0069] Softmax function: It is a common S-shaped function in biology, also known as the S-shaped growth curve. It is a saturated activation function in neural networks. In information science, due to its monotonic and inverse monotonic properties, the Sigmoid function is often used as an activation function of neural networks to map variables between 0 and 1.

[0070] ReLU function (Rectified Linear Unit), meaning rectified linear unit, Relu activation function is a commonly used neural activation function, which is actually a piecewise linear function, which changes all negative values ​​to 0, while the positive values ​​remain unchanged. This operation is called unilateral inhibition. The ReLu function has the advantage of sparsity, which enables the sparse model to better mine relevant features and fit the training data; in the area of ​​x>0, there will be no problems of gradient saturation and gradient disappearance; the computational complexity is low, no exponential operation is required, and the activation value can be obtained with just one threshold. The disadvantage is that the output is not symmetrical to 0. Since the ReLu activation function value is 0 when it is less than 0, the gradient is 0, so there are some neurons that will never be updated.

[0071] The self-attention mechanism is one of the core components of BERT. It is used to make the machine notice the correlation between different parts of the entire input. The self-attention mechanism solves the following problems: the input received by the neural network is a lot of vectors of different sizes, and there is a certain relationship between different vectors. However, during the actual training, the neural network cannot give full play to the relationship between these inputs, resulting in extremely poor model training results. For example, machine translation problems, part-of-speech tagging problems, semantic analysis problems, etc. The self-attention mechanism can be used for the problem that the fully connected neural network cannot establish correlation for multiple related inputs, and let the machine notice the correlation between different parts of the entire input. In simple terms, one of its implementation methods is to have multiple input vectors, and perform the following steps for each of these multiple input vectors:

[0072] The self-attention mechanism contains the weight matrix W q , weight matrix W k , weight matrix W v , for an input vector X1, calculate X1*W respectively q =Q1,X1*W k =K1,X1*W v = V1, get this input vector and compare it with other input vectors (X2, X3…X n ), and obtain the correlation matrix QK1. Perform Softmax operation or ReLU operation on this correlation matrix to obtain the correlation expression matrix QK1'. Further, according to the meaning of the input vector, Z1 = QK1'*V1 is obtained. In this way, the output result Z1 contains both the meaning of the input vector X1 itself and the relationship between the input vector X1 and other input vectors (X2, X3...X n Based on the above, it can be understood that the essence of the self-attention mechanism is to transform each input vector (X1, X2…X n ) is mapped to the output matrix point Z of the output result in the new space.

[0073] The technical solutions of the embodiments of the present application involve artificial intelligence, natural language processing and machine learning technologies. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0074] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0075] Among them, big data technology is a solution that can process and store large-scale data, including data collection, preprocessing, storage, calculation and analysis. "Big data" can be called huge data, which contains huge and complex data information. The amount of data involved is so large that it cannot be captured within a reasonable time through mainstream software tools. The collection of big data usually involves obtaining data from various data sources, such as social media, corporate databases, log files, etc. Data preprocessing is an important step in big data processing, including operations such as data cleaning, data integration and data conversion to ensure data quality and accuracy. Data storage generally involves traditional relational database technology and new database technology, such as NoSQL database, column storage database and distributed file system. Data calculation and analysis involves a variety of technologies such as distributed computing and stream processing. Among the aforementioned multiple big data technology links, data mining and machine learning are also involved, which can discover valuable data and patterns from large-scale data and predict future trends. Commonly used machine learning algorithms include classification, clustering, regression and prediction, which can be used in various scenarios, such as recommendation systems, anomaly detection and prediction models.

[0076] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies. Based on big data technology and natural language processing technology, it can realize data cleaning and data correction of large amounts of data, such as data correction of input batches of text to be predicted, using the correct target segmentation to replace the wrong second segmentation in the text to be predicted to obtain the correct target text, and then obtain a batch of correct target texts, etc. In the field of data where open source is scarce, based on this processing method, rich data information can be obtained.

[0077] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0078] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0079] The solution provided in the embodiments of the present application involves natural language processing technology, data processing technology and machine learning technology of artificial intelligence.

[0080] In the embodiment of the present application, the training of the target language model is applied to machine learning technology. During the training process, based on the batch of unlabeled texts in the unlabeled text set, the self-attention mechanism in the target language model is iteratively unsupervised trained, and the unlabeled text is analyzed using big data technology to obtain the unlabeled semantic association relationship between adjacent word segments in the unlabeled text. The unlabeled text is analyzed based on the unlabeled semantic association relationship, and the parameters of the self-attention mechanism are adjusted. In this way, a self-attention mechanism with the probability of preliminarily and autonomously analyzing the adjacency between adjacent word segments is obtained, so that when the subsequent target language model is unsupervisedly trained in the application process, the probability of adjacency between adjacent word segments obtained has a preliminary accuracy basis to ensure applicability.

[0081] In addition, it should be noted that the artificial neural network model in the embodiment of the present application can be trained online or offline, which is not specifically limited here. In this article, offline training is used as an example for illustration.

[0082] The following is a brief introduction to the design concept of the embodiment of the present application:

[0083] In the field of artificial intelligence technology, in order to meet the needs of life and work, more and more applications with text recognition and translation are available, such as multi-language translation applications, Cantonese and universal standard Chinese translation applications, etc. Among them, the Cantonese language model is usually used in the Cantonese and universal standard Chinese translation application for text analysis and processing to translate between Cantonese and universal standard Chinese. For example, by inputting the Cantonese text "This weather is unpredictable, it changes at any time", or the universal standard Chinese text "This weather is unpredictable, it changes at any time", the Chinese text "This weather is unpredictable, it changes at any time" or "This weather is unpredictable, it changes at any time" can be accurately output.

[0084] In the related art, in order to improve the Cantonese processing performance of the Cantonese language model, a large amount of high-quality data needs to be collected for training so that the Cantonese language model can accurately translate Cantonese and Chinese. However, during the data collection process, there is a lack of open source resources related to Cantonese, resulting in insufficient data to support model training. Even if social media can provide a large amount of data, there are still problems such as uneven data quality, mixing of Chinese and English, and homophonic substitution. Therefore, it is necessary to clean the social media data. Data cleaning usually adopts two methods: one is the manual correction method, which requires a lot of manpower and time, and the other is the rule correction method, which requires pre-defined rules and may not cover all types of text errors.

[0085] In view of this, the embodiments of the present application provide a data processing method, apparatus, computer equipment and storage medium. In the data processing method, for each word segmented in the predicted text, the semantic information connection between adjacent word segments is detected in turn to see if it is smooth. If there is a semantic information connection between adjacent word segments (the first word segmented in front and the second word segmented in the back), the target word segmented with a smooth semantic connection with the first word segmented is selected from the preset dictionary, and the second word segmented with a smooth semantic connection is replaced with the target word segmented. In this way, data cleaning can be achieved and "dirty data" can be transformed into valuable data.

[0086] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0087] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes Figure 1As shown, it is a schematic diagram of an application scenario of an embodiment of the present application, and the application scenario diagram includes any server 110 among multiple servers, and any server 120 among multiple servers, wherein server 110 can be used to collect data from relevant systems or platforms such as business systems and social media, and send the collected data to server 120. After receiving the collected data, server 120 performs data processing for each text to be detected in the collected data, obtains the target text of the text to be detected, and then obtains a large amount of high-value data. Alternatively, server 110 or server 120 can both collect data and process data.

[0088] Based on this application scenario, the present application embodiment provides another application scenario schematic diagram, such as Figure 2 As shown, any terminal device 210 among multiple terminal devices and any server 220 among multiple servers. The terminal device 210 can be used to upload the text to be detected, and the server 220 receives the text to be detected and performs data processing to obtain the target text of the text to be detected and returns it to the terminal device 210.

[0089] In the above Figure 1 and Figure 2 In the application scenario, the terminal device 210 includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; a client related to the data processing business can be installed on the terminal device, and the client can be a software (such as a browser, communication software, etc.), or a web page, a small program, etc. The server 110, server 120 and server 220 are the background service terminals corresponding to the software or web page, small program, etc., or are the background service terminals specially used to return the processing results of the data processing business to the client, and this application does not make specific restrictions. Server 110, server 120 and server 220 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0090] It should be noted that Figure 1 and Figure 2 What is shown is just an example. In fact, the number and communication mode of terminal devices and servers are not limited and are not specifically limited in the embodiments of the present application.

[0091] In addition, the data processing method of the embodiment of the present application can be applied to various scenarios, such as translation applications, document correction applications, cross-language conversation applications, etc., or used for massive data and data cleaning to obtain training data for relevant language models in these applications, etc. There is no specific limitation on the application scenarios of the data processing method here.

[0092] The following describes the data processing method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0093] See also Figure 3 As shown, it is a flow chart of the data processing method provided in the embodiment of the present application. Here, an example is given with a server as the execution subject. The specific implementation process of the method is as follows:

[0094] Step 301: perform word segmentation processing on the acquired text to be predicted to obtain a sequence of words to be predicted.

[0095] In one embodiment, the text to be predicted may be text data in literature or social media.

[0096] In one embodiment, the text to be predicted may be a Cantonese text. For example, a Cantonese dictionary may be collected through legal channels, covering relevant information such as standard writing form, pronunciation and phonetic symbols. For example, various data released by news media may be collected, including the text content of articles. For example, social media data may be collected, including main post content, comments, replies, and other relevant information. Correspondingly, the segmentation words in the preset dictionary and the preset segmentation words in the preset word library may also be Cantonese segmentation words.

[0097] In one embodiment, if massive data is obtained from news media, social media and other platforms, in order to ensure the effectiveness and accuracy of subsequent data processing and analysis, the massive data can be screened by language based on a language classification algorithm to obtain a batch of Cantonese texts to be predicted.

[0098] In one embodiment, the text to be predicted is "Have you eaten?", and the text to be predicted is segmented to obtain a word sequence to be predicted "Have you, eaten, rice, or not?". The example here is only used to clearly illustrate the segmentation of the text to be predicted, and does not limit the text to be predicted and its segmentation.

[0099] Step 302: Iterate each word segment in the word sequence to be predicted and perform the following steps respectively:

[0100] Step 3021: Obtain the semantic adjacency probabilities of the first word segment in the to-be-predicted word sequence with each preset word segment in the preset word library, and the original adjacency probability between the first word segment and the second word segment, where the second word segment is the word segment adjacent to the first word segment in the to-be-predicted word sequence.

[0101] In one embodiment, based on the above embodiment, if the first word segment is "你 (nǐ)", the second word segment is "食咗 (shí zuǒ)"; if the first word segment is "食咗 (shí zuǒ)", the second word segment is "饭 (fàn)", and so on. The second word segment is the word segment adjacent to the first word segment in the to-be-predicted word sequence.

[0102] In one embodiment, the semantic adjacency probability refers to the probability that the first word segment is followed by a preset word segment, and the original adjacency probability refers to the probability that the first word segment is followed by the second word segment. For example, if the first word segment is "天气 (tiān qì)", the probabilities of the following word segments "好 (hǎo)", "坏 (huài)", "晴朗 (qíng lǎng)" are greater than the probabilities of "吃 (chī)", "睡 (shuì)".

[0103] In one embodiment, methods such as support vector machines and Word2Vec can be used to obtain the semantic information between the first word segment and the second word segment, calculate the original adjacency probability between the first word segment and the second word segment according to the semantic information, and obtain the semantic information between the first word segment and the preset word segment, and calculate the semantic adjacency probability between the first word segment and the preset word segment according to the semantic relationship. Among them, there are various methods to obtain the original adjacency probability and the semantic adjacency probability. For example, neural network methods can also be used. The related technologies involved in these methods belong to relatively mature technologies and will not be elaborated here.

[0104] In one embodiment, the preset word library can be composed of a large number of preset word segments collected in advance, and the preset word segments can be obtained from relevant data sources such as literature materials and social media.

[0105] Step 3022: If the original adjacency probability is less than the probability threshold, select at least one candidate word segment from the preset dictionary based on the preset screening conditions and the second word segment.

[0106] In one embodiment, the original adjacency probability is the adjacency probability between the first word segment and the second word segment. If the original adjacency probability is less than the probability threshold, it can be considered that the connection between the first word segment and the second word segment is not smooth. Then, at least one candidate word segment that can replace the second word segment needs to be selected from the preset dictionary to find the target word segment with smooth adjacency to the first word segment from the at least one candidate word segment.

[0107] In one embodiment, the preset filtering condition may include: selecting a segmentation that is a homophone with the second segmentation from the preset dictionary as a candidate segmentation, selecting a segmentation with similar strokes to the second segmentation from the preset dictionary as a candidate segmentation, etc. In other words, the preset filtering condition at least includes selecting a segmentation that is a homophone with the second segmentation from the preset dictionary as a candidate segmentation, and selecting a segmentation with similar strokes to the second segmentation from the preset dictionary as a filtering condition among the candidate segmentations.

[0108] Step 3023: The semantic adjacency probability of the preset participle matching the candidate participle is used as the candidate adjacency probability, and the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target participle, and the target participle is used to replace the second participle in the word sequence to be predicted.

[0109] In one embodiment, after obtaining at least one candidate participle that may replace the second participle, in order to obtain the candidate adjacency probability between the first participle and the candidate participle, the semantic adjacency probability between the first participle and each preset participle is known in the previous step, then the candidate participle can be matched with the preset participle, and the preset participle that is the same as the candidate participle is found, and the semantic adjacency probability of the preset participle that is the same as the candidate participle can be used as the candidate adjacency probability. Then, the candidate participle corresponding to the largest candidate adjacency probability can be selected as the target participle, and the target participle is used to replace the second participle in the word sequence to be predicted.

[0110] In one embodiment, if the maximum candidate adjacency probability is still less than the probability threshold, the second word segmentation can be directly deleted, and the next cycle is entered to re-determine the original adjacency probability between the first word segmentation and the third word segmentation adjacent to the second word segmentation. If the original adjacency probability between the first word segmentation and the third word segmentation is not less than the probability threshold, the first word segmentation and the third word segmentation are adjacent, otherwise, the text to be predicted is discarded.

[0111] In one embodiment, if the original adjacency probability is not less than the probability threshold, the second word segment in the word sequence to be predicted is retained, and the next loop is entered, the second word segment of the current loop is used as the first word segment, and step 303 of the current loop and the text corresponding to the word sequence to be predicted after processing are used as the target text.

[0112] In one embodiment, after the processing of the above steps 301 and 302, the adjacency probabilities between adjacent word segments in the processed word sequence to be predicted are all greater than the probability threshold, that is, the semantic connections between adjacent word segments in the word sequence to be predicted are smooth, and they can be output as target text.

[0113] In the above method, the original adjacency probability between the first participle and the adjacent second participle in the word sequence to be predicted, and the semantic adjacency probability of the first participle and each preset participle in the preset word library are obtained. The relationship between the original adjacency probability and the probability threshold is determined. If the original adjacency probability is less than the probability threshold, it means that the semantic information connection between the first participle and the second participle is not smooth, and the second participle may be an incorrect word adjacent to the first participle. After that, according to the second participle and the preset screening condition, preset participles that are homophonic, similar in form, etc. with the second participle can be screened from the preset dictionary as candidate participles. Since the semantic adjacency probability between the first participle and each preset participle has been obtained before, the semantic adjacency probability corresponding to the preset participle that is the same as the candidate participle is the probability that the first participle and the candidate participle are adjacent in a sentence text. Therefore, the semantic adjacency probability of the preset participle that matches the candidate participle is used as the candidate adjacency probability, and the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target participle. In this way, the target word segmentation is most likely to be the correct word segmentation adjacent to the first word segmentation, and the target word segmentation is used to replace the second word segmentation in the sequence of words to be predicted. Compared with the related art, data cleaning is performed by manual error correction or rule correction. The natural language processing method used in this application can automatically and quickly perform data cleaning, save labor costs, shorten the data cleaning cycle, and does not require preset rules. It can flexibly respond to different data anomalies and improve the accuracy of data cleaning. Since the erroneous text to be detected is corrected to obtain the correct text, the amount of data obtained after data cleaning is also guaranteed.

[0114] Based on the above Figure 3 The data processing method in this application embodiment provides a method for obtaining the original adjacency probability and the semantic understanding probability, such as Figure 4 As shown, in the above step 3021, the semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, as well as the original adjacency probabilities of the first word and the second word are obtained, including:

[0115] Step 401, obtaining a first intermediate word sequence and a second intermediate word sequence in a word sequence to be predicted; the first intermediate word sequence is a word sequence of words before and including a first word segment in the word sequence to be predicted, and the second intermediate word sequence is a word sequence including the first intermediate word sequence and the second word segment in the word sequence to be predicted.

[0116] In one embodiment, based on the to-be-predicted word sequence in the above embodiment: "Have you eaten rice?", if the first participle is "eat", and the second participle is "rice", then the first intermediate word sequence is "you have eaten", and the second intermediate word sequence is "you have eaten rice".

[0117] Step 402: Concatenate the first intermediate word sequence with each preset word segment in the preset word library to obtain a concatenated word sequence corresponding to each preset word segment.

[0118] In one embodiment, based on the first intermediate word sequence in the above embodiment being "you, ate", assuming that the preset word library contains preset segmentation words "iron", "metal", "fruit", "steamed buns", etc., the first intermediate word sequence is respectively spliced ​​with each preset segmentation word in the preset word library to obtain multiple spliced ​​word sequences. For example, the spliced ​​word sequence: "you, ate, iron", the spliced ​​word sequence: "you, ate, metal", the spliced ​​word sequence: "you, ate, fruit", the spliced ​​word sequence: "you, ate, metal", the spliced ​​word sequence: "you, ate, steamed buns", etc.

[0119] Step 403: Based on the attention mechanism, extract the first semantic association relationship between the first word segment and the preset word segment in the concatenated word sequence, and the second semantic association relationship between the first word segment and the second word segment in the second intermediate word sequence, and obtain the semantic adjacency probability according to the first semantic association relationship, and obtain the original adjacency probability according to the second semantic association relationship.

[0120] In one embodiment, based on the above embodiment, the obtained second intermediate word sequence and each concatenated word sequence are respectively input into the self-attention mechanism to obtain the first semantic association relationship between the first participle and the adjacent preset participle in the concatenated word sequence, and the second semantic association relationship between the first participle and the second participle in the second intermediate word sequence. From the above introduction to the self-attention mechanism, it can be seen that the self-attention mechanism can retain the meaning expressed by the vector and can also obtain the correlation between the vectors, that is, the self-attention mechanism can retain the meaning of each participle in the second intermediate word sequence and can also obtain the second semantic association relationship between each participle. Similarly, the self-attention mechanism can retain the meaning of each participle in the concatenated word sequence and can also obtain the first semantic association relationship between each participle. Thus, based on the first semantic association relationship, the semantic adjacency probability between the first participle and the preset participle can be obtained to express the semantic fluency between the first participle and the preset participle. Similarly, based on the second semantic association relationship, the original adjacency probability between the first participle and the second participle can be obtained to express the semantic fluency between the first participle and the second participle.

[0121] In one embodiment, the self-attention mechanism here can be multi-layered, such as Figure 5 As shown, it is a schematic diagram of the structure of a self-attention mechanism provided in an embodiment of the present application. The self-attention mechanism can use a feedforward neural network and an activation function to improve the characteristic of the semantic features of the extracted word segmentations, optimize the correlation between the semantic features, and ensure the accuracy of the output results. The number of layers of the mechanism can be set as needed, and there is no specific limitation here.

[0122] Based on the above Figure 3 or Figure 4 The method in Figure 5 The mechanism structure of the self-attention mechanism in the above step 3023 is to use the candidate segmentation corresponding to the largest candidate adjacent probability among at least one candidate adjacent probability as the target segmentation, including:

[0123] Based on the Markov process, the maximum candidate adjacent probability is determined from at least one candidate adjacent probability; and the candidate participle corresponding to the maximum candidate adjacent probability is used as the target participle.

[0124] In one embodiment, during the iterative processing, for the processing of each word segment in the word sequence to be predicted, when the original adjacency probability between the first word segment in the current cycle and its adjacent second word segment is less than the probability threshold, the candidate word segment corresponding to the maximum candidate adjacency probability can be selected from the candidate word segments, and the candidate word segment is used as the target word segment to replace the second word segment. In other words, the Markov process only focuses on the semantic adjacency probability (state transition probability) between the current word segment (current state) and the next word segment (next state), and only seeks to obtain the maximum semantic adjacency probability (state transition probability).

[0125] In one embodiment, the Markov process is combined with the above-mentioned self-attention mechanism to obtain a target language model to input the text to be predicted. Then, based on the self-attention mechanism, the semantic situation between each segmentation in the sequence of words to be predicted of the text to be predicted can be obtained in a cyclic iteration. In the cyclic iteration process, the fluency of the semantic connection between the segmentations is obtained on the premise that the semantic situation of each ordered segmentation before the segmentation is retained in the segmentation, that is, the fluency of the text to be predicted is obtained. According to the Markov process, the segmentations with unsmooth connections in the text to be predicted are corrected to segmentations with smooth semantic connections to output the correct target text. This method can be called a data cleaning strategy based on reinforcement learning, which models natural language text based on the Markov decision process, focusing on long-term rewards rather than short-term benefits, that is, text correction focuses on the entire text rather than simple word corrections, so as to better handle complex data sets.

[0126] In one embodiment, when the data processing method is applied to data cleaning, it can also include deleting the text to be predicted. For example, if the word sequence to be predicted has a word segmentation exceeding a set percentage and the original adjacency probability between the word segmentation and the adjacent word segmentation is less than the probability threshold, the word sequence to be predicted can be deleted. For another example, if the text to be predicted is a text with incomplete semantic expression, it can be deleted, etc. There is no specific restriction on the rules for deleting text here, and it can be set as needed.

[0127] Based on the above Figure 3In the method flow, the embodiment of the present application provides a data processing method. In step 3022, before selecting at least one candidate participle from a preset dictionary based on a preset screening condition and a second participle, it also includes: if the second participle does not belong to a fixed language, translating the second participle into a second participle belonging to a fixed language based on a translation tool.

[0128] In one embodiment, if the original adjacency probability between the first participle and the second participle is less than the probability threshold, there is also a possibility that the second participle is in a non-fixed language, and the fluency between the cross-language participles is very small. Therefore, if the second participle does not belong to a fixed language, the language to which the second participle belongs can be determined based on a language classification algorithm, and then the second participle is translated into a fixed language based on a translation tool. For example, if the fixed language is Cantonese, the second participle may be English, Japanese, Korean, etc., and the second participle is translated into Cantonese based on the translation tool.

[0129] In one embodiment, the words in the preset dictionary are words of a fixed language, which may be standard Chinese or Cantonese.

[0130] Based on the above method flows and embodiments, the present application embodiment provides a data processing method, which is performed by a target language model. In the process of processing each word segment in the word sequence to be predicted:

[0131] If the original adjacency probability is not less than the probability threshold, adjusting the parameters of the target language model based on the first reward function;

[0132] If the original adjacency probability is less than the probability threshold and the second word segmentation belongs to a fixed language, the parameters of the target language model are adjusted based on the second reward function;

[0133] If the original adjacency probability is less than the probability threshold and the second word segmentation does not belong to a fixed language, the parameters of the target language model are adjusted based on the third reward function.

[0134] In one embodiment, the Markov process is combined with the self-attention mechanism to obtain the target language model. During the data processing, unsupervised training is performed at the same time. In this way, the more the target language model is used, the higher the accuracy of its data processing. If it is necessary to explain, the self-attention mechanism can also be replaced by a support vector machine, a neural network or other mechanism. The self-attention mechanism here is only used to clearly explain the data processing method of this solution.

[0135] In one embodiment, the target language model is obtained based on the above Markov process combined with the self-attention mechanism, and data processing is performed:

[0136] Step a: Segment the text to be predicted to obtain a sequence of words x1, x2, x3…, x4 of length N.N , and we get the state space S, which contains the strings: [x1], [x1,x2], [x1,x2,x3]... [x1,x2,x3…,x N-1 】,

x1,x2,x3…,x N

[0137] Step b, in the tth cycle (here it is assumed that the target language model does not set the initial character x0, then the tth cycle is the processing of the tth word segment in the word sequence to be predicted. In addition, if the target language model is set with the initial character x0, then the t+1th cycle is the processing of the tth word segment in the word sequence to be predicted), the target language model obtains the tth string x1, x2, x3…, x T (Status)s t ∈S, T≤F.

[0138] Step c: The action space of the target language model is recorded as A. t =x1,x2,x3…,x T , x t is the first participle, x t+1 is the second segmentation, and a preset dictionary is set in the action space A, P(x t+1 |s t ) is the first word segmentation x obtained by the target language model t and the second participle x t+1 The original adjacency probability between , φ is the probability threshold;

[0139] The first case: If P(x t+1 |s t )≥φ, the first participle x t and the second participle x t+1 The semantics are smooth, and the intelligent agent will take action a t =x t+1 ; That is, after confirming the first participle, the second participle is adjacent, and the environment state of the target language model is transferred to s t+1 ∈S, confirm s t+1 =【x1,x2,x3…,x t ,x t+1 ]; and get a larger reward, adjust the parameters of the target language model based on the first reward function, the first reward function is as follows:

[0140]

[0141] Wherein, F is the length of the string of the text to be predicted.

[0142] The second case: If P(x t+1|s t )<φ, the first participle x t and the second participle x t+1 There is no semantic flow between them, and x t+1 When the fixed language is Cantonese, the intelligent experience will select at least one candidate word segmentation from the preset dictionary (Cantonese dictionary) Z according to the preset filtering conditions. The semantic adjacency probability of the preset word that is the same as the candidate word is taken as the candidate adjacency probability, and the candidate word corresponding to the largest candidate adjacency probability is taken as the candidate word. As target word And replace the second participle x t+1 , the target language model environment state transfer is obtained as s t+1 ∈S, confirm And get a reward, adjust the parameters of the target language model based on the second reward function, the second reward function is as follows:

[0143]

[0144] Wherein, F is the length of the string of the text to be predicted.

[0145] The third case: If P(x t+1 |s t )<φ, the first participle x t and the second participle x t+1 There is no semantic flow between them, and x t+1 Not a fixed language (x t+1 can be English, Japanese, Korean, etc.), the intelligent experience will translate the second participle x that does not belong to the fixed language according to the translation tool C. t+1 Translated into the second participle of a fixed language Confirm x again t and The original adjacent probability of translation between Then the first participle x t and the second participle of a fixed language The semantics are smooth, As target participle And replace the second participle x t+1 , the target language model environment state transfer is obtained as s t+1 ∈S, confirm And get a reward, adjust the parameters of the target language model based on the third reward function, the third reward function is as follows:

[0146]

[0147] Wherein, F is the length of the string of the text to be predicted.

[0148] In addition, if P(c i |s t )<φ, then as in the second case, a synonym can be selected from the preset dictionary as a candidate participle, and the candidate participle with the largest candidate adjacency probability is determined from the candidate participle as the target participle to replace the second participle. and C All are vocabulary.

[0149] Step d: According to the processing results of each of the above cycles, adjust the parameters of the target language model.

[0150] Specifically, during the data processing of the target language model, unsupervised training is performed simultaneously. The parameters of the target language model are adjusted based on the first reward function, the second reward function, and the third reward function. It represents the weight of the overall impact of decision actions in different cycles, that is, it is used to balance the importance of current rewards and future rewards. γ∈[0,1] represents the attenuation coefficient or discount factor. T is the string length of the text to be predicted and the word sequence to be predicted. t is the tth word segment in the word sequence to be predicted.

[0151] The Markov process in the target language model combines the self-attention mechanism to generate a string text of length M based on the text to be predicted. In the iterative processing of the predicted text, the Markov process aims to maximize the probability P(w1,w2,…,w M ), according to the chain rule, the joint probability can be decomposed into the product of a series of conditional probabilities, such as:

[0152]

[0153] In order to optimize the above objective function, the negative log-likelihood is usually minimized so that the product operation of the joint probability is transformed into an addition operation, such as:

[0154]

[0155] γ can be set to 1, so that the influence of the selected vocabulary in the current state on the vocabulary selection in the future state is taken into account in the design of the reward model of the target language model.

[0156] In one implementation, the methods of steps a to d above may be implemented using a Q-learning method.

[0157] Based on the above method flows and embodiments, the embodiment of the present application provides a model training method, wherein the above data processing method is performed by a target language model, and before the acquired text to be predicted is segmented to obtain a sequence of words to be predicted, an unlabeled text set is used to perform unsupervised training on the self-attention mechanism in the target language model, including:

[0158] Step 1: Perform word segmentation processing on each unlabeled text in the unlabeled text set to obtain an unlabeled word sequence, and perform word segmentation damage processing on the unlabeled word sequence to obtain a damaged word sequence;

[0159] Step 2: Based on the self-attention mechanism, the unlabeled semantic association relationship between each word in the unlabeled word sequence and the damaged semantic association relationship between each word in the damaged word sequence are obtained respectively, and the unlabeled adjacency probability is obtained according to the unlabeled semantic association relationship, and the damaged adjacency probability is obtained according to the damaged semantic association relationship;

[0160] Step 3: Based on the difference between the unlabeled neighbor probability and the damaged neighbor probability, adjust the parameters of the self-attention mechanism.

[0161] In one embodiment, a certain amount of unlabeled text can be collected in advance to obtain an unlabeled text set. Based on the unlabeled text set, the self-attention mechanism preliminarily learns the semantic correlation between word segmentations. Then, during the application process, a relatively accurate adjacency probability between each word segmentation of the text to be predicted can be given.

[0162] Based on the above model training method, the embodiment of the present application provides a method for obtaining a preset word library, including: segmenting each unlabeled text in the unlabeled text set to obtain each preset word segmentation; and establishing a preset word library based on each preset word segmentation. In other words, each unlabeled text in the obtained high-quality unlabeled text set can be segmented, and each word obtained is used as each preset word segmentation to form a preset word library.

[0163] In one embodiment, the text to be predicted and the unlabeled text involved in the aforementioned methods and embodiments are Cantonese texts.

[0164] Based on the above methods and embodiments, the present application provides a data processing method, such as Figure 6 As shown, including:

[0165] Step 601: collect news media text data, and classify the news media text data into different languages ​​using a language classification algorithm. Error correction can be performed on the news media text data after language classification using rule correction and manual combination to obtain a high-quality unlabeled text set.

[0166] Step 602: Perform preliminary training on the self-attention mechanism in the target language model based on the unlabeled text set.

[0167] Step 603: perform word segmentation processing on each unlabeled text in the unlabeled text set, obtain each preset word segmentation, and establish a preset word library.

[0168] Step 604: acquiring new news media text data again, classifying the language of the new news media text data using a language classification algorithm, obtaining a batch of texts to be predicted, and performing word segmentation processing on each text to be predicted to obtain a word sequence to be predicted.

[0169] Step 605: Process each word in the sequence of words to be predicted in order. For each word in the sequence of words to be predicted, perform the following steps respectively:

[0170] Step 6051: Obtain a first intermediate word sequence and a second intermediate word sequence in the word sequence to be predicted; wherein the first intermediate word sequence is a word sequence of each word before and including the first word in the word sequence to be predicted, and the second intermediate word sequence is a word sequence including the first intermediate word sequence and the second word in the word sequence to be predicted. The second word is a word adjacent to the first word in the word sequence to be predicted.

[0171] Step 6052: Concatenate the first intermediate word sequence with each preset word segment in the preset word library to obtain a concatenated word sequence corresponding to each preset word segment.

[0172] Step 6053: Based on the attention mechanism, extract the first semantic association relationship between the first participle and the preset participle in the concatenated word sequence, and the second semantic association relationship between the first participle and the second participle in the second intermediate word sequence, and obtain the semantic adjacency probability according to the first semantic association relationship, and obtain the original adjacency probability according to the second semantic association relationship.

[0173] Step 6054: Is the original adjacency probability less than the probability threshold? If so, execute step 6055; otherwise, execute step 6058.

[0174] Step 6055: determine whether the second word segment does not belong to a fixed language. If so, execute step 6056; otherwise, execute step 6057.

[0175] Step 6056: Based on the translation tool, the second participle is translated into a second participle belonging to a fixed language. The translated second participle can be used as the target participle, and the second participle that does not belong to the fixed language can be replaced. Alternatively, step 6057 can be performed, based on the preset screening conditions, a participle with a similar meaning to the translated second participle is selected from the preset dictionary as a candidate participle, the semantic adjacency probability of the preset participle matching the candidate participle is used as the candidate adjacency probability, and the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target participle, and the target participle is used to replace the second participle in the word sequence to be predicted.

[0176] Step 6057: Based on the preset screening condition and the second word segmentation, at least one candidate word segmentation is selected from the preset dictionary, the semantic adjacency probability of the preset word segmentation matching the candidate word segmentation is used as the candidate adjacency probability, and the candidate word segmentation corresponding to the largest candidate adjacency probability among the at least one candidate adjacency probability is used as the target word segmentation, and the target word segmentation is used to replace the second word segmentation in the word sequence to be predicted.

[0177] Step 6058: retain the second word segmentation in the text to be predicted.

[0178] Step 6059: Adjust the parameters of the target language model according to the processing result of the text to be predicted.

[0179] Step 606: Use the processed text corresponding to the to-be-predicted word sequence as the target text.

[0180] It should be noted that the above method steps are not unique. For example, step 602 can also be performed before step 603.

[0181] Based on the same concept, Figure 7 As shown, an embodiment of the present application provides a data processing device 700, which includes:

[0182] The word segmentation unit 701 is used to perform word segmentation processing on the acquired text to be predicted to obtain a word sequence to be predicted;

[0183] The processing unit 702 is configured to perform the following steps for each word segment in the to-be-predicted word sequence:

[0184] Obtaining semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, where the second word is the word adjacent to the first word in the word sequence to be predicted;

[0185] If the original adjacency probability is less than the probability threshold, selecting at least one candidate word segmentation from a preset dictionary based on a preset screening condition and the second word segmentation;

[0186] Taking the semantic adjacency probability of the preset participle that matches the candidate participle as the candidate adjacency probability, taking the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target participle, and replacing the second participle in the to-be-predicted word sequence with the target participle;

[0187] The processing unit 702 is further configured to use the processed text corresponding to the to-be-predicted word sequence as the target text.

[0188] Optionally, the processing unit 702 is specifically configured to:

[0189] Obtaining a first intermediate word sequence and a second intermediate word sequence in the to-be-predicted word sequence;

[0190] The first intermediate word sequence is a word sequence of words before the first word in the word sequence to be predicted and including the first word, and the second intermediate word sequence is a word sequence including the first intermediate word sequence and the second word in the word sequence to be predicted;

[0191] splicing the first intermediate word sequence with each preset word in the preset word library to obtain a spliced ​​word sequence corresponding to each preset word;

[0192] Based on the attention mechanism, a first semantic association relationship between the first participle and the preset participle in the concatenated word sequence and a second semantic association relationship between the first participle and the second participle in the second intermediate word sequence are extracted, and the semantic adjacency probability is obtained according to the first semantic association relationship, and the original adjacency probability is obtained according to the second semantic association relationship.

[0193] Optionally, the processing unit 702 is specifically configured to:

[0194] Determining a maximum candidate adjacent probability from the at least one candidate adjacent probability based on a Markov decision process;

[0195] The candidate participle corresponding to the maximum candidate adjacency probability is used as the target participle.

[0196] Optionally, the processing unit 702 is further configured to:

[0197] If the original adjacency probability is not less than the probability threshold, the second word segmentation in the to-be-predicted word sequence is retained.

[0198] Optionally, the processing unit 702 is further configured to:

[0199] If the second participle does not belong to a fixed language, the second participle is translated into a second participle belonging to a fixed language based on a translation tool.

[0200] Optionally, the method is performed by a target language model, and the processing unit 702 is further configured to: in the process of processing each word segment in the to-be-predicted word sequence:

[0201] If the original adjacency probability is not less than the probability threshold, adjusting the parameters of the target language model based on a first reward function;

[0202] If the original adjacency probability is less than the probability threshold, and the second word segment belongs to a fixed language, adjusting the parameters of the target language model based on a second reward function;

[0203] If the original adjacency probability is less than the probability threshold, and the second word segment does not belong to a fixed language, adjusting the parameters of the target language model based on a third reward function.

[0204] Optionally, the method is performed by a target language model, and the processing unit 702 is further used to: use an unlabeled text set to perform unsupervised training on a self-attention mechanism in the target language model, including:

[0205] Performing word segmentation processing on each unlabeled text in the unlabeled text set to obtain an unlabeled word sequence, and performing word segmentation damage processing on the unlabeled word sequence to obtain a damaged word sequence;

[0206] Based on the self-attention mechanism, the unlabeled semantic association relationship between each word in the unlabeled word sequence and the damaged semantic association relationship between each word in the damaged word sequence are respectively obtained, and the unlabeled adjacency probability is obtained according to the unlabeled semantic association relationship, and the damaged adjacency probability is obtained according to the damaged semantic association relationship;

[0207] Based on the difference between the unlabeled neighbor probability and the damaged neighbor probability, a parameter of the self-attention mechanism is adjusted.

[0208] Optionally, the word segmentation unit 701 is further used to: segment each unlabeled text in the unlabeled text set to obtain each preset word segmentation;

[0209] The processing unit 702 is further configured to: establish the preset word library based on the preset word segmentations.

[0210] Optionally, the method further includes: the text to be predicted and the unlabeled text are Cantonese texts.

[0211] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0212] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0215] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Perform word segmentation on the acquired text to be predicted to obtain a word sequence to be predicted; For each word segmentation in the to-be-predicted word sequence, the following steps are performed respectively: Obtaining semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, where the second word is the word adjacent to the first word in the word sequence to be predicted; If the original adjacency probability is less than the probability threshold, selecting at least one candidate word segmentation from a preset dictionary based on a preset screening condition and the second word segmentation; Taking the semantic adjacency probability of the preset participle that matches the candidate participle as the candidate adjacency probability, taking the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target participle, and replacing the second participle in the to-be-predicted word sequence with the target participle; The text corresponding to the processed word sequence to be predicted is used as the target text.

2. The method according to claim 1, characterized in that The obtaining of semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, includes: Obtaining a first intermediate word sequence and a second intermediate word sequence in the to-be-predicted word sequence; The first intermediate word sequence is a word sequence of words before the first word in the word sequence to be predicted and including the first word, and the second intermediate word sequence is a word sequence including the first intermediate word sequence and the second word in the word sequence to be predicted; splicing the first intermediate word sequence with each preset word in the preset word library to obtain a spliced ​​word sequence corresponding to each preset word; Based on the attention mechanism, a first semantic association relationship between the first participle and the preset participle in the concatenated word sequence and a second semantic association relationship between the first participle and the second participle in the second intermediate word sequence are extracted, and the semantic adjacency probability is obtained according to the first semantic association relationship, and the original adjacency probability is obtained according to the second semantic association relationship.

3. The method according to claim 1, characterized in that The step of taking the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability as the target participle includes: Determining a maximum candidate adjacency probability from the at least one candidate adjacency probability based on a Markov process; The candidate participle corresponding to the maximum candidate adjacency probability is used as the target participle.

4. The method according to claim 1, characterized in that Also includes: If the original adjacency probability is not less than the probability threshold, the second word segmentation in the to-be-predicted word sequence is retained.

5. The method according to claim 1, characterized in that Before selecting at least one candidate word segmentation from a preset dictionary based on the preset screening condition and the second word segmentation, the method further includes: If the second participle does not belong to a fixed language, the second participle is translated into a second participle belonging to a fixed language based on a translation tool.

6. The method according to any one of claims 1 to 5, characterized in that: The method is performed by a target language model, and in the process of processing each word segment in the to-be-predicted word sequence: If the original adjacency probability is not less than the probability threshold, adjusting the parameters of the target language model based on a first reward function; If the original adjacency probability is less than the probability threshold, and the second word segment belongs to a fixed language, adjusting the parameters of the target language model based on a second reward function; If the original adjacency probability is less than the probability threshold, and the second word segment does not belong to a fixed language, adjusting the parameters of the target language model based on a third reward function.

7. The method according to any one of claims 1 to 5, characterized in that: The method is performed by a target language model. Before the acquired text to be predicted is segmented to obtain a sequence of words to be predicted, an unlabeled text set is used to perform unsupervised training on a self-attention mechanism in the target language model, including: Performing word segmentation processing on each unlabeled text in the unlabeled text set to obtain an unlabeled word sequence, and performing word segmentation damage processing on the unlabeled word sequence to obtain a damaged word sequence; Based on the self-attention mechanism, the unlabeled semantic association relationship between each word in the unlabeled word sequence and the damaged semantic association relationship between each word in the damaged word sequence are respectively obtained, and the unlabeled adjacency probability is obtained according to the unlabeled semantic association relationship, and the damaged adjacency probability is obtained according to the damaged semantic association relationship; Based on the difference between the unlabeled neighbor probability and the damaged neighbor probability, a parameter of the self-attention mechanism is adjusted.

8. The method according to claim 7, characterized in that Also includes: Performing word segmentation on each unlabeled text in the unlabeled text set to obtain each preset word segmentation; Based on the preset word segmentations, the preset word library is established.

9. The method according to claim 7, characterized in that Also includes: The to-be-predicted text and the unlabeled text are Cantonese texts.

10. A data processing device, characterized in that: The device comprises: A word segmentation unit is used to perform word segmentation processing on the acquired text to be predicted to obtain a word sequence to be predicted; The processing unit is used to iteratively process each word segment in the word sequence to be predicted, and respectively perform the following steps: Obtaining semantic adjacency probabilities of the first word in the word sequence to be predicted and each preset word in the preset word library, and original adjacency probabilities of the first word and the second word, where the second word is the word adjacent to the first word in the word sequence to be predicted; If the original adjacency probability is less than the probability threshold, selecting at least one candidate word segmentation from a preset dictionary based on a preset screening condition and the second word segmentation; The semantic adjacency probability of a preset participle matching the candidate participle is used as the candidate adjacency probability, and the candidate participle corresponding to the largest candidate adjacency probability among at least one candidate adjacency probability is used as the target participle, and the target participle is used to replace the second participle in the to-be-predicted word sequence; the processing unit is further used to use the text corresponding to the processed to-be-predicted word sequence as the target text.

Images