Chinese word segmentation method and device, equipment and storage medium
By combining syntactic analysis, semantic dependence analysis and attention mechanism, feature fusion and using deep learning models, the problem of difficulty in processing proper nouns and new nouns in the financial field is solved, and the accuracy and efficiency of word segmentation are improved.
Patent Information
- Application Number
- CN202411020129.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2025-01-17
AI Technical Summary
The Chinese word segmentation model in the financial field is difficult to deal with proper nouns and new nouns, resulting in a low accuracy of word segmentation results.
By combining syntactic analysis, semantic dependency analysis and attention mechanism, word vectors, syntactic vectors and dependency vectors are obtained, feature fusion is performed, and word segmentation is used to process using deep learning models.
It improves the accuracy and computational efficiency of word segmentation and enhances the ability to understand financial texts.
Smart Images
Figure CN119323218A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of financial technology, and in particular to a Chinese word segmentation method, a Chinese word segmentation device, a computer device, and a computer-readable storage medium. Background Art
[0002] Chinese word segmentation is an important task in natural language processing, and its purpose is to segment text into independent words or phrases according to certain rules. Especially in the financial field, Chinese word segmentation can play an important role in text classification or information extraction. By extracting important information, question-answering robots or intelligent customer service in the financial field can understand the user's semantics faster and give more accurate answers.
[0003] In the related technology, since there are many proper nouns in the financial field and new nouns are constantly updated, these words cannot be processed by conventional Chinese word segmentation models, which leads to low accuracy of word segmentation results. Summary of the invention
[0004] The present application provides a Chinese word segmentation method, a Chinese word segmentation device, a computer device and a computer-readable storage medium, aiming to improve the accuracy of text segmentation results in the financial field.
[0005] To achieve the above object, the present application provides a Chinese word segmentation method, the method comprising:
[0006] Acquire a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results;
[0007] Encoding the target text, the syntactic analysis result, and the semantic dependency analysis result respectively to obtain corresponding word vectors, syntactic vectors, and dependency relationship vectors;
[0008] Based on the attention mechanism, feature fusion is performed on the word vector, the syntax vector and the dependency vector to obtain an initial feature vector;
[0009] The initial feature vector is analyzed by a target deep learning model to obtain a word segmentation result of the target text.
[0010] To achieve the above purpose, the present application also provides a Chinese word segmentation device, including:
[0011] An acquisition module, the acquisition module is used to acquire a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results;
[0012] An encoding module, the encoding module is used to encode the target text, the syntactic analysis result and the semantic dependency analysis result respectively to obtain corresponding word vectors, syntactic vectors and dependency relationship vectors;
[0013] A feature fusion module, the feature fusion module is used to perform feature fusion on the word vector, the syntax vector and the dependency vector based on an attention mechanism to obtain an initial feature vector;
[0014] A word segmentation module is used to analyze the initial feature vector through a target deep learning model to obtain a word segmentation result of the target text.
[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of any one of the Chinese word segmentation methods provided in the embodiments of the present application when executing the computer program.
[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of any one of the Chinese word segmentation methods provided in the embodiments of the present application.
[0017] The Chinese word segmentation method, Chinese word segmentation device, computer equipment and computer-readable storage medium disclosed in the embodiments of the present application can perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results. The target text, syntactic analysis results and semantic dependency analysis results are encoded respectively to obtain corresponding word vectors, syntactic vectors and dependency vectors, and then the word vectors, syntactic vectors and dependency vectors are feature fused based on the attention mechanism to obtain the initial feature vector, and finally the initial feature vector is analyzed by the target deep learning model to obtain the word segmentation result of the target text. The present application can perform multi-level feature extraction and fusion on the target text by combining syntactic analysis, semantic dependency analysis and attention mechanism, generate an initial feature vector, and perform word segmentation processing through a deep learning model, thereby improving the accuracy and computational efficiency of word segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a scenario diagram of a Chinese word segmentation method provided in an embodiment of the present application;
[0020] Figure 2 It is a flowchart of a Chinese word segmentation method provided in an embodiment of the present application;
[0021] Figure 3 It is a schematic diagram of a process for obtaining a target deep learning model provided in an embodiment of the present application;
[0022] Figure 4 It is a schematic diagram of a process of obtaining an iterated target deep learning model provided in an embodiment of the present application;
[0023] Figure 5 is a schematic block diagram of a Chinese word segmentation device provided in an embodiment of the present application;
[0024] Figure 6 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.
[0027] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0029] like Figure 1 As shown, the Chinese word segmentation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 can obtain the target text, and perform syntactic analysis and semantic dependency analysis on the target text respectively, to obtain corresponding syntactic analysis results and semantic dependency analysis results. Then, the target text, the syntactic analysis results and the semantic dependency analysis results are encoded respectively to obtain corresponding word vectors, syntactic vectors and dependency vectors, and the word vectors, syntactic vectors and dependency vectors are feature fused based on the attention mechanism to obtain the initial feature vector. Finally, the initial feature vector is analyzed by the target deep learning model to obtain the word segmentation result of the target text, and the word segmentation result of the target text is sent to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
[0030] See also Figure 2 , Figure 2 Schematic diagram of a Chinese word segmentation method provided in the embodiment of the present application. Figure 2 As shown, the method includes steps S11 to S14.
[0031] Step S11: Obtain a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results.
[0032] The target text may be any natural language text, such as a sentence, a paragraph or an article, etc., and this application does not limit this.
[0033] Syntactic analysis is to analyze the structure of a sentence and determine the syntactic relationship between each word. The syntactic relationship includes subject-predicate relationship, object relationship, modification relationship, etc. For example, the target text can be syntactically analyzed by an NLP tool library (such as spaCy or StanfordNLP) to obtain the corresponding syntactic analysis results, which can be presented in the form of a tree structure or a dependency graph.
[0034] Semantic dependency analysis is to analyze the semantic relationship between words in a sentence to determine deep semantic information such as who performed what action on whom. For example, a semantic dependency analysis can be performed on the target text using a specific tool library (such as LTP or SUDAG) to obtain a semantic analysis result, where the semantic analysis result is usually presented in the form of a graph structure or a relationship pair.
[0035] Optionally, syntactic analysis and semantic dependency analysis are performed on the target text respectively to obtain corresponding syntactic analysis results and semantic dependency analysis results, including: performing a preprocessing operation on the target text to obtain a preprocessed target text, wherein the preprocessing operation includes at least one of a stop word removal operation, a punctuation removal operation, and a special character removal operation; performing syntactic analysis and semantic dependency analysis on the preprocessed target text respectively to obtain corresponding syntactic analysis results and semantic dependency analysis results.
[0036] Specifically, at least one of the operations of removing stop words, removing punctuation marks, and removing special characters can be performed on the target text to obtain the preprocessed target text, so as to prevent invalid data from affecting the subsequent word segmentation of the target text. Furthermore, the preprocessed target text can be subjected to syntactic analysis and semantic dependency analysis respectively to obtain corresponding syntactic analysis results and semantic dependency analysis results.
[0037] The above method can more deeply understand and process the target text by performing syntactic analysis and semantic dependency analysis on the target text, providing a basis for subsequent word segmentation processing.
[0038] Step S12: Encode the target text, the syntactic analysis result and the semantic dependency analysis result respectively to obtain corresponding word vectors, syntactic vectors and dependency relationship vectors.
[0039] Optionally, the target text, syntactic analysis results and semantic dependency analysis results are encoded respectively to obtain corresponding word vectors, syntactic vectors and dependency vectors, including: encoding the target text through a pre-trained language model to obtain word vectors; encoding the syntactic analysis results and speech dependency analysis results through a long short-term memory network model to obtain corresponding syntactic vectors and dependency vectors.
[0040] Specifically, the target text can be encoded through a pre-trained language model (such as BERT, GPT, XLNet, etc.) to obtain a word vector. Among them, the word vector can capture the semantic information and contextual relationship of the target text. Furthermore, the syntactic analysis results and speech dependency analysis results can be encoded through a long short-term memory network model to obtain the corresponding syntactic vector and dependency relationship vector.
[0041] It should be noted that the Long Short-Term Memory (LSTM) network model is a special recursive neural network used to process and predict long-distance dependency problems in time series data. The LSTM network model can effectively capture and retain important information in long time series data and is widely used in natural language processing, speech recognition, time series prediction and other fields.
[0042] The above method, by combining the pre-trained language model and the long short-term memory network model, can capture the semantic and syntactic information in the target text, so that in the word segmentation processing of the target text, the obtained word vector, syntactic vector and dependency vector can help the model more accurately identify and extract the key information of the target text.
[0043] Step S13: Based on the attention mechanism, feature fusion is performed on the word vector, syntax vector and dependency vector to obtain an initial feature vector.
[0044] Optionally, feature fusion is performed on word vectors, syntactic vectors and dependency vectors based on an attention mechanism to obtain an initial feature vector, including: performing attention calculation on word vectors and syntactic vectors based on the attention mechanism to obtain a first feature vector; performing attention calculation on word vectors and dependency vectors based on the attention mechanism to obtain a second feature vector; and feature fusion of the word vectors, the first feature vector and the second feature vector to obtain an initial feature vector.
[0045] Among them, the attention mechanism can generate a weight matrix to calculate the correlation between vectors.
[0046] Specifically, the word vector and the syntactic vector can be dot-producted by the attention mechanism, and then the attention weights can be obtained by the softmax function, and then these weights can be applied to the syntactic vector to obtain the first feature vector. The word vector and the dependency vector can also be dot-producted by the attention mechanism, and the attention weights can be obtained by the softmax function, and then these weights can be applied to the dependency vector to obtain the second feature vector. In this way, the word vector, the first feature vector and the second feature vector obtained above can be fused to obtain the initial feature vector.
[0047] It can be understood that the initial feature vector generated in the above manner contains the basic information, syntactic information and dependency information of the word, and can therefore be used for subsequent word segmentation tasks of the target text.
[0048] Step S14: Analyze the initial feature vector through the target deep learning model to obtain the word segmentation result of the target text.
[0049] Optionally, the target deep learning model includes a fully connected layer and a conditional random field layer. The initial feature vector is analyzed by the target deep learning model to obtain a word segmentation result of the target text, including: performing feature conversion on the initial feature vector through the fully connected layer to obtain a target feature vector, wherein the dimension of the target feature vector is the same as the input dimension of the conditional random field layer; inputting the target feature vector into the conditional random field layer, and decoding the target feature vector through the conditional random field layer to obtain an analysis result of the target text.
[0050] Specifically, the fully connected layer can convert the initial feature vector into a target feature vector so that its dimension is the same as the input dimension of the conditional random field layer, ensuring the smoothness of the data flow and the stability of the model. Furthermore, the target feature vector can be input into the conditional random field layer, and the target feature vector can be decoded through the conditional random field layer to obtain the analysis result of the target text. Among them, the conditional random field layer can consider global information, find the optimal label sequence through the decoding process, and improve the accuracy and consistency of the word segmentation results. At the same time, the conditional random field layer can also effectively process the boundary information of words and improve the model's adaptability to the target text.
[0051] In an embodiment of the present application, by introducing a fully connected layer in the target deep learning model, the validity of the target feature vector can be ensured. At the same time, through the global optimization and decoding process of the conditional random field layer, the accuracy and computational efficiency of word segmentation can be improved.
[0052] The Chinese word segmentation method disclosed in the embodiment of the present application can perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results. The target text, syntactic analysis results and semantic dependency analysis results are encoded respectively to obtain corresponding word vectors, syntactic vectors and dependency vectors, and then the word vectors, syntactic vectors and dependency vectors are subjected to feature fusion based on the attention mechanism to obtain the initial feature vector, and finally the initial feature vector is analyzed through the target deep learning model to obtain the word segmentation result of the target text. The present application can perform multi-level feature extraction and fusion on the target text by combining syntactic analysis, semantic dependency analysis and attention mechanism, generate an initial feature vector, and perform word segmentation processing through a deep learning model, thereby improving the accuracy and computational efficiency of word segmentation.
[0053] Please continue reading Figure 3 , Figure 3 Schematic diagram of a process for obtaining a target deep learning model provided in an embodiment of the present application. Figure 3 As shown, the target deep learning model can be obtained through steps S21 to S23.
[0054] Step S21: Obtain a training data set and an initial deep learning model, wherein the training data set includes a number of texts.
[0055] Step S22: annotate the training data set to obtain an annotation result, wherein the annotation result includes a word segmentation result corresponding to each text.
[0056] Step S23: Train the initial deep learning model using the training data set and the annotation results to obtain a target deep learning model.
[0057] It should be noted that this application does not limit the deep learning model, such as BERT, BiLSTM, CRF, GPT and ELMo, etc. This application takes the deep learning model of CRF as an example for explanation.
[0058] The CRF (conditional random field algorithm) model is a probabilistic graph model that is often used in tasks such as sequence labeling or text processing. The CRF model is an undirected graph model used to model the conditional probability distribution between a label sequence and an input sequence. The main idea of the CRF model is to use contextual information to jointly label the elements in the sequence. It takes into account the dependencies between adjacent elements and the conditional dependencies between the features of the current element and the label. By learning the relationship between the features and labels in the training data, the CRF model can predict the most likely label for each input element, thereby completing the sequence labeling task. In addition, the CRF model improves the performance of the model by considering contextual information and conditional dependencies in the sequence labeling task, making label prediction more accurate. This makes the CRF model a commonly used model for processing tasks such as named entity recognition, part-of-speech tagging, syntactic analysis, and text segmentation in natural language processing.
[0059] Specifically, a training data set containing several texts may be collected for training. For example, the training data set may be obtained by manual collection, web crawlers, or public data sets, etc., which is not limited in this application.
[0060] Furthermore, each text can be annotated to obtain the word segmentation results corresponding to each text. The word segmentation results are then used as labels for the group of input data, and then each group of training data sets with labels is input into the CRF model for supervised learning. When the training end conditions are met, such as when the number of training times reaches the number threshold or the output accuracy of the model reaches the accuracy threshold, the training ends and a trained event extraction model is obtained.
[0061] In the embodiment of the present application, the training data set and the annotation results can be input into the CRF model for supervised learning, and then the target deep learning model can be trained to obtain the target text segmentation result.
[0062] Please continue reading Figure 4 , Figure 4 is a flow chart of obtaining a target deep learning model after iteration provided by an embodiment of the present application. Figure 4 As shown, the iterated target deep learning model can be obtained through steps S24 to S26.
[0063] Step S24: Iteratively train the target deep learning model based on the training data set and the annotation results to extract data features and calculate the loss function.
[0064] Step S25: Iteratively train the loss function using a preset method to reduce the value of the loss function until the expected threshold value is met.
[0065] Step S26: Based on the loss function after iterative training, obtain the target deep learning model after iteration.
[0066] It can be understood that in order to train a target deep learning model with higher accuracy, the target deep learning model can be repeatedly iterated to continuously reduce the loss function until the loss function meets the expected threshold requirement, and then a more accurate word segmentation result corresponding to the target text can be obtained based on the iterated target deep learning model.
[0067] It should be noted that the present application does not limit the above-mentioned preset method and expected threshold. For example, the preset method may be a gradient descent algorithm, a batch gradient descent algorithm, a stochastic gradient descent algorithm, etc. The present application takes the gradient descent algorithm as an example for explanation.
[0068] The purpose of the gradient descent algorithm is to find the minimum value of the loss function by iteration, or to converge to the minimum value. In a geometric sense, the gradient descent algorithm is that the gradient decreases fastest in the direction opposite to the vector where the function changes the fastest, so it is easier to find the minimum value of the function. Based on this, in an embodiment of the present application, the target deep learning model can be repeatedly iterated and trained by the gradient descent algorithm so that the loss function is continuously reduced, thereby reducing the error of the calculation result.
[0069] In an embodiment of the present application, the target deep learning model is repeatedly iteratively trained using a gradient descent algorithm so that the loss function is continuously reduced to obtain an iterative target deep learning model, and then a more accurate word segmentation result of the target text can be obtained based on the iterative target deep learning model.
[0070] See also Figure 5 , Figure 5 1 is a schematic block diagram of a Chinese word segmentation device provided in an embodiment of the present application. The Chinese word segmentation device can be configured in a server to execute the aforementioned Chinese word segmentation method.
[0071] like Figure 5 As shown, the Chinese word segmentation device 200 includes: an acquisition module 201, an encoding module 202, a feature fusion module 203 and a word segmentation module 204.
[0072] The acquisition module 201 is used to acquire a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results;
[0073] The encoding module 202 is used to perform entity recognition on the medical text to obtain an entity recognition result; and determine the trigger word corresponding to the medical text based on the attention weight matrix;
[0074] A feature fusion module 203, configured to perform feature fusion on the word vector, the syntactic vector and the dependency vector based on an attention mechanism to obtain an initial feature vector;
[0075] The word segmentation module 204 is used to analyze the initial feature vector through a target deep learning model to obtain a word segmentation result of the target text.
[0076] The acquisition module 201 is also used to perform a preprocessing operation on the target text to obtain a preprocessed target text, wherein the preprocessing operation includes at least one of a stop word removal operation, a punctuation removal operation, and a special character removal operation; and perform syntactic analysis and semantic dependency analysis on the preprocessed target text to obtain the corresponding syntactic analysis results and the semantic dependency analysis results.
[0077] The encoding module 202 is also used to encode the target text through a pre-trained language model to obtain the word vector; encode the syntactic analysis result and the speech dependency analysis result through a long short-term memory network model to obtain the corresponding syntactic vector and the dependency relationship vector.
[0078] The feature fusion module 203 is also used to perform attention calculation on the word vector and the syntactic vector based on the attention mechanism to obtain a first feature vector; perform attention calculation on the word vector and the dependency vector based on the attention mechanism to obtain a second feature vector; and perform feature fusion on the word vector, the first feature vector and the second feature vector to obtain the initial feature vector.
[0079] The word segmentation module 204 is also used to perform feature conversion on the initial feature vector through the fully connected layer to obtain a target feature vector, wherein the dimension of the target feature vector is the same as the input dimension of the conditional random field layer; input the target feature vector into the conditional random field layer, and decode the target feature vector through the conditional random field layer to obtain the analysis result of the target text.
[0080] The acquisition module 201 is also used to obtain a training data set and an initial deep learning model, wherein the training data set includes a number of texts; annotate the training data set to obtain an annotation result, wherein the annotation result includes a word segmentation result corresponding to each text; and train the initial deep learning model through the training data set and the annotation result to obtain the target deep learning model.
[0081] The acquisition module 201 is also used to iteratively train the target deep learning model based on the training data set and the annotation results to extract data features, and calculate the loss function; iteratively train the loss function using a preset method for the purpose of reducing the value of the loss function until the expected threshold requirement is met; based on the loss function after iterative training, obtain the iterative target deep learning model.
[0082] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0083] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.
[0084] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 6 Runs on the computer device shown.
[0085] See also Figure 6 , Figure 6 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0086] like Figure 6As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.
[0087] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any Chinese word segmentation method.
[0088] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0089] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any Chinese word segmentation method.
[0090] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0091] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0092] Among them, in some embodiments, the processor will obtain a target text, and perform syntactic analysis and semantic dependency analysis on the target text respectively to obtain corresponding syntactic analysis results and semantic dependency analysis results; encode the target text, the syntactic analysis results and the semantic dependency analysis results respectively to obtain corresponding word vectors, syntactic vectors and dependency vectors; perform feature fusion on the word vectors, the syntactic vectors and the dependency vectors based on the attention mechanism to obtain an initial feature vector; analyze the initial feature vector through a target deep learning model to obtain a word segmentation result of the target text.
[0093] In some embodiments, the processor is also used to perform a preprocessing operation on the target text to obtain a preprocessed target text, wherein the preprocessing operation includes at least one of a stop word removal operation, a punctuation removal operation, and a special character removal operation; and perform syntactic analysis and semantic dependency analysis on the preprocessed target text to obtain the corresponding syntactic analysis results and the semantic dependency analysis results.
[0094] In some embodiments, the processor is further used to encode the target text through a pre-trained language model to obtain the word vector; encode the syntactic analysis results and the speech dependency analysis results through a long short-term memory network model to obtain the corresponding syntactic vector and the dependency relationship vector.
[0095] In some embodiments, the processor is further used to perform attention calculation on the word vector and the syntactic vector based on the attention mechanism to obtain a first feature vector; perform attention calculation on the word vector and the dependency vector based on the attention mechanism to obtain a second feature vector; and perform feature fusion on the word vector, the first feature vector, and the second feature vector to obtain the initial feature vector.
[0096] In some embodiments, the processor is also used to obtain a training data set and an initial deep learning model, wherein the training data set includes a number of texts; annotate the training data set to obtain an annotation result, wherein the annotation result includes a word segmentation result corresponding to each text; and train the initial deep learning model using the training data set and the annotation result to obtain the target deep learning model.
[0097] In some embodiments, the processor is also used to perform feature conversion on the initial feature vector through the fully connected layer to obtain a target feature vector, wherein the dimension of the target feature vector is the same as the input dimension of the conditional random field layer; input the target feature vector into the conditional random field layer, and decode the target feature vector through the conditional random field layer to obtain an analysis result of the target text.
[0098] In some embodiments, the processor is also used to iteratively train the target deep learning model based on the training data set and the annotation results to extract data features, and calculate the loss function; use a preset method to iteratively train the loss function for the purpose of reducing the value of the loss function until the expected threshold requirement is met; based on the loss function after iterative training, obtain the iterative target deep learning model.
[0099] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any one of the Chinese word segmentation methods provided in the embodiments of the present application is implemented.
[0100] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0101] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.
[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A Chinese word segmentation method, characterized in that: The method comprises: Acquire a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results; Encoding the target text, the syntactic analysis result, and the semantic dependency analysis result respectively to obtain corresponding word vectors, syntactic vectors, and dependency relationship vectors; Based on the attention mechanism, feature fusion is performed on the word vector, the syntax vector and the dependency vector to obtain an initial feature vector; The initial feature vector is analyzed by a target deep learning model to obtain a word segmentation result of the target text.
2. The method according to claim 1, characterized in that The target text is subjected to syntactic analysis and semantic dependency analysis respectively to obtain corresponding syntactic analysis results and semantic dependency analysis results, including: Performing a preprocessing operation on the target text to obtain a preprocessed target text, wherein the preprocessing operation includes at least one of an operation of removing stop words, an operation of removing punctuation marks, and an operation of removing special characters; The preprocessed target text is subjected to syntactic analysis and semantic dependency analysis respectively to obtain the corresponding syntactic analysis result and the semantic dependency analysis result.
3. The method according to claim 1, characterized in that The target text, the syntactic analysis result and the semantic dependency analysis result are respectively encoded to obtain corresponding word vectors, syntactic vectors and dependency relationship vectors, including: Encoding the target text through a pre-trained language model to obtain the word vector; The syntactic analysis result and the speech dependency analysis result are encoded through a long short-term memory network model to obtain the corresponding syntactic vector and the dependency relationship vector.
4. The method according to claim 1, characterized in that: The feature fusion of the word vector, the syntax vector and the dependency vector based on the attention mechanism to obtain an initial feature vector includes: Performing attention calculation on the word vector and the syntactic vector based on the attention mechanism to obtain a first feature vector; Performing attention calculation on the word vector and the dependency vector based on the attention mechanism to obtain a second feature vector; The word vector, the first feature vector and the second feature vector are subjected to feature fusion to obtain the initial feature vector.
5. The method according to claim 1, characterized in that The target deep learning model includes a fully connected layer and a conditional random field layer. The initial feature vector is analyzed by the target deep learning model to obtain the word segmentation result of the target text, including: Performing feature transformation on the initial feature vector through the fully connected layer to obtain a target feature vector, wherein the dimension of the target feature vector is the same as the input dimension of the conditional random field layer; The target feature vector is input into the conditional random field layer, and the target feature vector is decoded by the conditional random field layer to obtain the analysis result of the target text.
6. The method according to claim 1, characterized in that The method further comprises: Obtaining a training data set and an initial deep learning model, wherein the training data set includes a plurality of texts; Annotating the training data set to obtain an annotation result, wherein the annotation result includes a word segmentation result corresponding to each text; The initial deep learning model is trained using the training data set and the annotation results to obtain the target deep learning model.
7. The method according to claim 6, characterized in that After the initial deep learning model is trained by the training data set and the annotation result to obtain the target deep learning model, the method includes: Iteratively train the target deep learning model based on the training data set and the annotation results to extract data features, and calculate the loss function; Iteratively training the loss function using a preset method for the purpose of reducing the value of the loss function until the expected threshold value is met; Based on the loss function after iterative training, the target deep learning model after iteration is obtained.
8. A Chinese word segmentation device, characterized in that: The Chinese word segmentation device comprises: An acquisition module, the acquisition module is used to acquire a target text, and perform syntactic analysis and semantic dependency analysis on the target text to obtain corresponding syntactic analysis results and semantic dependency analysis results; An encoding module, the encoding module is used to encode the target text, the syntactic analysis result and the semantic dependency analysis result respectively to obtain corresponding word vectors, syntactic vectors and dependency relationship vectors; A feature fusion module, the feature fusion module is used to perform feature fusion on the word vector, the syntax vector and the dependency vector based on an attention mechanism to obtain an initial feature vector; A word segmentation module is used to analyze the initial feature vector through a target deep learning model to obtain a word segmentation result of the target text.
9. A computer device, characterized in that: include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the Chinese word segmentation method as described in any one of claims 1 to 7 by running the program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the Chinese word segmentation method according to any one of claims 1 to 7.