Word segmentation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of natural language processing, and discloses a word segmentation method and device, equipment and a storage medium. The method comprises the following steps: for any sentence in a text, segmenting the sentence to obtain a plurality of word segmentation results corresponding to the sentence; for any word segmentation result in the plurality of word segmentation results, according to a word vector of any word segmentation in the word segmentation result, based on a frequency domain vector converted by the word vector and a font vector of the word segmentation, determining fusion features corresponding to the word segmentation; wherein the frequency domain vector of the segmented word is obtained by processing the word vector of the segmented word based on Fourier transform, and the font vector of the segmented word is obtained according to the stroke of the segmented word; and determining a target word segmentation result corresponding to the sentence from the plurality of word segmentation results according to the fusion feature corresponding to each word segmentation in each word segmentation result in the plurality of word segmentation results. Therefore, the accuracy of the word segmentation result can be improved.

Description

Technical Field

[0001] This application relates to the technical field of natural language processing, and in particular, to a word segmentation method, apparatus, device, and storage medium. Background Art

[0002] Chinese word segmentation is the most basic function in the process of natural language processing. Usually, a word segmentation tool is used to segment the sentences in the text. Then, different word segmentations may have different meanings in different contexts, which are commonly known as ambiguous words. Ambiguous words mainly include combined ambiguous words, intersection ambiguous words, and true ambiguous words. Among them, combined ambiguous words are caused by different fine-grained word segmentations resulting in different word segmentation results. For example, for "the People's Republic of China", the coarse-grained word segmentation result is "the People's Republic of China", and the fine-grained word segmentation result is "China / People / Republic". Intersection ambiguous words are those where different segmentation results share the same characters, and different combinations before and after lead to different segmentation results. For example, for "Nanjing Yangtze River Bridge", it can be segmented as "Nanjing City / Yangtze River / Big Bridge", or it can be segmented as "Nanjing / Mayor / Yangtze River Bridge". True ambiguous words are those where the grammar is correct, and even manual segmentation will produce ambiguity. For example, "The badminton auction is over" can be segmented as "Badminton / Auction / Over", or it can be segmented as "Badminton racket / Sell / Over". For the same sentence, different segmentation methods may have multiple different word segmentation results. How to improve the accuracy of the word segmentation results still needs further research. Summary of the Invention

[0003] Embodiments of the present invention provide a word segmentation method, apparatus, device, and storage medium. By extracting the word vector, frequency domain vector, and glyph vector of any word in each of multiple word segmentation results, the fusion feature corresponding to the word segmentation is determined, and based on the fusion feature corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results to improve the accuracy of the word segmentation result.

[0004] In a first aspect, an embodiment of the present application provides a word segmentation method, which includes: for any sentence in the text, segment the sentence to obtain multiple word segmentation results corresponding to the sentence; for any word segmentation result in the multiple word segmentation results, determine the fusion feature corresponding to the word segmentation according to the word vector of any word in the word segmentation result, the frequency domain vector converted based on the word vector, and the glyph vector of the word segmentation; wherein, the frequency domain vector of the word segmentation is obtained by processing the word vector of the word segmentation through Fourier transform, and the glyph vector of the word segmentation is obtained according to the strokes of the word segmentation; determine the target word segmentation result corresponding to the sentence from the multiple word segmentation results according to the fusion feature corresponding to each word in each of the multiple word segmentation results.

[0005] Using the above method, the word vector, frequency domain vector, and character shape vector of any word in each of the multiple word segmentation results are extracted to determine the fusion features corresponding to the word segmentation. Based on the fusion features corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results to improve the accuracy of the word segmentation results.

[0006] In one optional implementation, the frequency domain vector of any word segment in the word segmentation result is obtained by processing the word vector of the word segmentation based on Fourier transform, including: the word vector of the word segmentation has a dimension of n, and the frequency domain vector of the word segmentation has the same dimension as the word vector of the word segmentation; based on the word vector of the word segmentation and the Fourier transform, the real part and imaginary part of the k-th dimension in the n dimensions of the frequency domain vector of the word segmentation are calculated; the modulus corresponding to the real part and imaginary part of the k-th dimension is calculated to obtain the value corresponding to the k-th dimension in the frequency domain vector of the word segmentation, where n is an integer greater than or equal to k.

[0007] Using the above method, determining the corresponding frequency domain vector based on the word vectors of word segmentation makes the vectors more robust to handling ambiguous words. Introducing "spectral domain frequency transformation" treats the frequency sequence as a signal and performs frequency domain decomposition, separating it into real and imaginary parts. This differs from purely statistical methods and can better quantify the "frequency oscillations" of ambiguity, helping to improve the accuracy of frequency domain vector extraction.

[0008] In one optional implementation, the glyph vector of any word in the word segmentation result is obtained based on the strokes of the word segmentation, including: constructing a stroke vector space based on basic stroke types, where each dimension of the stroke vector space corresponds to a stroke type; for any character in the word segmentation, counting the number of different stroke types in the character, and obtaining the stroke vector corresponding to the character according to the stroke vector space; and normalizing the stroke vector corresponding to each character in the word segmentation to obtain the glyph vector of the word segmentation.

[0009] Using the above approach, glyph vectors based on basic stroke types primarily serve as visual feature enhancers, injecting "writing geometry" information into Chinese characters to help resolve ambiguity and boundary issues in Chinese word segmentation. Glyph vectors mainly encode visual structure into a low-dimensional numerical representation by capturing the distribution and density of stroke styles (such as horizontal, vertical, and left-falling strokes). Furthermore, glyph vectors compensate for the shortcomings of word vectors (semantic) and frequency domain vectors, providing complementary information at the visual-structural level.

[0010] In one optional implementation, determining the fusion feature corresponding to the segmented word based on the word vector of any segmented word in the segmentation result, the frequency domain vector based on word vector transformation, and the character shape vector of the segmented word includes: determining the context vector corresponding to the sentence in the text; determining the fusion feature corresponding to the segmented word based on the word vector of any segmented word in the segmentation result, the frequency domain vector based on word vector transformation, the character shape vector of the segmented word, and their respective weights; wherein the respective weights are determined based on the word vector, the frequency domain vector, the character shape vector, and the context vector.

[0011] Using the above method, the word vectors, frequency domain vectors, and character shape vectors of word segmentation are combined to obtain the corresponding fusion features. The word vectors of word segmentation help to capture semantic information, the frequency domain vectors of word segmentation help to retain more useful information, and the character shape vectors of word segmentation mainly capture the distribution and density of the stroke styles of Chinese characters (such as horizontal, vertical, and left-falling strokes), encode the visual structure into a low-dimensional numerical representation, which helps to capture the geometric features of word segmentation and further improve the accuracy of fusion features.

[0012] In one optional implementation, the respective weights are determined based on the word vector, the frequency domain vector, the glyph vector, and the context vector, including: determining a first attention score corresponding to the word segment based on the word vector and the context vector; determining a second attention score corresponding to the word segment based on the frequency domain vector and the context vector; determining a third attention score corresponding to the word segment based on the glyph vector and the context vector; processing the first attention score, the second attention score, and the third attention score respectively using an exponential function and then summing them to obtain a comprehensive value; obtaining the weight corresponding to the word vector of the word segment based on the word vector and the comprehensive value; obtaining the weight corresponding to the frequency domain vector of the word segment based on the frequency domain vector of the word segment and the comprehensive value; and obtaining the weight corresponding to the glyph vector of the word segment based on the glyph vector of the word segment and the comprehensive value.

[0013] Using the above method, the weights of each word vector, frequency domain vector, and character shape vector are determined based on their respective vectors and context vectors. Associating any word in the segmentation result with the context vector of the text helps to improve the depth of semantic understanding of the sentence, thereby improving the accuracy of the segmentation results.

[0014] In one optional implementation, determining the target word segmentation result corresponding to the sentence from the plurality of word segmentation results based on the fusion features corresponding to each word in each of the plurality of word segmentation results includes: for any word segmentation result in the plurality of word segmentation results, calculating the similarity between the fusion features corresponding to each word in the word segmentation result and the fusion features corresponding to the next word in the word segmentation result; obtaining the total score corresponding to the word segmentation result based on the similarity between each adjacent word segmentation result; and determining the target word segmentation result corresponding to the sentence from the plurality of word segmentation results based on the total score corresponding to each word segmentation result in the plurality of word segmentation results.

[0015] Using the above method, the total score corresponding to each of the multiple segmentation results is calculated, and the target segmentation result for the sentence is determined from the multiple segmentation results based on the total score of each segmentation result. In this way, the optimal segmentation result can be selected as the target segmentation result from multiple segmentation results.

[0016] Secondly, embodiments of this application provide a word segmentation device, the device comprising: The segmentation module is used to segment any sentence in the text to obtain multiple word segmentation results corresponding to the sentence; The determination module is used to determine the fusion feature corresponding to any one of the multiple word segmentation results, based on the word vector, the frequency domain vector based on word vector transformation, and the character shape vector of the word segmentation; wherein, the frequency domain vector of the word segmentation is obtained by processing the word vector of the word segmentation based on Fourier transform, and the character shape vector of the word segmentation is obtained based on the strokes of the word segmentation; and based on the fusion feature corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results.

[0017] In one optional implementation, the module determines that the dimension of the word vector used for word segmentation is n, and the frequency domain vector of the word segmentation has the same dimension as the word vector of the word segmentation; based on the word vector of the word segmentation and the Fourier transform, the real part and imaginary part of the k-th dimension in the n dimensions of the frequency domain vector of the word segmentation are calculated; the modulus corresponding to the real part and imaginary part of the k-th dimension is calculated to obtain the value corresponding to the k-th dimension in the frequency domain vector of the word segmentation, where n is an integer greater than or equal to k.

[0018] In an optional implementation, the determining module is further configured to construct a stroke vector space based on basic stroke types, wherein each dimension of the stroke vector space corresponds to a stroke type; for any character in the word segmentation, count the number of different stroke types in the character, and obtain the stroke vector corresponding to the character according to the stroke vector space; and normalize the stroke vector corresponding to each character in the word segmentation to obtain the character shape vector of the word segmentation.

[0019] In an optional implementation, the determining module is further configured to determine the context vector corresponding to the sentence in the text; and to determine the fusion feature corresponding to the segmented word based on the word vector of any segmented word in the segmentation result, the frequency domain vector based on the word vector transformation, the character shape vector of the segmented word, and their respective weights; the respective weights are determined based on the word vector, the frequency domain vector, the character shape vector, and the context vector.

[0020] In an optional implementation, the determining module is further configured to: determine a first attention score corresponding to the word segment based on the word vector and the context vector; determine a second attention score corresponding to the word segment based on the frequency domain vector and the context vector; determine a third attention score corresponding to the word segment based on the character shape vector and the context vector; process the first attention score, the second attention score, and the third attention score respectively using an exponential function and then sum them to obtain a comprehensive value; obtain the weight corresponding to the word vector of the word segment based on the word vector and the comprehensive value; obtain the weight corresponding to the frequency domain vector of the word segment based on the frequency domain vector of the word segment and the comprehensive value; and obtain the weight corresponding to the character shape vector of the word segment based on the character shape vector of the word segment and the comprehensive value.

[0021] In an optional implementation, the determining module is further configured to, for any one of the multiple segmentation results, calculate the similarity between the fusion feature corresponding to each segmentation in the segmentation result and the fusion feature corresponding to the next segmentation in the segmentation result; obtain the total score corresponding to the segmentation result based on the similarity between adjacent segments; and determine the target segmentation result corresponding to the sentence from the multiple segmentation results based on the total score corresponding to each segmentation result.

[0022] Thirdly, embodiments of the present invention provide a word segmentation device, the device comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect according to the obtained program when executing the computer program stored in the memory.

[0023] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when a computer reads and executes the computer program, the method described in the first aspect is performed.

[0024] Fifthly, embodiments of the present invention provide a computer program product that, when read and executed by a computer, causes the method described in the first aspect to be executed. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of the method corresponding to the existing word segmentation method; Figure 2 This is a schematic diagram of the method corresponding to the existing word segmentation method 2; Figure 3 A schematic diagram illustrating a word segmentation method provided in an embodiment of this application; Figure 4 A flowchart corresponding to a word segmentation method provided in an embodiment of this application; Figure 5 A schematic diagram illustrating a word segmentation method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of the word segmentation device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of a word segmentation device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0029] Before introducing the word segmentation method provided in the embodiments of this application, the background technology of the embodiments of this application will be introduced in detail for ease of understanding.

[0030] Existing word segmentation methods mainly include two types: segmentation method one and segmentation method two. The following sections will provide detailed descriptions of segmentation method one and segmentation method two respectively.

[0031] (1) Word segmentation method one

[0032] Figure 1 A diagram illustrating the corresponding method for existing word segmentation techniques, such as... Figure 1 As shown, the specific steps of segmentation method one include: segmenting the sentence to be segmented using both forward maximum matching and backward maximum matching to obtain segmentation results R1 and R2; determining whether the number of segments in segmentation result R1 and segmentation result R2 are the same; if the number of segments in segmentation result R1 and segmentation result R2 are the same, then comparing the number of characters in the longest segment in segmentation result R1 and segmentation result R2, and taking the segmentation result with the fewest characters as the final segmentation result R; if the segmentation result R1 is different from the previous one, then the segmentation result R2 is segmented using the previous one. If the number of words in segmentation result R1 is different from the number of words in segmentation result R2, then it is determined whether the longest word in segmentation result R1 and segmentation result R2 has the same number of characters. If the longest word in segmentation result R1 and segmentation result R2 has the same number of characters, then the segmentation result with the fewest words in segmentation result R1 and segmentation result R2 is taken as the final segmentation result R. If the longest word in segmentation result R1 and segmentation result R2 has different number of characters, then either segmentation result R1 or segmentation result R2 is selected as the final segmentation result R. Using segmentation method one, segmentation is performed simultaneously from two directions to obtain different segmentation results, and then a reliable result is selected according to preset rules. However, the above method is heavily dependent on the quality of the segmentation library, and the above segmentation method does not analyze the semantic information of the segmented words. Therefore, the accuracy in handling ambiguous words is low.

[0033] (2) Word segmentation method two

[0034] Figure 2 This is a schematic diagram of the method corresponding to the existing word segmentation method two, such as... Figure 2As shown, the specific steps of word segmentation method two include: using a word segmentation tool to segment the sentence to be segmented, obtaining the corresponding segmentation results, determining the word vector of each word in the segmentation results, calculating the word vector score of each word, and filtering out the word with the highest score; comparing the word with the highest score with each word in the segmentation results in turn, determining whether the word with the highest score is the next word of the current word, and if the word with the highest score is the next word of the current word, then combining the current word and the word with the highest score as a combined word, calculating the word frequency of the combined word, and determining the combined word... If the word frequency is greater than a threshold, the combined word segment is added to the word segmentation library; if it is less than the threshold, the current result is the final word segmentation result. If the highest-scoring word segment is the next word segment of the current word segment, the length of the adjacent words of the current word segment is calculated, as well as the information entropy and mutual information of the adjacent words. It is then determined whether the information entropy and mutual information are greater than the threshold. If they are greater than the threshold, the current word segment and adjacent words are combined into a combined word, and the word frequency of the combined word segment is calculated. Subsequent steps are the same as above and will not be repeated here. If they are less than the threshold, the current word segmentation result is taken as the final word segmentation result. Using word segmentation method two, after determining the highest-scoring word segment based on the word segmentation vector score, it is further determined whether the highest-scoring word segment needs to be combined and merged with adjacent words. This method usually has high requirements for text quality. When the text format is varied, the format is not uniform, and the relationship between the corresponding word vectors is not strong, it may lead to the inability to compare scores. In addition, word segmentation method two is only suitable for merging words with excessively fine granular segmentation, and not suitable for the reverse. Therefore, the accuracy of the word segmentation result is not ideal.

[0035] Based on this, this application provides a word segmentation method that extracts the word vector, frequency domain vector, and character shape vector of any word in each of the multiple word segmentation results to determine the fusion features corresponding to the word segmentation. Based on the fusion features corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results to improve the accuracy of the word segmentation results.

[0036] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0037] Figure 3 An exemplary schematic diagram of a word segmentation method provided in an embodiment of this application is shown, such as... Figure 3As shown, this application scenario includes at least one terminal device 301 and at least one server 302. The terminal device 301 can be a mobile phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal device 301 can have a client application for the word segmentation method installed. The client application can be software (such as a browser, instant messaging software, etc.), or a webpage, mini-program, etc. In this embodiment, the user terminal device 301 can use the aforementioned client application for the word segmentation method to upload the text to be segmented and transmit the text to the server 302 for analysis.

[0038] Furthermore, server 302 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In this embodiment, the server 302 may be equipped with a word segmentation device corresponding to the client, used to process word segmentation tasks initiated by the user on the client related to the word segmentation method on the terminal device 301. According to the method provided in this embodiment, the text is segmented into sentences for processing, and the target word segmentation result corresponding to each sentence is obtained, thereby improving the accuracy of the word segmentation result.

[0039] In some scenarios, terminal device 301 can access the network and communicate with server 302 through cellular mobile communication technology, which may include 5th generation mobile networks (5G) technology.

[0040] In other scenarios, terminal device 301 can access the network and communicate with server 302 via short-range wireless communication, which may include Wireless Fidelity (Wi-Fi) technology.

[0041] It should be noted that, Figure 3 The examples shown are merely illustrative; in reality, the number of terminal devices 301 and servers 302 is not limited and is not specifically limited in this embodiment.

[0042] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in combination.

[0043] Figure 4 This is a flowchart illustrating a word segmentation method provided in an embodiment of this application. This process can be executed by a word segmentation device, such as... Figure 3 The word segmentation device mounted on server 302 shown is used to improve the accuracy of word segmentation results, such as... Figure 4 As shown, the process includes the following steps: Step 401: For any sentence in the text, the word segmentation device segments the sentence to obtain multiple word segmentation results corresponding to the sentence.

[0044] For example, for any sentence in the text, the word segmentation device can use a word segmentation tool to segment the sentence, obtaining multiple word segmentation results corresponding to the sentence. Optionally, after obtaining multiple word segmentation results, they can be manually evaluated first, and custom word segments can be added to the word segmentation library. The word segmentation device then re-segments the words based on the updated word segmentation library, obtaining multiple word segmentation results. For example, assuming the sentence to be segmented is "Nanjing Yangtze River Bridge", the corresponding multiple word segmentation results may include word segmentation result A and word segmentation result B, where word segmentation result A is "Nanjing / Mayor / River / Bridge", and word segmentation result B is "Nanjing / Yangtze River / Bridge". Any sentence in the text can have multiple word segmentation results. This application embodiment uses two word segmentation results for example.

[0045] Step 402: For any word segmentation result among multiple word segmentation results, the word segmentation device determines the fusion feature corresponding to the word segmentation based on the word vector of any word segmentation result, the frequency domain vector based on word vector transformation, and the character shape vector of the word segmentation.

[0046] The following sections will provide a detailed introduction to word vectors, frequency domain vectors, and character shape vectors used in word segmentation: (1) Word vectors Word vectors are a method of representing discrete words by mapping them to a continuous high-dimensional vector space. Word vectors are typically multi-dimensional, but each dimension lacks a clear, independent semantic interpretation. Word vectors are not "interpretable" one-hot or handcrafted features, but rather abstract, distributed representations automatically learned from data through neural networks. In word segmentation, word vectors represent the "implicit features" of words, allowing the model to "learn" language rules from massive amounts of data, rather than hard-coded rules. Therefore, the word segmentation device in this embodiment extracts the word vectors of any segmented word from any segmentation result, and can use Word2Vec to extract the word vectors corresponding to any segmented word. Taking the word segmentation results A and B above as examples, and assuming the generated word vectors have a dimension of 10, for each word in segmentation result A, the word vector corresponding to Nanjing is: C(“Nanjing”) = [0.2, 0.8, 0.1, 0.5, 0.3, 0.6, 0.9, 0.1, 0.4, 0.7], the word vector corresponding to Mayor is: C(“Mayor”) = [0.5, 0.2, 0.9, 0.1, 0.8, 0.3, 0.2, 0.6, 0.7, 0.1], the word vector corresponding to River is: C(“River”) = [-0.3, 0.5, 0.2, -0.4, 0.1, 0.9, 0.3, 0.5, -0.2, 0.6], and the word vector corresponding to Bridge is: C(“Bridge”) = [0.4, 0.6, [0.1, 0.8, 0.2, 0.5, 0.7, 0.3, 0.3, 0.9]. For each word segment in the segmentation result B, the word vector corresponding to Nanjing City is: C(“Nanjing City”) = [0.3, 0.9, 0.2, 0.6, 0.4, 0.7, 0.8, 0.2, 0.5, 0.8], the word vector corresponding to Yangtze River is: C(“Yangtze River”) = [0.1, 0.7, 0.3, 0.2, 0.1, 0.9, 0.6, 0.4, 0.1, 0.8], and the word vector corresponding to Bridge is: C(“Bridge”) = [0.4, 0.6, 0.1, 0.8, 0.2, 0.5, 0.7, 0.3, 0.3, 0.9].

[0047] (2) Frequency domain vector

[0048] Since word vectors from word segmentation are often affected by training noise, the frequency domain allows for the application of filters (such as low-pass filtering to preserve low-frequency semantics and high-pass filtering to remove high-frequency noise) to generate cleaner representations and reduce error propagation in downstream tasks. Therefore, in this application, the frequency domain vector of any word segmentation result is obtained by processing the word vectors from the word segmentation based on Fourier transform. Specifically, the word vectors of the word segmentation have a dimension of n, and the frequency domain vectors of the word segmentation have the same dimension as the word vectors. Based on the word vectors and Fourier transform, the real and imaginary parts of the k-th dimension in the n-th dimension of the frequency domain vector of the word segmentation are calculated. The modulus corresponding to the real and imaginary parts of the k-th dimension is calculated to obtain the value corresponding to the k-th dimension in the frequency domain vector of the word segmentation, where n is an integer greater than or equal to k.

[0049] For example, taking a word vector dimension of 10 as an example, for any 10-dimensional word vector... Based on Fourier transform Decompose it into real and imaginary parts. The Fourier transform of a given expression satisfies the following formula: The real part is The imaginary part is The formulas for calculating the modulus corresponding to the real and imaginary parts satisfy: The word vector for Nanjing is [0.2, 0.8, 0.1, 0.5, 0.3, 0.6, 0.9, 0.1, 0.4, 0.7]. Corresponding , ; Corresponding , =-[0.2 sin(0°) + 0.8 sin(36°) + 0.1 sin(72°) + ... + 0.7 sin(324°)]≈ -0.74, ; And so on, calculate the modulus corresponding to k = 0 to k = 9, and obtain the frequency domain vector corresponding to Nanjing: [4.60, 1.25, 0.62, 0.86, 0.66, 0.80, 0.66, 0.86, 0.62, 1.25]. Similar to the calculation method of Nanjing, the frequency domain vectors of other word segments in word segmentation result A and word segmentation result B can be obtained. For the frequency domain vector of any word segment in word segmentation result A, the frequency domain vector corresponding to Nanjing: F("Nanjing") = [4.60, 1.25, 0.62, 0.86, 0.66, 0.80, 0.66, 0.86, 0.62, 1.25], the frequency domain vector corresponding to "Mayor": F("Mayor") = [4.40, 0.56, 1.35, 1.13, 0.87, 0.80, 0.87, 1.13, 1.35, 0.56], the frequency domain vector corresponding to "Jiang": F("Jiang") = [2.60, 1.26, 1.48, 1.38, 0.68, 1.00, 0.68, 1.38, 1.48, 1.26], the frequency domain vector corresponding to "Bridge": F("Bridge") = [4.80, 0.78, 1.02, 0.67, 0.33, 0.80, 0.33, 0.67, 1.02, 0.78]; For the frequency domain vector of any word segment in word segmentation result B, as follows: the frequency domain vector corresponding to "Nanjing City": F("Nanjing City") = [5.40, 1.26, 0.68, 0.61, 0.62, 0.60, 0.62, 0.61, 0.68, 1.26], the frequency domain vector corresponding to "Yangtze River": F("Yangtze River") = [4.20, 1.50, 0.61, 0.86, 0.81, 1.20, 0.81, 0.86, 0.61, 1.50], the frequency domain vector corresponding to "Bridge": F("Bridge") = [4.80, 0.78, 1.02, 0.67, 0.33, 0.80, 0.33, 0.67, 1.02, 0.78]. Determining the frequency domain vector corresponding to a word segment based on the word vector of word segmentation can make the vector more robust in dealing with ambiguous words. Introduce "spectral domain frequency transformation", regard the frequency sequence as a signal, perform frequency domain decomposition, and extract harmonic features (low-frequency harmonics represent stable collocations, and high-frequency represents sudden ambiguity). This is different from pure statistical methods and can better quantify the "frequency oscillation" of ambiguity. Chinese ambiguity often stems from semantic overlap (such as "Yangtze River Bridge", where "Jiang" can refer to a river or be abstract). The Word2Vec word vectors may be similar, but the Fourier transform can amplify the differences: low-frequency similarity indicates a shared theme (for example, both "Mayor" and "Yangtze River" have the sub-structure of "Chang", and the low-frequency peaks coincide), and high-frequency differences highlight ambiguity (for example, the high-frequency noise of "Mayor" is low and the administration is pure; the high-frequency of "Jiang Bridge" is strong and there are geographical fluctuations).That is to say, "Mayor" is frequent in political corpus, and "Jiang Daqiao" is frequent in the context of bridges.

[0050] (3)Glyph vector

[0051] The glyph vector of any word segment in any word segmentation result is obtained according to the strokes of the word segment. Specifically, a stroke vector space is constructed according to the basic stroke types, and each dimension in the stroke vector space corresponds to a stroke type; for any word in the word segment, count the number of different stroke types in the word, and obtain the stroke vector corresponding to the word according to the stroke vector space; normalize the stroke vectors corresponding to each word in the word segment to obtain the glyph vector of the word segment.

[0052] For example, create a stroke space with 10 dimensions, and each dimension corresponds to a basic stroke type. Dimension 0: horizontal, Dimension 1: vertical, Dimension 2: left-falling stroke, Dimension 3: right-falling stroke, Dimension 4: dot, Dimension 5: rising stroke, Dimension 6: fold, Dimension 7: hook, Dimension 8: bend, Dimension 9: others. Take Nanjing as an example. The stroke vector corresponding to "南" in Nanjing: V("南") = [horizontal: 2, vertical: 2, left-falling stroke: 1, right-falling stroke: 0, dot: 2, rising stroke: 0, fold: 2, hook: 0, bend: 0, others: 0]. The stroke vector corresponding to "京" in Nanjing: V("京") = [horizontal: 2, vertical: 1, left-falling stroke: 1, right-falling stroke: 1, dot: 3, rising stroke: 0, fold: 0, hook: 0, bend: 0, others: 0]. The stroke vector corresponding to "市" in Nanjing: V("市") = [horizontal: 1, vertical: 1, left-falling stroke: 0, right-falling stroke: 0, dot: 1, rising stroke: 0, fold: 1, hook: 1, bend: 0, others: 0]. Add the stroke vectors of "南", "京", and "市" to get the stroke vector of "南京市": V("南京市") = V("南") + V("京") + V("市") = [2 + 2 + 1, 2 + 1 + 1, 1 + 1 + 0, 0 + 1 + 0, 2 + 3 + 1, 0 + 0 + 0, 2 + 0 + 1, 0 + 0 + 1, 0 + 0 + 0, 0 + 0 + 0] = [5, 4, 2, 1, 6, 0, 3, 1, 0, 0]. By calculating the norm of the stroke vector of Nanjing and performing normalization, the glyph vector corresponding to Nanjing is obtained. Among them, the norm of Nanjing: The final glyph vector corresponding to Nanjing City is: V("Nanjing City") = [5 / 9.59, 4 / 9.59, 2 / 9.59, 1 / 9.59, 6 / 9.59, 0 / 9.59, 3 / 9.59, 1 / 9.59, 0 / 9.59, 0 / 9.59] = [0.521, 0.417, 0.209, 0.104, 0.626, 0.000, 0.313, 0.104, 0.000, 0.000]. Following the above calculation rules, the glyph vectors of any word segmented in segmentation results A and B are obtained. The glyph vector for any word segmented in segmentation result A is as follows: The glyph vector corresponding to Nanjing is: V(“Nanjing”) = [0.47, 0.36, 0.24, 0.12, [0.59, 0.00, 0.24, 0.00, 0.00, 0.00], the glyph vector corresponding to Mayor: V(“Mayor”)=[0.55, 0.55, 0.28, 0.28, 0.28, 0.00, 0.28, 0.28, 0.00, 0.00], the glyph vector corresponding to River: V(“River”)=[0.41, 0.41, 0.00, 0.00, 0.41, 0.41, 0.41, 0.41, 0.00, 0.00], the glyph vector corresponding to Bridge: V(“Bridge”)=[0.35, 0.52, 0.52, 0.35, 0.00, 0.00, 0.17, 0.00, 0.00, 0.17]. The glyph vectors for any word in the word segmentation result B are as follows: The glyph vector for Nanjing City is: V(“Nanjing City”) = [0.52, 0.42, 0.21, 0.10, 0.63, 0.00, 0.31, 0.10, 0.00, 0.00], the glyph vector for Yangtze River is: V(“Yangtze River”) = [0.53, 0.53, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.00, 0.00], and the glyph vector for Bridge is: V(“Bridge”) = [0.35, 0.52, 0.52, 0.35, 0.00, 0.00, 0.17, 0.00, 0.00, 0.17]. Character shape vectors based on basic stroke types primarily serve as visual feature enhancers, injecting "writing geometry" information into Chinese characters to help resolve ambiguity and boundary issues in Chinese word segmentation. Character shape vectors mainly encode visual structure into a low-dimensional numerical representation by capturing the distribution and density of stroke styles (such as horizontal, vertical, and left-falling strokes). Furthermore, character shape vectors compensate for the shortcomings of word vectors (semantic) and frequency domain vectors, providing complementary information at the visual-structural level.

[0053] Further, the specific steps by which the word segmentation device determines the fusion features corresponding to any segmented word based on the word vector, the frequency domain vector based on word vector transformation, and the character shape vector of the segmented word in the word segmentation result are as follows: determine the context vector corresponding to the sentence in the text; determine the fusion features corresponding to the segmented word based on the word vector, the frequency domain vector based on word vector transformation, the character shape vector of the segmented word, and their respective weights; the respective weights are determined based on the word vector, the frequency domain vector, the character shape vector, and the context vector. Among them, determining the context vector corresponding to the sentence in the text is done by inputting the text into a preset model, outputting the word vector corresponding to the text through the model, and generating the corresponding context vector based on the word vector corresponding to the text, for example, context vector u = [0.10, 0.50, 0.20, -0.30, 0.60, 0.40, -0.10, 0.20, 0.50, -0.20].

[0054] For example, the corresponding weights are determined based on word vectors, frequency domain vectors, and context vectors. Specifically, this includes: determining a first attention score for each word segment based on the word vectors and context vectors; determining a second attention score for each word segment based on the frequency domain vectors and context vectors; determining a third attention score for each word segment based on the character vectors and context vectors; processing the first, second, and third attention scores separately using an exponential function and then summing them to obtain a comprehensive value; obtaining the weights corresponding to the word vectors based on the word vectors and the comprehensive value; obtaining the weights corresponding to the frequency domain vectors based on the frequency domain vectors and the comprehensive value; and obtaining the weights corresponding to the character vectors based on the character vectors and the comprehensive value.

[0055] For example, taking "mayor" as an example, the word vector corresponding to "mayor" is multiplied by the context vector to obtain the first attention score for "mayor": score_C("mayor") = C("mayor") · u = 0.5 0.1 + 0.2 0.5 +0.9 0.2 + 0.1 (-0.3) + 0.8 0.6 + 0.3 0.4 + 0.2 (-0.1) + 0.6 0.2 + 0.7 0.5 +0.1 (-0.2) = 0.05 + 0.10 + 0.18 - 0.03 + 0.48 + 0.12 - 0.02 + 0.12 + 0.35 -0.02 = 1.33; Multiplying the frequency domain vector and context vector corresponding to the mayor, we obtain the second attention score for the mayor: score_F(“Mayor”) = F(“Mayor”)· u = 4.40 0.1 + 0.56 0.5 + 1.35 0.2 + 1.13 (-0.3) + 0.87 0.6 + 1.80 0.4 + 0.87 (-0.1) + 1.13 0.2 + 1.35 0.5 + 0.56 (-0.2) = 0.44 + 0.28 + 0.27 - 0.34 + 0.52 + 0.72 - 0.09 + 0.23 + 0.68 - 0.11 = 2.60; Multiplying the glyph vector corresponding to the mayor with the context vector yields the third attention score for the mayor: score_V(“Mayor”) = V(“Mayor”)· u = 0.55 0.1 + 0.55 0.5 + 0.28 0.2 + 0.28 (-0.3) + 0.28 0.6 + 0.0 0.4 + 0.28 (-0.1) + 0.28 0.2 + 0.0 0.5 + 0.0 (-0.2) = 0.055 + 0.275 + 0.056 - 0.084 + 0.168 - 0.028 + 0.056 = 0.498; The first, second, and third attention scores are processed using exponential functions, resulting in exp(score_C(“Mayor”)) = exp(1.33) ≈ 3.78, exp(score_F(“Mayor”)) = exp(2.60) ≈ 13.46, and exp(score_V(“Mayor”)) = exp(0.498) ≈ 1.65. The sum of exp(score_C(“Mayor”)), exp(score_F(“Mayor”)), and exp(score_F(“Mayor”)) yields the comprehensive value: Sum_exp = 3.78 + 13.46 + 1.65 = 18.89; Based on the mayor's word vector and comprehensive value, the weight corresponding to the mayor's word vector is obtained: α_C (“Mayor”) = exp(score_C(“Mayor”)) / Sum_exp = 3.78 / 18.89 = 0.200; Based on the mayor's frequency domain vector and comprehensive value, the weight corresponding to the mayor's frequency domain vector is obtained: α_F(“Mayor”) = exp(score_F(“Mayor”)) / Sum_exp = 13.46 / 18.89 = 0.713; Based on the mayor's glyph vector and comprehensive value, the weight corresponding to the mayor's glyph vector is obtained: α_V (“Mayor”) = exp(score_V(“Mayor”)) / Sum_exp = 1.65 / 18.89 =0.087; At this point, based on the mayor's word vector, frequency domain vector, character shape vector, and their respective weights, the fusion feature corresponding to the mayor is obtained: fused(“Mayor”)=α_C(“Mayor”) C(“Mayor”) + α_F(“Mayor”) F(“Mayor”) + α_V(“Mayor”) V(“Mayor”) = [3.285, 0.487, 1.167, 0.767, 0.793, 1.343, 0.748, 0.907, 1.100, 0.419]. Using the same calculation method above, calculate the fusion features of any token in the tokenization result A and the tokenization result B respectively. Among them, the fusion features of each token in the tokenization result A are as follows: The fusion feature corresponding to “Nanjing”: fused(“Nanjing”) = [3.670, 0.825, 0.612, 0.635, 0.562, 1.077, 0.669, 0.713, 0.575, 1.054]; The fusion feature corresponding to “Mayor”: fused(“Mayor”) = [3.285, 0.487, 1.167, 0.767, 0.793, 1.343, 0.748, 0.907, 1.100, 0.419]; The fusion feature corresponding to “Jiang”: fused(“Jiang”) = [1.979, 1.114, 1.059, 0.852, 0.596, 1.551, 0.591, 0.992, 0.978, 0.910]; The fusion feature corresponding to “Bridge”: fused(“Bridge”) = [2.193, 0.603, 0.505, 0.224, 0.218, 0.838, 0.420, 0.439, 0.582, 0.835]. The fusion features of each token in the tokenization result B are as follows: fused(“Nanjing City”) = [2.784, 1.025, 0.472, 0.198, 0.509, 1.071, 0.627, 0.457, 0.509, 0.880]; The fusion feature corresponding to “Yangtze River”: fused(“Yangtze River”) = [3.541, 1.294, 0.640, 0.697, 0.751, 1.624, 0.712, 0.769, 0.613, 1.309]; The fusion feature corresponding to “Bridge”: fused(“Bridge”) = [2.193, 0.603, 0.505, 0.224, 0.218, 0.838, 0.420, 0.439, 0.582, 0.835].

[0056] Step 403, the tokenization device determines the target tokenization result corresponding to the sentence from multiple tokenization results according to the fusion features corresponding to each token in each tokenization result among the multiple tokenization results.

[0057] For example, for any one of the multiple word segmentation results, the word segmentation device calculates the similarity between the fusion feature corresponding to each word in the segmentation result and the fusion feature corresponding to the next word in the segmentation result; based on the similarity between adjacent words, it obtains the total score corresponding to the word segmentation result; based on the total score corresponding to each word segmentation result in the multiple word segmentation results, it determines the target word segmentation result corresponding to the sentence from the multiple word segmentation results. Cosine similarity can be used to calculate the similarity between the fusion feature corresponding to each word and the fusion feature corresponding to the next word in the segmentation result, and this is not limited here.

[0058] For example, for word segmentation result A, we calculate the similarity between the fused features corresponding to "Nanjing" and "Mayor", the similarity between the fused features corresponding to "Mayor" and "Jiang", and the similarity between the fused features corresponding to "Jiang" and "Daqiao". For example, cos_sim(fused("Nanjing"),fused("Mayor")) = 0.811; cos_sim(fused("Mayor"), fused("Jiang")) = 0.441; cos_sim(fused("Jiang"), fused("Daqiao")) = 0.658. Based on the similarity between adjacent words, the total score corresponding to the word segmentation result is obtained. The total score corresponding to word segmentation result A is: Score(A) = (cos_sim(fused("Nanjing"), fused("Mayor")) + cos_sim(fused("Mayor"), fused("Jiang")) + cos_sim(fused("Jiang"), fused("Bridge"))) / 3 = (0.811+0.441+0.658) / 3 = 0.637. For word segmentation result B, the similarity between the fusion feature corresponding to "Nanjing City" and the fusion feature corresponding to "Yangtze River", and the similarity between the fusion feature corresponding to "Yangtze River" and the fusion feature corresponding to "Bridge" are calculated respectively. For example, cos_sim(fused("Nanjing City"), fused("Yangtze River")) = 0.953; cos_sim(fused("Yangtze River"), fused("Bridge")) = 0.852. Based on the similarity between adjacent words, the total score corresponding to the word segmentation result is obtained. The total score corresponding to word segmentation result B is: Score(B) = (cos_sim(fused("Nanjing City"), fused("Yangtze River")) + cos_sim(fused("Yangtze River"), fused("Bridge"))) / 2 = (0.953+0.852) / 2 = 0.903. By comparing the total score corresponding to word segmentation result A with the total score corresponding to word segmentation result B, the target word segmentation result corresponding to the sentence is determined from word segmentation result A and word segmentation result B. Here, since Score(A) < Score(B), word segmentation result B is better than word segmentation result A. For the input sentence "Nanjing City Yangtze River Bridge", the target word segmentation result is: word segmentation result B "Nanjing City / Yangtze River / Bridge".

[0059] Using the above methods, such as Figure 5As shown, by extracting the word vector, frequency domain vector, and character shape vector of any word in each of the multiple word segmentation results, the fusion features corresponding to the word segmentation are determined. Based on the fusion features corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results to improve the accuracy of the word segmentation results.

[0060] Based on the same technical concept, embodiments of this application provide a word segmentation device 6000. Figure 6 This is a schematic diagram of the structure of the word segmentation device provided in the embodiments of this application, such as... Figure 6 As shown, the device 6000 includes: The segmentation module 601 is used to segment any sentence in the text to obtain multiple word segmentation results corresponding to the sentence; The determining module 602 is used to determine the fusion feature corresponding to any one of the multiple word segmentation results, based on the word vector, the frequency domain vector based on word vector transformation, and the character shape vector of the word segmentation; wherein, the frequency domain vector of the word segmentation is obtained by processing the word vector of the word segmentation based on Fourier transform, and the character shape vector of the word segmentation is obtained based on the strokes of the word segmentation; and to determine the target word segmentation result corresponding to the sentence from the multiple word segmentation results based on the fusion feature corresponding to each word segmentation result in each of the multiple word segmentation results.

[0061] In one optional implementation, the determining module 602 specifically uses n dimensions for the word vectors of the word segmentation, and the frequency domain vector of the word segmentation has the same dimension as the word vectors of the word segmentation; based on the word vectors of the word segmentation and the Fourier transform, the real part and imaginary part of the k-th dimension in the n dimensions of the frequency domain vector of the word segmentation are calculated; the modulus corresponding to the real part and imaginary part of the k-th dimension is calculated to obtain the value corresponding to the k-th dimension in the frequency domain vector of the word segmentation, where n is an integer greater than or equal to k.

[0062] In an optional implementation, the determining module 602 is further configured to construct a stroke vector space based on basic stroke types, wherein each dimension of the stroke vector space corresponds to a stroke type; for any character in the word segmentation, count the number of different stroke types in the character, and obtain the stroke vector corresponding to the character according to the stroke vector space; and normalize the stroke vector corresponding to each character in the word segmentation to obtain the character shape vector of the word segmentation.

[0063] In an optional implementation, the determining module 602 is further configured to determine the context vector corresponding to the sentence in the text; and to determine the fusion feature corresponding to the segmented word based on the word vector of any segmented word in the segmentation result, the frequency domain vector based on the word vector transformation, the character shape vector of the segmented word, and their respective weights; the respective weights are determined based on the word vector, the frequency domain vector, the character shape vector, and the context vector.

[0064] In an optional implementation, the determining module 602 is further configured to: determine a first attention score corresponding to the word segment based on the word vector and the context vector; determine a second attention score corresponding to the word segment based on the frequency domain vector and the context vector; determine a third attention score corresponding to the word segment based on the character shape vector and the context vector; process the first attention score, the second attention score, and the third attention score respectively using an exponential function and then sum them to obtain a comprehensive value; obtain the weight corresponding to the word vector of the word segment based on the word vector and the comprehensive value; obtain the weight corresponding to the frequency domain vector of the word segment based on the frequency domain vector of the word segment and the comprehensive value; and obtain the weight corresponding to the character shape vector of the word segment based on the character shape vector of the word segment and the comprehensive value.

[0065] In an optional implementation, the determining module 602 is further configured to, for any one of the multiple segmentation results, calculate the similarity between the fusion feature corresponding to each segmentation in the segmentation result and the fusion feature corresponding to the next segmentation in the segmentation result; obtain the total score corresponding to the segmentation result based on the similarity between adjacent segments; and determine the target segmentation result corresponding to the sentence from the multiple segmentation results based on the total score corresponding to each segmentation result.

[0066] Based on the same technological concept Figure 7 This is a schematic diagram of the structure of a word segmentation device 7000 provided in an embodiment of this application, as shown below. Figure 7 As shown, the device 7000 includes at least one processor 701 and a memory 702 connected to the at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 Taking the connection between processor 701 and memory 702 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc. In this embodiment of the invention, memory 702 stores instructions that can be executed by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can implement the steps of the above-mentioned word segmentation method.

[0067] The processor 701 is the control center of the computer device, capable of connecting various parts of the computer device via various interfaces and lines. It performs resource configuration by running or executing instructions stored in the memory 702 and accessing data stored in the memory 702. Optionally, the processor 701 may include one or more processing units. The processor 701 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 701. In some embodiments, the processor 701 and the memory 702 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0068] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0069] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 702 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0070] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when run on a computer, causes the computer to execute any of the word segmentation methods discussed above. Since the principle by which the above-described computer-readable storage medium solves the problem is similar to that of the word segmentation methods, the implementation of the above-described computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0071] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the word segmentation methods discussed above. Since the principle by which the above computer program product solves the problem is similar to that of the word segmentation method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A word segmentation method, characterized in that, The method includes: For any sentence in the text, the sentence is segmented to obtain multiple word segmentation results corresponding to the sentence; For any one of the multiple word segmentation results, the fusion feature corresponding to the word segmentation is determined based on the word vector of any one of the word segmentation results, the frequency domain vector based on word vector transformation, and the character shape vector of the word segmentation; wherein, the frequency domain vector of the word segmentation is obtained by processing the word vector of the word segmentation based on Fourier transform, and the character shape vector of the word segmentation is obtained based on the strokes of the word segmentation; Based on the fusion features corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results.

2. The method according to claim 1, characterized in that, The frequency domain vector of any word in the word segmentation result is obtained by processing the word vector of the segmented word based on Fourier transform, including: The word vector of the segmented word has a dimension of n, and the frequency domain vector of the segmented word has the same dimension as the word vector of the segmented word. Based on the word vectors and Fourier transforms of the word segmentation, calculate the real and imaginary parts of the k-th dimension out of n dimensions in the frequency domain vector of the word segmentation; Calculate the modulus corresponding to the real and imaginary parts of the k-th dimension to obtain the value corresponding to the k-th dimension in the frequency domain vector of the word segmentation, where n is an integer greater than or equal to k.

3. The method according to claim 1, characterized in that, The glyph vector of any word in the word segmentation result is obtained based on the strokes of the word segmentation, including: A stroke vector space is constructed based on the basic stroke types, and each dimension of the stroke vector space corresponds to a stroke type. For any character in the word segmentation, count the number of different stroke types in the character, and obtain the stroke vector corresponding to the character according to the stroke vector space; The stroke vector corresponding to each character in the word segmentation is normalized to obtain the character shape vector of the word segmentation.

4. The method according to claim 1, characterized in that, The step of determining the fusion feature corresponding to the segmented word based on the word vector of any segmented word in the segmentation result, the frequency domain vector based on word vector transformation, and the character shape vector of the segmented word includes: Determine the context vector corresponding to the sentence in the text; Based on the word vector, frequency domain vector based on word vector transformation, character shape vector of the word segment, and their respective weights, the fusion feature corresponding to the word segment is determined; the respective weights are determined based on the word vector, the frequency domain vector, the character shape vector, and the context vector.

5. The method according to claim 4, characterized in that, The respective weights are determined based on the word vector, the frequency domain vector, the glyph vector, and the context vector, including: Based on the word vectors and context vectors of the segmented words, determine the first attention score corresponding to the segmented words; The second attention score corresponding to the word segmentation is determined based on the frequency domain vector and the context vector. The third attention score corresponding to the word segmentation is determined based on the glyph vector and the context vector. The first attention score, the second attention score, and the third attention score are processed separately using an exponential function and then summed to obtain a comprehensive value. The weights corresponding to the word vectors of the segmented words are obtained based on the word vectors and the comprehensive value; the weights corresponding to the frequency domain vectors of the segmented words are obtained based on the frequency domain vectors of the segmented words and the comprehensive value; and the weights corresponding to the character shape vectors of the segmented words are obtained based on the character shape vectors of the segmented words and the comprehensive value.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the target word segmentation result corresponding to the sentence from the multiple word segmentation results based on the fusion features corresponding to each word in each word segmentation result includes: For any one of the multiple word segmentation results, calculate the similarity between the fusion feature corresponding to each word in the word segmentation result and the fusion feature corresponding to the next word in the word segmentation result; based on the similarity between each adjacent word segmentation result, obtain the total score corresponding to the word segmentation result; Based on the total score corresponding to each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results.

7. A word segmentation device, characterized in that, The device includes: The segmentation module is used to segment any sentence in the text to obtain multiple word segmentation results corresponding to the sentence; The determination module is used to determine the fusion feature corresponding to any one of the multiple word segmentation results, based on the word vector, the frequency domain vector based on word vector transformation, and the character shape vector of the word segmentation; wherein, the frequency domain vector of the word segmentation is obtained by processing the word vector of the word segmentation based on Fourier transform, and the character shape vector of the word segmentation is obtained based on the strokes of the word segmentation; and based on the fusion feature corresponding to each word in each of the multiple word segmentation results, the target word segmentation result corresponding to the sentence is determined from the multiple word segmentation results.

8. A word segmentation device, characterized in that, The device includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-6 in accordance with the obtained program instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the method as described in any one of claims 1-6 to be performed.

10. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes any one of claims 1-6 to be executed.

Images