Word segmentation method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a word segmentation method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining to-be-segmented word data; performing preliminary word segmentation on the to-be-segmented data to obtain an initial word segmentation result; performing re-word segmentation on the data to be subjected to word segmentation by using a user-defined word segmentation device to obtain a re-word segmentation result, the user-defined word segmentation device being obtained through deployment of a trained large model, the trained large model being obtained based on professional corpus training of a target field, and the target field corresponding to a field to which the data to be subjected to word segmentation belongs; and combining the initial word segmentation result and the re-word segmentation result to obtain a target word segmentation result. The problem that the word segmentation result is not accurate enough in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a word segmentation method, device, electronic device and computer-readable storage medium. Background Art

[0002] ElasticSearch (ES) is a widely used search engine, and its word segmenter is crucial for text processing. Currently, the Chinese word segmenter commonly used in ES search engines is mainly IK word segmenter, which is a word segmentation tool for Chinese text processing. It usually performs word segmentation based on rules or dictionaries. These methods perform well when performing word segmentation on regular texts, but cannot consider contextual information. When faced with word segmentation tasks with professional terms or complex language structures, the existing technology is not flexible enough, resulting in inaccurate word segmentation results. Summary of the invention

[0003] In view of this, the present application provides a word segmentation method, device, electronic device and computer-readable storage medium to solve the problem of inaccurate word segmentation results in the prior art.

[0004] In a first aspect of the present application, a word segmentation method is provided, including: obtaining data to be segmented; performing preliminary word segmentation on the data to be segmented to obtain an initial word segmentation result; using a custom word segmenter to re-segment the data to be segmented to obtain a re-segmentation result, wherein the custom word segmenter is obtained by deploying a trained large model, and the trained large model is trained based on professional corpus in a target field, and the target field corresponds to the field to which the data to be segmented belongs; combining the initial word segmentation result and the re-segmentation result to obtain a target word segmentation result.

[0005] According to a second aspect of the present application, a word segmentation device is provided, including: an acquisition module configured to acquire data to be segmented; a preliminary word segmentation module configured to perform preliminary word segmentation on the data to be segmented to obtain an initial word segmentation result; a re-word segmentation module configured to use a custom word segmenter to re-segment the data to be segmented to obtain a re-word segmentation result, wherein the custom word segmenter is obtained by deploying a trained large model, and the trained large model is obtained by training based on professional corpus in a target field, and the target field corresponds to the field to which the data to be segmented belongs; and a combination module configured to combine the initial word segmentation result and the re-word segmentation result to obtain a target word segmentation result.

[0006] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0009] By obtaining the data to be segmented, the data to be segmented is preliminarily segmented to obtain the initial segmentation result, and a trained large language model is called to deploy a custom segmenter, and the data to be segmented is re-segmented by the custom segmenter to obtain the re-segmentation result, the initial segmentation result is combined with the re-segmentation result to obtain the target segmentation result, and the result is saved. Among them, when performing preliminary segmentation on the data to be segmented, a traditional segmenter can be used for segmentation. The preliminary segmentation can quickly process the data to be segmented, and the large model can perform secondary optimization in complex situations, taking into account both efficiency and accuracy. In this way, an effective multi-level segmentation strategy can be formed by effectively combining traditional segmentation tools and trained large models, which not only improves the accuracy and efficiency of segmentation, but also realizes the automation and flexibility of the process, improves the intelligence level of the system, and enhances the accuracy of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 It is a flowchart of a word segmentation method provided in an embodiment of the present application;

[0012] Figure 2 It is a flowchart of another word segmentation method provided in an embodiment of the present application;

[0013] Figure 3 is a structural schematic diagram of a word segmentation device provided in an embodiment of the present application;

[0014] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0016] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0017] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0018] ES is a powerful search engine. Before executing a search task, a word segmenter can be used to convert text data into terms that are easy to index and retrieve. However, in the prior art, word segmentation tasks often rely on preset rules and dictionaries, which may achieve good results when processing regular texts, but when faced with professional terms, complex language structures or industry-specific expressions, their accuracy and flexibility may be limited. Traditional word segmenters may not understand the subtle differences between words, may not fully understand the contextual connections, may ignore abbreviations and dialects that are not in the dictionary, and thus cannot accurately segment the text data in the system, resulting in inaccurate index construction, and cannot ensure the accuracy of search results. Therefore, the present application designs a word segmentation method based on a large model, performs multiple word segmentation checks on text data, and obtains accurate word segmentation results to avoid the problem of inaccurate word segmentation results of traditional word segmenters.

[0019] The word segmentation task in a search engine is used as an example embodiment of the present application. Of course, the word segmentation method of the present application is not only applicable to search engines, but can also be applied to other reasonable word segmentation tasks, such as document analysis, etc., which are not limited here.

[0020] A word segmentation method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 Schematic diagram of a word segmentation method provided in an embodiment of the present application. Figure 1 As shown, the word segmentation method includes:

[0022] S101, obtaining data to be segmented.

[0023] Specifically, the data to be segmented may represent data input to the segmenter for segmentation. The data to be segmented may be text data to be processed that the system obtains from user input or stored data sources. The data to be segmented may be paragraphs, sentences, or other text formats in natural language, and may include text data in various fields, such as professional knowledge in the real estate field, professional terms in the legal industry, etc. In the search engine, the data to be segmented may be query statements input by users and data to be retrieved in the system, which is not limited here.

[0024] S102, performing preliminary word segmentation on the data to be segmented to obtain an initial word segmentation result.

[0025] Specifically, a traditional word segmentation tool can be used to perform preliminary word segmentation on the word segmentation data to generate an initial word segmentation result, wherein the traditional word segmentation tool can be an IK word segmenter, which can be understood as a tool based on dictionaries and rules, and is suitable for general Chinese word segmentation. In this way, an initial word segmentation result can be generated by a traditional word segmentation tool, providing the system with a benchmark word segmentation result for subsequent steps to judge and re-segment. For example, when the initial word segmentation result is inaccurate, the initial word segmentation result is used as a priori result of re-segmentation, that is, the process of re-segmentation avoids the wrong initial word segmentation result to speed up the generation of the re-segmentation result and improve the accuracy of the re-segmentation result. When the initial word segmentation result is accurate, auxiliary re-segmentation generates a re-segmentation result that is the same or similar to the correct initial word segmentation result, so as to verify the accuracy of the re-segmentation result in multiple ways. Of course, if the initial word segmentation result is accurate, it is also possible not to re-segment, and directly execute the next task of the system, such as directly establishing an index based on the initial word segmentation result so that the system performs a retrieval task to save system resources, which is not limited here.

[0026] S103, using a custom word segmenter to re-segment the data to be segmented to obtain a re-segmentation result. The custom word segmenter is obtained by deploying a trained large model. The trained large model is trained based on professional corpus in the target field. The target field corresponds to the field to which the data to be segmented belongs.

[0027] Specifically, after the initial word segmentation, the system can use a custom word segmenter to re-segment the data to be segmented. The custom word segmenter can be obtained by fine-tuning the large model. The large model can be trained based on professional corpus in the target field (such as data in specific fields such as real estate and finance), so that it can understand complex professional terms and field-specific language expressions.

[0028] Furthermore, the custom tokenizer can combine the professional corpus, semantic context and complex language structure of the target field to segment the data to produce re-segmentation results that are more in line with the target field. For example, a proper noun such as "real estate" may be incorrectly split in the initial tokenization, but can be accurately identified and retained as a complete vocabulary in the custom tokenizer of the large model.

[0029] In this way, the contextual understanding and professional domain knowledge of the large model can be used to more accurately process the complex vocabulary and semantic structures that were not correctly processed in the initial word segmentation results, so as to increase the accuracy of the system's word segmentation results.

[0030] S104, combining the initial word segmentation result and the re-word segmentation result to obtain the target word segmentation result.

[0031] Specifically, after obtaining the re-segmentation results, they can be combined with the initial segmentation results to jointly generate the target segmentation results. By combining the preliminary segmentation results and the re-segmentation results, the system can make full use of the high efficiency of traditional segmenters and the accuracy of large model segmenters to maximize the accuracy and processing speed of segmentation. This not only ensures the accuracy of the segmentation results, but also avoids relying entirely on a single segmentation strategy, which leads to segmentation deviations, thereby improving the quality and robustness of the segmentation results.

[0032] Furthermore, when combining the initial word segmentation results and the re-word segmentation results, the target word segmentation results can be obtained by retaining the same terms, replacing wrong word segmentations, and combining weights. Among them, retaining the same terms can be expressed as: if there are the same sub-segmentation results in the initial word segmentation results and the re-word segmentation results, indicating that they are accurate, then these word segmentation results are retained to obtain the target word segmentation results. Replacing wrong word segmentations can be expressed as: if some sub-segmentation results in the initial word segmentation results are judged to be inaccurate, then these wrong sub-segmentation results are replaced with the corresponding parts in the re-word segmentation results to obtain the target word segmentation results. Weight combination can be expressed as: assigning different weights to the initial word segmentation results and the re-word segmentation results, and combining them based on selecting the sub-segmentation results with higher weights to obtain the target word segmentation results. Of course, no restrictions are made here.

[0033] According to the technical solution provided in the above embodiment, the data to be segmented is segmented through preliminary segmentation and customized segmentation, thereby ensuring the accuracy of the target segmentation results. By combining traditional rules and large models, suitable segmentators can be dynamically selected for different text types, greatly improving the adaptability of the system. Through the contextual understanding ability of the large model, more precise segmentation can be obtained based on the context to obtain accurate segmentation results.

[0034] In some embodiments, the initial word segmentation result and the re-word segmentation result are combined to obtain a target word segmentation result, including: judging the confidence of the initial word segmentation result based on a first preset rule to obtain a first judgment result; based on the first judgment result, processing the initial word segmentation result and the re-word segmentation result to obtain the target word segmentation result.

[0035] Specifically, after completing the preliminary word segmentation to obtain the initial word segmentation result, the accuracy of the initial word segmentation result can also be judged to obtain a first judgment result. The first judgment result can be obtained by calculating the confidence of the initial word segmentation result through a first preset rule. After obtaining the first judgment result, it can be known whether the initial word segmentation result is accurate, and then the combination of the initial word segmentation result and the re-word segmentation result can be adaptively selected to obtain the target word segmentation result.

[0036] As some examples, the confidence can be calculated through the dictionary matching degree stored in the system, that is, if the sub-segmentation result in the initial word segmentation result can be highly matched with the target domain dictionary, the confidence is high, otherwise it is low; natural language understanding technology can also be used to judge the confidence or rationality of the initial word segmentation result through the contextual relationship of the sub-segmentation result. If the confidence is higher, the accuracy of the initial word segmentation result is higher. The first judgment result can be determined based on the calculated confidence. Similarly, based on the adaptability of the first judgment result, it can be judged which sub-segmentation results and re-segmentation results in the initial word segmentation result are combined to obtain a better result. Moreover, when the confidence of the initial word segmentation result is high (for example, the confidence exceeds 0.9), its first judgment result can indicate that the initial word segmentation result is accurate, and the next step of re-segmentation can be performed directly based on the initial word segmentation result without performing the next step of re-segmentation. The next task in the system, such as retrieval, is executed directly based on the initial word segmentation result. Of course, there is no restriction here.

[0037] It should be noted that the system can set a confidence threshold (for example, 0.8). When the confidence exceeds the confidence threshold, it indicates that the credibility of the initial word segmentation result is high. At this time, the first judgment result represents that the initial word segmentation result is accurate. It should be noted that in addition to calculating the confidence of the overall initial word segmentation result, the sub-word segmentation confidence of the sub-word segmentation result in the initial word segmentation result can also be calculated. Then, when the confidence of the overall initial word segmentation result exceeds the confidence threshold, the system can directly extract the sub-word segmentation result whose sub-word segmentation confidence is higher than another confidence threshold and compare it with the re-word segmentation result. If the two have the same word segmentation results, these terms can be extracted and combined as the target word segmentation result. For example, assuming that both the preliminary word segmentation result and the re-word segmentation result have "real estate" as a complete term, then the term can be directly retained as the target word segmentation result.

[0038] If the confidence of the initial word segmentation result is low (for example, lower than 0.8), it indicates that there may be errors in the initial word segmentation result, that is, the first judgment result can indicate that the initial word segmentation result is not accurate enough. Then, the corresponding new word segmentation result can be extracted from the re-word segmentation result to replace or correct the erroneous word segmentation result in the initial word segmentation result (wherein, the erroneous word segmentation in the initial word segmentation result can be expressed as a sub-word segmentation result whose confidence is lower than another confidence threshold) to obtain an accurate target word segmentation result. In this way, the initial word segmentation result and the re-word segmentation result can be combined to generate the final target word segmentation result.

[0039] Furthermore, after obtaining the target word segmentation results, they can be used for subsequent text processing tasks, such as text indexing, query optimization, etc.

[0040] According to the technical solution provided in the above embodiment, by making a confidence judgment on the initial word segmentation result, the initial word segmentation result and the re-word segmentation result are adaptively combined to obtain the target word segmentation result. This can reduce the need for human intervention and automatically improve the word segmentation effect to help the system generate more accurate target word segmentation results.

[0041] In other embodiments, the initial word segmentation result includes at least two sub-word segmentation results, and the confidence of the initial word segmentation result is judged based on a first preset rule to obtain a first judgment result, including: obtaining sub-word segmentation results in the initial word segmentation result whose character length exceeds the preset character length; determining the confidence of the initial word segmentation result based on the number of sub-word segmentation results whose character length exceeds the preset character length and their proportion in all sub-word segmentation results, to obtain the first judgment result.

[0042] Specifically, the first preset rule may represent comparing the character length of the sub-segmentation results in the initial word segmentation results with the preset character length, and judging the sub-segmentation results based on the comparison results, that is, the confidence of the initial word segmentation results may be calculated by the proportion of the number of sub-segmentation results in which the character length of each sub-segmentation in the initial word segmentation results exceeds the preset character length in all sub-segmentation results, wherein the preset character length may be set based on the specifics of the corresponding field. For example, the terms in a certain field are usually 2-5 characters long, and the preset character length may be set to 5, indicating that the sub-segmentation results with a length of more than 5 characters may have inaccurate segmentation, and then the confidence of the initial word segmentation result is calculated based on the proportion of the number of sub-segmentation results with a length of more than 5 characters, and there is no restriction here.

[0043] It should be noted that the embodiment of the present application can also set the character length to 1. In this way, if the secondary word segmentation results in the initial word segmentation result do not exceed 1 character, it can indicate that the initial word segmentation result is inaccurate. At this time, the confidence level may be 0, and the initial word segmentation result is unusable. It can be regarded as the initial word segmentation result combined with the re-word segmentation result is an empty set.

[0044] In addition, a character length range can also be set to calculate the ratio of the sub-segmentation results that fall within the character length range to all sub-segmentation results, and then calculate the confidence of the initial segmentation result. Of course, in this solution, the more sub-segmentation results that fall within the character length range, the better. Instead, the system needs to perform semantic analysis in combination with the corresponding field to obtain the best ratio result or ratio result range. When the ratio of the character length of the sub-segmentation result in the initial segmentation result falls within the character length range, falls within the ratio result range or is the best ratio result, it means that the initial segmentation result is highly accurate, and the confidence can be calculated based on the corresponding ratio to obtain the first judgment result. The first judgment result may include information such as the confidence of the initial segmentation result, the sub-segmentation confidence, and no restrictions are made here.

[0045] According to the technical solution of the above embodiment, by comparing the character length of the sub-segmentation result with a preset threshold, the accuracy of the initial word segmentation result can be quickly judged, and the confidence of the initial word segmentation result can be calculated to obtain a first judgment result. In this way, the accuracy of the word segmentation result can be converted into a quantitative confidence by calculating the proportion of abnormal or normal sub-segmentation results, which provides a reliable basis for subsequent processing and helps the system obtain accurate target word segmentation results.

[0046] In other embodiments, based on the first judgment result, the initial word segmentation result and the re-word segmentation result are processed to obtain a target word segmentation result, including: if the first judgment result represents a confidence level that exceeds a preset threshold, extracting the same word segmentation results from the initial word segmentation result and the re-word segmentation result to obtain the target word segmentation result; if the first judgment result represents a confidence level that does not exceed the preset threshold, removing the same word segmentation results as the initial word segmentation results from the re-word segmentation result to obtain the target word segmentation result.

[0047] Specifically, after obtaining the first word segmentation result, it can be determined whether the confidence of the initial word segmentation result exceeds a preset threshold to select how to combine the initial word segmentation result and the re-word segmentation result to obtain the target word segmentation result.

[0048] When the confidence of the initial word segmentation result exceeds a preset threshold (for example, 0.8), the first judgment result can indicate that most of the sub-word segmentation results in the initial word segmentation result are accurate. In this case, the system can extract the same word segmentation results in the initial word segmentation result and the re-word segmentation result, and use them as the target word segmentation results.

[0049] When the confidence of the initial word segmentation result does not exceed the preset threshold, the first judgment result may indicate that there are more inaccurate word segmentations in the initial word segmentation result. In this case, a sub-segmentation result that is the same as the initial word segmentation result may be proposed from the re-segmentation result. Of course, the confidence of the sub-segmentation result in the initial word segmentation result may also be calculated, and the sub-segmentation results whose confidence in the sub-segmentation result does not exceed another preset threshold in the same sub-segmentation result may be proposed, and the sub-segmentation results not proposed in the re-segmentation result may be retained to obtain the target word segmentation result.

[0050] Furthermore, the target word segmentation results can be saved for subsequent query, retrieval or further processing. For example, in the ES engine, the word segmentation results of the text data in the system are indexed, which can facilitate subsequent retrieval and word segmentation and improve the overall efficiency and accuracy of the search system.

[0051] In some embodiments, a custom word segmenter is used to re-segment the data to be segmented to obtain a re-segmentation result, including: using the custom word segmentation result to re-segment the data to be segmented to obtain a first word segmentation result, using a second preset rule to judge the accuracy of the first word segmentation result to obtain a second judgment result; if the second judgment result indicates that the first word segmentation result is accurate, then determining the first word segmentation result to be the re-segmentation result; if the second judgment result indicates that the first word segmentation result is inaccurate, then using the custom word segmenter to segment the data to be segmented again to obtain a second word segmentation result, until the second word segmentation result is accurate, and determining the second word segmentation result to be the re-segmentation result.

[0052] Specifically, when a custom word segmenter is used to segment data to be segmented, the segmentation result obtained may not be accurate. Therefore, the accuracy of the segmentation result may also be judged to obtain an accurate re-segmentation result output by the custom word segmenter.

[0053] Furthermore, the accuracy of the first word segmentation result can be evaluated by a second preset rule to obtain a second judgment result, wherein the second preset rule can be set according to specific application scenarios and field characteristics, such as the consistency of terms in the first word segmentation result, the character length of the terms, etc., which are not limited here.

[0054] If the second judgment result indicates that the first word segmentation result meets the accuracy standard, the first word segmentation result can be directly used as the re-word segmentation result, and the re-word segmentation result and the initial word segmentation result can be processed to obtain the target word segmentation result. If the second judgment result indicates that the first word segmentation result is inaccurate, a custom word segmenter can be used to re-segment the word segmentation data to obtain a second word segmentation result, and the accuracy of the second word segmentation result can be judged based on the second preset rule. In this way, an accurate second word segmentation result is obtained through iteration, and the accurate second word segmentation result is determined as the re-word segmentation result.

[0055] It should be noted that, in this embodiment, the first word segmentation result and the second word segmentation result obtained can be understood as the word segmentation results iteratively generated in the process of re-segmenting the word segmentation data using the custom word segmentor, without specific limitation.

[0056] According to the technical solution provided by the above embodiment, the accuracy of the first word segmentation result can be judged to obtain a second judgment result, and the re-word segmentation result can be determined based on the second judgment result. In this way, the custom word segmenter can be dynamically adjusted to ensure the accuracy and domain adaptability of the re-word segmentation result. Even if the first word segmentation result is inaccurate, the system can automatically adjust it through multiple iterative processes and finally generate a re-word segmentation result that meets the needs, which can help the system to improve the efficiency of word segmentation processing while ensuring the accuracy of the results.

[0057] In some embodiments, the accuracy of the first word segmentation result is judged using a second preset rule to obtain a second judgment result, including: establishing an index based on the first word segmentation result; obtaining a query statement, searching from the index based on the query statement, obtaining a retrieval result, and determining a second judgment result based on the retrieval result, and the query statement corresponds to the target field; if the retrieval result meets the first preset state, it indicates that the first word segmentation result is accurate; if the retrieval result meets the second preset state, it indicates that the first word segmentation result is inaccurate.

[0058] Specifically, the second preset rule may represent that whether the first word segmentation result is accurate is determined by constructing an index for the first word segmentation result and performing a search based on a query statement.

[0059] First, after obtaining the first word segmentation result, the first word segmentation result can be indexed to provide an operation basis for subsequent queries, so that the system can efficiently search and match query statements.

[0060] A query statement is obtained based on the professional corpus of the target field, and the query statement is input into the search engine. A search operation is performed through the established index, and then one or a series of search results matching the query statement are returned. The search statement can be set manually or automatically generated by the model.

[0061] Then, whether the first word segmentation result is accurate can be judged by whether the search result satisfies the first preset state or the second preset state, wherein the first preset state can represent that the search result is at least not empty, and the similarity between the search result and the query statement is greater than a threshold value (for example, 0.6); the second preset state can represent that the search result is empty or the similarity between the search result and the query statement is not greater than a threshold value (for example, 0.6). Of course, other reasonable judgment mechanisms are also possible and are not limited here.

[0062] According to the technical solution provided in the above embodiment, by constructing an index and querying, judging whether the first word segmentation result is accurate based on the search results can significantly improve the accuracy of word segmentation, especially when facing complex corpus or domain-specific vocabulary, the quality of system word segmentation can be improved.

[0063] In some embodiments, the trained large model is trained based on professional corpus in the target field, including: obtaining professional corpus in the target field, and annotating the professional corpus in the target field based on the professionalism of the target field to obtain training data; performing supervised learning on the training data to obtain a trained large model.

[0064] Specifically, the large model of the target domain can be trained through supervised learning methods to obtain a trained large model, and a custom word segmenter can be deployed to perform word segmentation on the data to be segmented to obtain accurate word segmentation results.

[0065] During training, you can first collect documents, data, and texts related to the target field (such as real estate, medical, finance and other professional fields), and annotate these texts. The annotation process can manually or semi-automatically process the corpus according to predetermined rules to generate data with clear labels. These labels may include part-of-speech tagging (such as nouns, verbs, etc.), grammatical relations, entity recognition (such as company names, place names, professional terms), etc., without limitation here. In this way, the annotated data forms a training set, which can be used to learn the large model so that it can better understand the professional terms and context in the target field.

[0066] According to the technical solution provided in the above embodiment, through the collection, annotation and supervised learning training of professional corpus in the target field, the obtained large model has the ability to process the professional language in the field. This process ensures that the large model can understand and generate content that is highly relevant to the target field when applied, significantly improving the performance of the word segmenter or other natural language processing tasks, especially when processing domain-specific vocabulary, complex sentence structures or professional documents, and thus helping the system to obtain accurate target word segmentation results.

[0067] Figure 2 This is a flow chart of another word segmentation method provided in the embodiment of the present application. Figure 2 The embodiments of the present application are further described.

[0068] In the ES search engine, the written data can be segmented by using the traditional word segmenter combined with the large model word segmenter (i.e., the custom word segmenter) to create an index.

[0069] First, the written data (i.e., the data to be segmented) can be preliminarily segmented in the form of IK+target domain professional dictionary to obtain the initial segmentation result, and determine whether the segmentation is accurate. If accurate, an index is established based on the initial segmentation result.

[0070] If the initial word segmentation result is inaccurate, the written data can be re-segmented based on the large model word segmenter. Similarly, when the large model word segmenter performs word segmentation, it is also necessary to determine whether the word segmentation result is accurate. If it is inaccurate (i.e., a bad case), the large model word segmenter can be reused for word segmentation until an accurate re-segmentation result is obtained, and an index is established based on the re-segmentation result.

[0071] Among them, the big model word segmenter can train a big language model through professional corpus in the target field (such as professional corpus in the real estate field), and integrate the trained big model into the ES search engine. When the big model word segmenter is needed, the system can directly request the trained big model. If the word segmentation result is inaccurate during the word segmentation process using the big model word segmenter, the big model can be retrained and the retrained big model can be requested to perform word segmentation on the input data. There is no restriction here.

[0072] This can help ES process domain-related texts more accurately during indexing and searching based on the trained single model, thereby improving the performance of the search system and user experience.

[0073] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0074] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0075] Figure 3 Schematic diagram of a word segmentation device provided in an embodiment of the present application. Figure 3 As shown, the word segmentation device includes:

[0076] The acquisition module 301 is configured to acquire data to be segmented;

[0077] The preliminary word segmentation module 302 is configured to perform preliminary word segmentation on the word segmentation data to obtain an initial word segmentation result;

[0078] The re-segmentation module 303 is configured to re-segment the data to be segmented using a custom segmenter to obtain a re-segmentation result, wherein the custom segmenter is obtained by deploying a trained large model, and the trained large model is obtained by training based on professional corpus of a target field, and the target field corresponds to the field to which the data to be segmented belongs;

[0079] The combining module 304 is configured to combine the initial word segmentation result and the re-word segmentation result to obtain a target word segmentation result.

[0080] In some embodiments, the combination module 304 is specifically used to judge the confidence of the initial word segmentation result based on the first preset rule to obtain a first judgment result; based on the first judgment result, the initial word segmentation result and the re-word segmentation result are processed to obtain a target word segmentation result.

[0081] In some embodiments, the combination module 304 is specifically used to obtain sub-segmentation results in the initial word segmentation results whose character length exceeds the preset character length; based on the number of sub-segmentation results whose character length exceeds the preset character length and their proportion in all sub-segmentation results, the confidence of the initial word segmentation results is determined to obtain a first judgment result.

[0082] In some embodiments, the combination module 304 is specifically used to extract the same segmentation results in the initial segmentation result and the re-segmentation result if the confidence represented by the first judgment result exceeds a preset threshold, and obtain the target segmentation result; if the confidence represented by the first judgment result does not exceed the preset threshold, then remove the segmentation results that are the same as the initial segmentation results from the re-segmentation result to obtain the target segmentation result.

[0083] In some embodiments, the re-segmentation module 303 is specifically used to re-segment the data to be segmented using the custom segmentation result to obtain a first segmentation result, and use a second preset rule to judge the accuracy of the first segmentation result to obtain a second judgment result; if the second judgment result indicates that the first segmentation result is accurate, then the first segmentation result is determined to be the re-segmentation result; if the second judgment result indicates that the first segmentation result is inaccurate, then the custom segmentation device is used to segment the data to be segmented again to obtain a second segmentation result, until the second segmentation result is accurate, and the second segmentation result is determined to be the re-segmentation result.

[0084] In some embodiments, the re-segmentation module 303 is specifically used to establish an index based on the first segmentation result; obtain a query statement, search from the index based on the query statement, obtain a search result, and determine a second judgment result based on the search result, and the query statement corresponds to the target field; if the search result meets the first preset state, it indicates that the first segmentation result is accurate; if the search result meets the second preset state, it indicates that the first segmentation result is inaccurate.

[0085] In some embodiments, the re-segmentation module 303 is specifically used to obtain professional corpus in the target field, and annotate the professional corpus in the target field based on the professionalism of the target field to obtain training data; perform supervised learning on the training data to obtain a trained large model.

[0086] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0087] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present application. Figure 4As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0088] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.

[0089] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0090] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0091] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A word segmentation method, characterized in that: include: Get the data to be segmented; Performing preliminary word segmentation on the data to be segmented to obtain an initial word segmentation result; Re-segmenting the data to be segmented by using a custom segmenter to obtain a re-segmentation result, wherein the custom segmenter is obtained by deploying a trained large model, and the trained large model is obtained by training based on professional corpus of a target field, and the target field corresponds to the field to which the data to be segmented belongs; The initial word segmentation result and the re-word segmentation result are combined to obtain a target word segmentation result.

2. The method according to claim 1, characterized in that The combining the initial word segmentation result and the re-word segmentation result to obtain a target word segmentation result includes: The confidence of the initial word segmentation result is judged based on a first preset rule to obtain a first judgment result; Based on the first judgment result, the initial word segmentation result and the re-word segmentation result are processed to obtain the target word segmentation result.

3. The method according to claim 2, characterized in that The initial word segmentation result includes at least two sub-word segmentation results, and the confidence of the initial word segmentation result is judged based on the first preset rule to obtain a first judgment result, including: Obtaining sub-segmentation results whose character length exceeds a preset character length in the initial segmentation results; Based on the number of sub-segmentation results whose character length exceeds the preset character length and their proportion in all the sub-segmentation results, the confidence of the initial segmentation result is determined to obtain the first judgment result.

4. The method according to claim 2, characterized in that: The processing of the initial word segmentation result and the re-word segmentation result based on the first judgment result to obtain the target word segmentation result includes: If the first judgment result indicates that the confidence level exceeds a preset threshold, extracting the same segmentation results from the initial segmentation result and the re-segmentation result to obtain the target segmentation result; If the first judgment result indicates that the confidence level does not exceed the preset threshold, the word segmentation results that are the same as the initial word segmentation results are removed from the re-word segmentation results to obtain the target word segmentation results.

5. The method according to claim 1, characterized in that The using the custom word segmenter to re-segment the data to be segmented to obtain a re-segmentation result includes: Re-segmenting the data to be segmented using the custom segmentation result to obtain a first segmentation result, and judging the accuracy of the first segmentation result using a second preset rule to obtain a second judgment result; If the second judgment result indicates that the first word segmentation result is accurate, determining the first word segmentation result as the re-word segmentation result; If the second judgment result indicates that the first word segmentation result is inaccurate, the data to be segmented is segmented again using the custom word segmenter to obtain a second word segmentation result, until the second word segmentation result is accurate, and the second word segmentation result is determined to be the re-segmentation result.

6. The method according to claim 5, characterized in that The using the second preset rule to judge the accuracy of the first word segmentation result to obtain a second judgment result includes: Establishing an index based on the first word segmentation result; Obtaining a query statement, searching from the index based on the query statement to obtain a search result, and determining the second judgment result based on the search result, wherein the query statement corresponds to the target field; If the search result satisfies a first preset state, it indicates that the first word segmentation result is accurate; If the search result satisfies a second preset state, it indicates that the first word segmentation result is inaccurate.

7. The method according to claim 1, characterized in that The trained large model is trained based on professional corpus in the target field, including: Acquire professional corpus in the target field, and annotate the professional corpus in the target field based on the professionalism of the target field to obtain training data; Supervised learning is performed on the training data to obtain the trained large model.

8. A word segmentation device, characterized in that: include: An acquisition module is configured to acquire data to be segmented; A preliminary word segmentation module is configured to perform preliminary word segmentation on the data to be segmented to obtain an initial word segmentation result; a re-segmentation module, configured to re-segment the data to be segmented using a custom segmenter to obtain a re-segmentation result, wherein the custom segmenter is obtained by deploying a trained large model, and the trained large model is obtained by training based on professional corpus of a target field, and the target field corresponds to the field to which the data to be segmented belongs; The combining module is configured to combine the initial word segmentation result and the re-word segmentation result to obtain a target word segmentation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images