Text data processing method and device, computer device, and storage medium

Abstract

The application relates to a text data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: performing segmentation processing on a to-be-processed text in a target field to obtain a plurality of to-be-processed text blocks; matching a professional word in a preset field word set with each to-be-processed text block to determine the number of times that the professional word appears in each to-be-processed text block, wherein the professional word comprises a weight professional word and a professional new word, the weight professional word is determined according to the weight of sample segmentation in a sample text in the target field, and the professional new word is generated according to the sample segmentation; in response to a judgment result that the number of times that the professional word appears satisfies a preset condition, taking the to-be-processed text block as a target text block; and generating a triple relationship of the target field according to the target text block. The method can reduce the data processing amount of a computer, thereby improving the construction efficiency of the triple relationship.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for processing text data. Background Technology

[0002] Domain-specific texts typically encompass a broad range of knowledge and contain numerous domain-specific terms. Constructing knowledge graphs for such domains demands a high level of expertise from technical personnel. For instance, when building a knowledge graph for the financial domain using operational texts, technical personnel must first understand financial terminology such as P2P (peer-to-peer lending), trust, and money supply within the texts before they can extract knowledge and construct the corresponding knowledge graph.

[0003] In traditional techniques, experts in the field typically extract triples from specialized texts based on their business experience, identifying triple relationships between nodes, relations, and attributes. Then, using data annotation or pre-trained models, knowledge is extracted and populated to form a knowledge graph for the specific domain. However, traditional text data processing methods require significant manpower, resulting in high processing costs. Summary of the Invention

[0004] Therefore, it is necessary to provide a low-cost method, apparatus, computer device, computer-readable storage medium, and computer program product for processing text data to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for processing text data. The method includes:

[0006] The text to be processed in the target domain is segmented to obtain multiple text blocks to be processed.

[0007] The professional terms in the preset domain term set are matched with each of the text blocks to be processed to determine the number of times the professional terms appear in each of the text blocks to be processed. The professional terms include weighted professional terms and new professional terms. The weighted professional terms are determined according to the weight of the sample word segmentation in the sample text under the target domain. The new professional terms are generated according to the sample word segmentation.

[0008] In response to the judgment result that the number of times the professional term appears meets the preset condition, the text block to be processed is taken as the target text block;

[0009] Generate triplet relations for the target domain based on the target text block.

[0010] In one embodiment, before determining that the number of times the specialized term appears meets a preset condition and designating the text block to be processed as the target text block, the method further includes:

[0011] Obtain the text length of each of the text blocks to be processed;

[0012] The frequency of the professional term in each text block to be processed is determined based on the text length of each text block to be processed and the number of times the professional term appears.

[0013] If the frequency is greater than a preset threshold, a judgment result is generated indicating that the number of times the professional term appears meets the preset condition.

[0014] In one embodiment, the method for generating the preset domain terminology set includes:

[0015] The sample text is segmented into words to obtain multiple sample words in the sample text;

[0016] In response to the judgment result that the weight of the sample word segmentation meets the first preset requirement, the sample word segmentation is used as the weighted professional word;

[0017] Multiple sample word segments are combined to obtain multiple sample new words, and the professional new words are determined from the multiple sample new words;

[0018] The preset domain term set is generated using the weighted professional terms and the new professional terms.

[0019] In one embodiment, generating the preset domain terminology set using the weighted professional terms and the new professional terms includes:

[0020] Based on the attributes of the weighted professional terms and the new professional terms, the weighted professional terms and the new professional terms are filtered to obtain the filtered target professional terms;

[0021] The preset domain terminology set is generated using the target professional terms.

[0022] In one embodiment, before determining that the weight of the sample word segmentation meets a first preset requirement and using the sample word segmentation as the weighted professional term, the method further includes:

[0023] Obtain the first occurrence frequency of each of the sample word segments in the sample text;

[0024] The weight of each sample word is determined based on its first occurrence frequency.

[0025] In one embodiment, the step of combining multiple sample word segments to obtain multiple new sample words, and determining the specialized new word from the multiple new sample words, includes:

[0026] Each of the sample words is segmented to obtain word fragments corresponding to each sample word;

[0027] Based on the position information of the word segmentation fragments, the word segmentation fragments corresponding to adjacent sample word segments are combined to generate multiple new sample words;

[0028] An evaluation parameter for each new word is generated based on its second frequency of occurrence in the sample text.

[0029] In response to the judgment result that the evaluation parameters meet the second preset requirements, the sample new words are taken as the professional new words.

[0030] Secondly, this application also provides a text data processing apparatus. The apparatus includes:

[0031] The text segmentation module is used to segment the text to be processed in the target domain, resulting in multiple text blocks to be processed.

[0032] The frequency determination module is used to match the professional words in the preset domain word set with each of the text blocks to be processed, and determine the frequency of the professional words in each text block to be processed. The professional words include weighted professional words and new professional words. The weighted professional words are determined according to the weight of the sample word segmentation in the sample text under the target domain, and the new professional words are generated according to the sample word segmentation.

[0033] The result response module is used to respond to the judgment result that the number of times the professional term appears meets the preset conditions, and to take the text block to be processed as the target text block.

[0034] The relation generation module is used to generate triple relations of the target domain based on the target text block.

[0035] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the text data processing method described in any of the embodiments of the first aspect.

[0036] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the text data processing method described in any of the embodiments of the first aspect.

[0037] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the text data processing method described in any of the embodiments of the first aspect.

[0038] The aforementioned text data processing method, apparatus, computer equipment, storage medium, and computer program product segment the text to be processed within a target domain to obtain multiple text blocks. Each text block is then matched with specialized terms from a predefined domain terminology set to determine the frequency of each specialized term's occurrence. These specialized terms include weighted specialized terms and new specialized terms. Weighted specialized terms are determined based on the weights of sample word segments in the sample text within the target domain, while new specialized terms are generated based on the sample word segments. In response to a judgment result indicating that the frequency of specialized term occurrences meets a predefined condition, the text block to be processed is designated as the target text block. A target domain triple relation is then generated based on the target text block. This method can utilize specialized terms from a predefined domain terminology set to determine the target text block from multiple text blocks within the target domain, thereby improving the efficiency of target text block determination. Furthermore, the target text block is used to construct a target domain triple relation, simplifying the triple relation construction process, reducing the data processing volume for computer triple relation construction, and thus improving the efficiency of triple relation construction. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a text data processing method in one embodiment. Figure 1 ;

[0040] Figure 2 This is a flowchart illustrating the steps for generating the judgment result in one embodiment;

[0041] Figure 3 This is a flowchart illustrating the steps for generating a preset domain term set in one embodiment;

[0042] Figure 4 This is a flowchart illustrating the steps for determining new professional terms in another embodiment;

[0043] Figure 5 This is a flowchart illustrating a text data processing method in another embodiment. Figure 2 ;

[0044] Figure 6 This is a structural block diagram of a text data processing device in one embodiment;

[0045] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0048] In one embodiment, such as Figure 1 As shown, a method for processing text data is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart TVs, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0049] In this embodiment, the method includes the following steps:

[0050] Step S102: Segment the text to be processed in the target domain to obtain multiple text blocks to be processed.

[0051] The target field may include, but is not limited to, any one or more of the following professional fields: clinical medicine, biology, finance, etc.

[0052] The text block to be processed can include, but is not limited to, any one or more of the following types of text blocks: chapter text block, paragraph text block, or sentence text block.

[0053] Specifically, the server can pre-deploy a text segmentation algorithm. In response to a text data processing request, the server can retrieve the text to be processed within the target domain from a local database, or it can download the text to be processed within the target domain from the internet. The server can then invoke the text segmentation algorithm to segment the text to be processed, obtaining multiple text blocks corresponding to the original text.

[0054] The text segmentation algorithm can be implemented by setting a similarity segmentation threshold. For example, word embedding is performed on the text to be processed, converting the text into a corresponding vector sequence. The text is then segmented based on the comparison between the semantic similarity between the vectors in the vector sequence and the similarity segmentation threshold. Alternatively, a length segmentation threshold can be set. For example, full-text word segmentation is performed on the text to be processed. Starting from the first word in the text, the number of word segments is accumulated sequentially until the accumulated number of word segments equals the length segmentation threshold. The accumulated word segments are then concatenated as a single text block to be processed. This process is repeated for the unconcatenated word segments until the last word in the text is traversed, resulting in multiple text blocks corresponding to the original text.

[0055] Step S104: Match the professional terms in the preset domain term set with each text block to be processed to determine the number of times the professional terms appear in each text block to be processed.

[0056] Among them, the preset domain term set can be used to represent the set of professional terms in the target domain.

[0057] Specialized terms can include weighted specialized terms and newly generated specialized terms. Weighted specialized terms are used to represent specialized terms determined based on the weights of word segmentation in sample text within the target domain. Newly generated specialized terms are used to represent specialized terms generated based on sample word segmentation.

[0058] Specifically, during the construction phase of a preset domain lexicon, the server can acquire sample texts from multiple target domains. Each sample text can be segmented, and the weight of each segment can be determined. Segments whose weights meet a preset weight threshold are designated as weighted segments. The segmented words in the sample texts can be recombine to generate new specialized terms. The weighted segments and new specialized terms in each sample text are then filtered and deduplicated. Using the weighted segments and new specialized terms obtained after filtering and deduplication, a preset domain lexicon corresponding to the target domain is constructed and stored on the server.

[0059] During the application phase of the preset domain terminology set, the following operations can be performed on each text block to be processed: The text block to be processed is segmented into multiple text block segments. Each text block segment in the text block to be processed is matched with each professional term in the preset domain terminology set. If the matching degree between the text block segment and the professional term is greater than a preset matching degree threshold, it is determined that the professional term appears in the text block to be processed, and the number of times the professional term appears in the text block to be processed is recorded.

[0060] Step S106: In response to the judgment result that the number of times the professional words appear meets the preset conditions, the text block to be processed is taken as the target text block.

[0061] Specifically, the server can pre-store preset conditions corresponding to the frequency of occurrence of specialized terms. For each text block to be processed, the following operations can be performed: the frequency of occurrence of specialized terms in the text block is judged based on the preset conditions. If the frequency meets the preset conditions, the server can determine that the current text block to be processed has a high probability of containing triples in the target domain, or a high density of triples in the target domain, which is beneficial for subsequent construction of triples. In response to the judgment result that the frequency of specialized terms meets the preset conditions, the server can designate the text block to be processed as the target text block. The preset conditions can be either a frequency greater than a preset threshold, or a frequency within a preset range.

[0062] Step S108: Generate triplet relations for the target domain based on the target text block.

[0063] Specifically, the server can extract entity relations from the target text block and generate triple relations for the target domain using the extracted text entity relations. Preferably, the server can directly extract entity relations from the word segmentation of the text block that matches the professional term and generate triple relations for the target domain using the word segmentation entity relations corresponding to the word segmentation.

[0064] Alternatively, the server can send the target text block to a display input device, instructing the display input device to show the target text block to the user in the display interface, and then return the entity relation extraction results of the target text block entered by the user in the input interface to the server through the display input device. The server can process the received entity relation extraction results to generate triple relations in the target domain.

[0065] In the above text data processing method, the text to be processed under the target domain is segmented to obtain multiple text blocks to be processed; professional terms from a preset domain terminology set are matched with each text block to determine the frequency of professional terms in each text block. These professional terms include weighted professional terms and new professional terms. Weighted professional terms are determined based on the weights of sample word segments in the sample text under the target domain, while new professional terms are generated based on sample word segments. In response to the judgment result that the frequency of professional terms meets a preset condition, the text block to be processed is designated as the target text block; and a triple relation for the target domain is generated based on the target text block. This method can utilize professional terms from a preset domain terminology set to determine the target text block from multiple text blocks to be processed under the target domain, thereby improving the efficiency of target text block determination. Furthermore, the target text block is used to construct a triple relation for the target domain, simplifying the triple relation construction process, reducing the data processing volume for computer construction of triple relations, and thus improving the efficiency of triple relation construction.

[0066] In one embodiment, such as Figure 2 As shown, in step S106, before using the text block to be processed as the target text block in response to the judgment result that the number of times the professional term appears meets the preset condition, the following steps are also included:

[0067] Step S202: Obtain the text length of each text block to be processed.

[0068] Step S204: Determine the frequency of professional terms in each text block to be processed based on the text length and the number of times professional terms appear in each text block to be processed.

[0069] Step S206: If the frequency is greater than a preset threshold, generate a judgment result that the number of times the professional term appears meets the preset conditions.

[0070] Specifically, the preset condition stored in the server can be that the frequency of professional terms appearing in the text block to be processed is greater than a preset threshold. The server can perform the following operations on each text block to be processed: perform word segmentation on the text block, and determine the text length of the text block based on the number of words obtained from the word segmentation. Then, calculate the ratio between the text length and the frequency of professional terms in the text block, and use this ratio as the frequency of the professional term in the text block. Finally, compare the frequency with a preset threshold; if the frequency is greater than the preset threshold, generate a judgment result indicating that the frequency of professional terms appearing in the text block meets the preset condition.

[0071] In one example, the preset threshold can be 25:1. The length of the text block to be processed obtained by the server can be 500 bytes, and the number of times the professional words appear in the text block can be 30. Then, the text length of the text block to be processed and the number of times the professional words appear are calculated, resulting in a frequency ratio of 50:3 for the professional words. This frequency is compared with the preset threshold, and if the frequency is greater than the preset threshold, the result is that the number of times the professional words appear in the text block meets the preset condition.

[0072] In this embodiment, the text length of the text block to be processed and the number of times professional words appear in the text block to be processed are calculated. Based on the comparison result that the frequency of professional words appearing in the text block is greater than a preset threshold, the text block to be processed is taken as the target text block. This can comprehensively consider the text length of the text block and the frequency of professional words appearing, thereby improving the accuracy of the target text block.

[0073] In one embodiment, such as Figure 3 The diagram illustrates the steps involved in generating a preset domain terminology set, including:

[0074] Step S302: Perform word segmentation on the sample text to obtain multiple sample words in the sample text.

[0075] Step S304: In response to the judgment result that the weight of the sample word segmentation meets the first preset requirement, the sample word segmentation is used as the weight professional word.

[0076] Specifically, the server can perform word segmentation on sample text within the target domain, obtaining multiple sample words. The weight of each sample word in the sample text can be determined based on its semantic similarity to the sample text, or it can be determined based on the frequency of each sample word's occurrence in the sample text. The weight of each sample word is then judged using pre-stored first preset requirements, generating a corresponding judgment result. If the weight of a sample word meets the first preset requirements, in response to the judgment result that the weight of the sample word meets the first preset requirements, the sample word is designated as a weighted professional term.

[0077] Step S306: Combine multiple sample word segments to obtain multiple sample new words, and identify professional new words from the multiple sample new words.

[0078] Specifically, the server can split and recombine characters from multiple sample word segments to generate multiple new sample words. Specialized new words can be determined from these sample words based on the semantic similarity between each new sample word and the sample text, or based on evaluation parameters obtained from evaluating each new sample word using preset evaluation rules.

[0079] Step S308: Generate a preset domain term set using weighted professional terms and new professional terms.

[0080] Specifically, the server can perform deduplication filtering on weighted professional terms and new professional terms to obtain an initial set of professional terms consisting of non-repeating weighted professional terms and new professional terms. Preferably, after performing part-of-speech filtering on the initial set of professional terms, the part-of-speech filtered initial set of professional terms can be displayed to the user, who can then manually filter it again, using the manually filtered initial set of professional terms as the preset domain term set corresponding to the target domain.

[0081] In this embodiment, by performing word segmentation on the sample text, the weight of each sample word in the sample text is obtained, and weighted professional words are determined based on the weights. Multiple sample words are split and combined to obtain multiple new sample words. Professional new words are determined from the multiple new sample words. A preset domain word set is generated using the weighted professional words and the professional new words. This can expand the vocabulary range covered by the preset domain word set, so that the preset domain word set not only covers professional words whose weights meet the first preset requirement, but also includes the professional new words obtained by combination. This improves the accuracy of subsequent filtering of the text block to be processed based on the professional words in the preset domain word set to obtain the target text block.

[0082] In one embodiment, step S308, generating a preset domain term set using weighted professional terms and new professional terms, includes: filtering the weighted professional terms and new professional terms according to their attributes to obtain filtered target professional terms, and generating a preset domain term set using the target professional terms.

[0083] The attributes can include, but are not limited to, any one of the various word category attributes such as nouns, verbs, and adjectives.

[0084] Specifically, the server can store filtering attributes. The server can obtain the attributes of each weighted professional term and each new professional term, and compare the attributes of each weighted professional term or each new professional term with the filtering attributes. When it is determined that the attributes of a weighted professional term or a new professional term are the same as the filtering attributes, the currently compared weighted professional term or new professional term is filtered, and the filtered weighted professional term and / or new professional term is used as the filtered target professional term. A preset domain term set corresponding to the target domain is generated using the target professional term. The filtering attributes can be verbs; or adjectives; or non-nouns.

[0085] In this embodiment, by performing part-of-speech filtering on weighted professional terms and new professional terms, and using the filtered weighted professional terms and new professional terms as target professional terms to construct a preset domain term set, the accuracy of the preset domain term set can be improved.

[0086] In one embodiment, before using sample words as weighted professional words in step S304 in response to the judgment result that the weight of the sample word meets the first preset requirement, the steps include: obtaining the first occurrence frequency of each sample word in the sample text, and determining the weight of each sample word based on the first occurrence frequency of each sample word.

[0087] Specifically, the server can obtain the total number of sample words in the sample text, and the number of times each sample word appears in the sample text. It then performs calculations on the total number of sample words and the number of occurrences of each sample word to obtain the first occurrence frequency of each sample word in the sample text. The first occurrence frequencies of each sample word are then sorted from highest to lowest to generate a ranking result for each sample word. The weight of each sample word is determined based on its ranking result.

[0088] In one example, if the sample word segmentation ranking results are in the top 30%, the server can generate a judgment result that the weight of the sample word segmentation meets the first preset requirement.

[0089] In this embodiment, by obtaining the first occurrence frequency of the sample word segment in the sample text, sorting the sample word segments according to the first occurrence frequency, and determining the weight of the sample word segments based on the sorting result, it is beneficial to improve the accuracy of the weighted professional words determined subsequently based on the weight of the sample word segments.

[0090] In one embodiment, such as Figure 4 As shown, in step S306, multiple sample word segments are combined to obtain multiple sample new words, and specialized new words are determined from these multiple sample new words, including:

[0091] Step S402: Perform segmentation processing on each sample word segment to obtain the word fragments corresponding to each sample word segment.

[0092] Step S404: According to the position information of the word segmentation fragments, combine the word segmentation fragments corresponding to adjacent sample word segments to generate multiple sample new words.

[0093] Among them, word segmentation fragments can be used to represent individual characters obtained after sample word segmentation processing.

[0094] Specifically, the server can perform unary segmentation on each sample word segment to obtain multiple word fragments corresponding to each sample word segment. Based on the position of each word fragment in the sample text, the multiple word fragments corresponding to adjacent sample words are combined to generate multiple new sample words.

[0095] In one example, the sample text may include ABCDEF. The server performs word segmentation on the sample text, resulting in sample word segments AB, CD, and EF. Further segmentation of the sample words yields word fragments A, B, C, D, E, and F. Combining these fragments generates new sample words BC, DE, etc.

[0096] Step S406: Generate evaluation parameters for each new word in the sample text based on the second frequency of each new word in the sample text.

[0097] Step S408: In response to the judgment result that the evaluation parameters meet the second preset requirements, the sample new words are regarded as professional new words.

[0098] The evaluation parameters can be used to characterize the evaluation results of the sample new words as professional new words. The evaluation parameters can include, but are not limited to, one or more of the following parameters: degree of freedom parameter, cohesion parameter, frequency of occurrence, etc.

[0099] The degree of freedom parameter can be used to characterize the degree of freedom in sample word segmentation, that is, the probability that sample word segmentation appears adjacent to other words.

[0100] The coagulation parameter can be used to characterize the coagulation between word fragments in sample word segmentation.

[0101] Frequency of occurrence can be used to characterize how often sample words appear in sample text.

[0102] Specifically, the server can obtain the number of times each new word appears in the sample text, as well as the number of sample words in the sample text. The server then performs calculations on the number of times each new word appears in the sample text and the number of sample words in the sample text to obtain the second frequency of each new word in the sample text.

[0103] For each new word sample, obtain the third occurrence probability of each segmented word corresponding to the new word. Calculate the third occurrence probability of each segmented word and the second occurrence probability of the new word sample to obtain the cohesion parameter of the new word sample. Obtain the new word sample and adjacent word combinations composed of adjacent characters / words. Determine the degree of freedom parameter of the new word sample based on the occurrence probability of adjacent word combinations. Use at least one of the second occurrence frequency, cohesion parameter, or degree of freedom parameter of the new word sample as the evaluation parameter of the new word sample.

[0104] The evaluation parameters of each sample new word are judged according to the second preset requirements. If the evaluation parameters of the sample new word meet the second preset requirements, the server can recognize the sample new word as a professional new word in response to the judgment result that the evaluation parameters meet the second preset requirements.

[0105] In one example, if the evaluation parameters include a second frequency of occurrence, a judgment result indicating that the evaluation parameters meet the second preset requirement can be obtained if the second frequency of occurrence is greater than a preset frequency threshold.

[0106] In another example, when the evaluation parameters include degrees of freedom parameters, a judgment result indicating that the evaluation parameters meet the second preset requirement can be obtained if the degrees of freedom parameters are less than the preset degrees of freedom threshold.

[0107] In another example, when the evaluation parameters include a solidification parameter, a judgment result indicating that the evaluation parameters meet the second preset requirement can be obtained if the solidification parameter is greater than a preset solidification threshold.

[0108] Since specialized new terms in specific fields may appear infrequently, it is difficult to identify them using word segmentation weights. Therefore, the specialized new term identification method provided in this embodiment is adopted. By segmenting sample words, combining the segmented word fragments, sample new terms are obtained. Based on the evaluation parameters of the sample new terms, specialized new terms are identified from multiple sample new terms. This simplifies the specialized new term extraction process and improves the extraction efficiency and accuracy.

[0109] In one embodiment, such as Figure 5 As shown, a method for processing text data is provided, including:

[0110] Step S502: Determine the weight of the sample word based on the first occurrence frequency of the sample word in the sample text. In response to the judgment result that the weight of the sample word meets the first preset requirement, the sample word is used as the weight professional word.

[0111] Specifically, during the construction phase of a preset domain terminology set, the server can perform word segmentation on the sample text to obtain multiple sample word segments corresponding to the sample text. The first occurrence frequency of each sample word is obtained, and each sample word is sorted according to its first occurrence probability to obtain a sorting result. The weight of each sample word is determined based on its sorting result. The weight of each sample word is judged according to a first preset requirement. If the weight of a sample word meets the first preset requirement, it can be used as a weighted professional term in response to the judgment result that the weight of the sample word meets the first preset requirement. The specific operation of determining weighted professional terms can be implemented with reference to the weighted professional term generation method provided in the above embodiments, and will not be elaborated here.

[0112] Step S504: Combine the segmented fragments corresponding to the sample segmentation to generate new sample words. In response to the judgment result that the evaluation parameters meet the second preset requirements, the new sample words are used as professional new words.

[0113] Specifically, during the construction phase of the preset domain vocabulary set, the server can segment the sample words to obtain segmentation fragments corresponding to the sample words. Based on the position of the segmentation fragments, the server combines the segmentation fragments corresponding to adjacent sample words to generate multiple new sample words. Evaluation parameters for each new sample word are obtained. The evaluation parameters of the new sample words are then judged according to a second preset requirement. If the evaluation parameters of the new sample word meet the second preset requirement, the new sample word can be designated as a professional new word in response to the judgment result that the evaluation parameters of the new sample word meet the second preset requirement. The specific operation for determining professional new words can be implemented with reference to the method for determining professional new words provided in the above embodiments, and will not be elaborated here.

[0114] Step S506: Filter weighted professional terms and new professional terms according to attributes to generate a preset domain term set.

[0115] Specifically, during the construction phase of a preset domain terminology set, the server can obtain the attributes of weighted professional terms and new professional terms. Based on these attributes, the server filters the weighted professional terms and new professional terms, using the filtered weighted professional terms and new professional terms as target professional terms. The target professional terms are sent to the client for display to the user, who then manually filters the target professional terms to obtain the final professional terms. The server then uses these manually filtered professional terms to generate a preset domain terminology set. The specific operations for generating the preset domain terminology set can be implemented using the method described in the above embodiments, and will not be elaborated upon here.

[0116] Step S508: Match the text block to be processed corresponding to the text to be processed in the target domain with the professional words in the preset domain word set to determine the number of times the professional words appear in the text block to be processed.

[0117] Specifically, during the construction phase of the triple relation, the server can segment the text to be processed in the target domain to obtain multiple text blocks to be processed. Preferably, the text blocks to be processed can be paragraph text blocks. The server matches each text block with professional terms from a preset domain terminology set to determine the frequency of professional terms appearing in the text blocks. The server obtains the text length of each text block. The server performs calculations on the text length of each text block and the frequency of professional terms appearing in the text blocks to determine the frequency of professional terms appearing in each text block. The server compares the frequency of professional terms appearing in each text block with a preset threshold. If the frequency of professional terms appears greater than the preset threshold, a judgment result is generated indicating that the frequency of professional terms appears meets a preset condition. The specific operation for determining the frequency of professional terms can be implemented with reference to the method for determining the frequency of professional terms provided in the above embodiments, and will not be elaborated here.

[0118] Step S510: In response to the judgment result that the number of times the professional words appear meets the preset conditions, the text block to be processed is taken as the target text block, and a triple relation of the target domain is generated based on the target text block.

[0119] Specifically, during the construction phase of the triple relation, in response to a judgment result indicating that the frequency of occurrence of specialized terms meets preset conditions, the server can use the text block to be processed as the target text block. The target text block is sent to the client for display to the user, and the server obtains the entity relation extraction results returned after the user extracts entity relations from the target text block. Based on the entity relation extraction results of the target text block, triple relations for the target domain are generated. The specific triple relation generation operation can be implemented with reference to the triple relation generation method provided in the above embodiments, and will not be elaborated here.

[0120] In this embodiment, by segmenting sample text within the target domain, determining weights based on the frequency of occurrence of the segmented words, and ranking them to determine weighted professional terms, the accuracy of weighted professional terms can be improved. By recombining segmented fragments to generate new sample words, and determining new professional terms based on the evaluation parameters of these new sample words, the process of determining new professional terms can be simplified, improving the efficiency of new professional term determination. By filtering weighted professional terms and new professional terms to generate a preset domain term set, the amount of data required to construct the preset domain term set can be reduced, improving the generation efficiency of the preset domain term set. By matching the text block to be processed within the target domain with professional terms in the preset domain term set, the target text block is determined. Using the target text block to generate triple relations within the target domain not only reduces the difficulty of constructing triple relations but also reduces the amount of data required, improving the construction efficiency of triple relations.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0122] Based on the same inventive concept, this application also provides a text data processing apparatus for implementing the text data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more text data processing apparatus embodiments provided below can be found in the limitations of the text data processing method described above, and will not be repeated here.

[0123] In one embodiment, such as Figure 6 As shown, a text data processing device 600 is provided, including: a text segmentation module 602, a frequency determination module 604, a result response module 606, and a relationship generation module 608, wherein:

[0124] The text segmentation module 602 is used to segment the text to be processed in the target domain to obtain multiple text blocks to be processed.

[0125] The frequency determination module 604 is used to match the professional words in the preset domain word set with each text block to be processed, and determine the frequency of professional words in each text block to be processed. The professional words include weighted professional words and new professional words. The weighted professional words are determined according to the weight of the sample word segmentation in the sample text under the target domain, and the new professional words are generated according to the sample word segmentation.

[0126] The result response module 606 is used to respond to the judgment result that the number of times the professional words appear meets the preset conditions, and to take the text block to be processed as the target text block.

[0127] The relation generation module 608 is used to generate triple relations of the target domain based on the target text block.

[0128] In one embodiment, the result response module 606 further includes: a length acquisition unit, used to acquire the text length of each text block to be processed; a frequency determination unit, used to determine the frequency of professional words appearing in each text block to be processed based on the text length of each text block to be processed and the number of times professional words appear; and a result generation unit, used to generate a judgment result that the number of times professional words appear meets a preset condition when the frequency is greater than a preset threshold.

[0129] In one embodiment, the text data processing apparatus 600 further includes a word set generation module.

[0130] The word set generation module includes: a word segmentation processing unit, which performs word segmentation processing on the sample text to obtain multiple sample words in the sample text; a weight judgment unit, which, in response to the judgment result that the weight of the sample word meets the first preset requirement, uses the sample word as a weighted professional word; a new word generation unit, which combines multiple sample words to obtain multiple sample new words and determines professional new words from the multiple sample new words; and a word set generation unit, which uses the weighted professional words and professional new words to generate a preset domain word set.

[0131] In one embodiment, the word set generation unit is further configured to filter the weighted professional words and new professional words according to their attributes to obtain filtered target professional words; and use the target professional words to generate a preset domain word set.

[0132] In one embodiment, the weight determination unit is further configured to obtain the first occurrence frequency of each sample word in the sample text; and determine the weight of each sample word based on the first occurrence frequency of each sample word.

[0133] In one embodiment, the new word generation unit further includes: a segmentation processing subunit, used to segment each sample word to obtain a segmentation fragment corresponding to each sample word; a fragment combination subunit, used to combine the segmentation fragments corresponding to adjacent sample words according to the position information of the segmentation fragments to generate multiple sample new words; a new word evaluation subunit, used to generate evaluation parameters for each sample new word based on the second occurrence frequency of each sample new word in the sample text; and a result response subunit, used to recognize the sample new word as a professional new word in response to the judgment result that the evaluation parameters meet the second preset requirements.

[0134] Each module in the aforementioned text data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0135] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores specialized terminology data from a predefined domain terminology set. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a text data processing method.

[0136] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0138] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above method embodiments.

[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing text data, characterized in that, The method includes: The text to be processed in the target domain is segmented to obtain multiple text blocks to be processed. The professional terms in the preset domain term set are matched with each of the text blocks to be processed to determine the number of times the professional terms appear in each of the text blocks to be processed. The professional terms include weighted professional terms and new professional terms. The weighted professional terms are determined according to the weight of the sample word segmentation in the sample text under the target domain. The new professional terms are generated according to the sample word segmentation. In response to the judgment result that the number of times the professional term appears meets the preset condition, the text block to be processed is taken as the target text block; Generate triplet relations for the target domain based on the target text block; The method for generating the preset domain term set includes: The sample text is segmented to obtain multiple sample words in the sample text; In response to the judgment result that the weight of the sample word segmentation meets the first preset requirement, the sample word segmentation is used as the weighted professional word; Multiple sample word segments are combined to obtain multiple sample new words, and the professional new words are determined from the multiple sample new words; The preset domain term set is generated using the weighted professional terms and the new professional terms.

2. The method according to claim 1, characterized in that, Before determining that the number of occurrences of the specialized term meets a preset condition and designating the text block to be processed as the target text block, the method further includes: Obtain the text length of each of the text blocks to be processed; The frequency of the professional term in each text block to be processed is determined based on the text length of each text block to be processed and the number of times the professional term appears. If the frequency is greater than a preset threshold, a judgment result is generated indicating that the number of times the professional term appears meets the preset condition.

3. The method according to claim 1, characterized in that, The step of generating the preset domain term set using the weighted professional terms and the new professional terms includes: Based on the attributes of the weighted professional terms and the new professional terms, the weighted professional terms and the new professional terms are filtered to obtain the filtered target professional terms; The target professional terms are used to generate the preset domain term set.

4. The method according to claim 1, characterized in that, Before determining that the weight of the sample word segmentation meets the first preset requirement and using the sample word segmentation as the weighted professional term, the method further includes: Obtain the first occurrence frequency of each of the sample word segments in the sample text; The weight of each sample word is determined based on its first occurrence frequency.

5. The method according to claim 1, characterized in that, The process of combining multiple sample word segments to obtain multiple new sample words, and determining the specialized new words from the multiple new sample words, includes: Each of the sample words is segmented to obtain word fragments corresponding to each sample word; Based on the position information of the word segmentation fragments, the word segmentation fragments corresponding to adjacent sample word segments are combined to generate multiple new sample words; An evaluation parameter for each new word is generated based on its second frequency of occurrence in the sample text. In response to the judgment result that the evaluation parameters meet the second preset requirements, the sample new words are taken as the professional new words.

6. A text data processing apparatus, characterized in that, The device includes: The text segmentation module is used to segment the text to be processed in the target domain, resulting in multiple text blocks to be processed. The frequency determination module is used to match the professional words in the preset domain word set with each of the text blocks to be processed, and determine the frequency of the professional words in each text block to be processed. The professional words include weighted professional words and new professional words. The weighted professional words are determined according to the weight of the sample word segmentation in the sample text under the target domain, and the new professional words are generated according to the sample word segmentation. The result response module is used to respond to the judgment result that the number of times the professional term appears meets the preset conditions, and to take the text block to be processed as the target text block. The relation generation module is used to generate triple relations of the target domain based on the target text block; The device further includes a word set generation module, which includes: The word segmentation processing unit performs word segmentation processing on the sample text to obtain multiple sample words in the sample text; The weight judgment unit is used to determine the weight of the sample word as the weighted professional word in response to the judgment result that the weight of the sample word meets the first preset requirement. The new word generation unit is used to combine multiple sample word segments to obtain multiple sample new words, and to determine the professional new word from the multiple sample new words; The word set generation unit uses the weighted professional words and the new professional words to generate the preset domain word set.

7. The apparatus according to claim 6, characterized in that, The result response module further includes: The length acquisition unit is used to acquire the text length of each of the text blocks to be processed; A frequency determination unit is used to determine the frequency of the professional term in each text block to be processed based on the text length of each text block to be processed and the number of times the professional term appears. The result generation unit is used to generate a judgment result that the number of times the professional term appears meets the preset condition when the frequency is greater than a preset threshold.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images