Text processing method, corpus expansion method, electronic equipment and storage medium

Abstract

The invention discloses a text processing method, a corpus expansion method, electronic equipment and a storage medium, and relates to the field of large model technologies and text processing. The method comprises the steps that an original text set and a knowledge graph of a target field are obtained, the original text set comprises original texts of multiple fields, and the multiple fields comprise the target field; matching a plurality of entities contained in the knowledge graph of the target domain with original texts in the original text set, and determining relevancy between the original texts and the target domain; and based on the relevancy between the original text and the target field, screening the original text to obtain a target text of the target field. According to the method and the device, the technical problem that the processing capacity of a machine learning model trained by utilizing the training corpus in the specified field is affected due to the fact that the quality of the training corpus in the specified field is relatively poor and the relevancy with the specified field is relatively low in the related technology is solved.

Description

Technical Field

[0001] This application relates to large model technology and the field of text processing, and more specifically, to a text processing method, a corpus expansion method, an electronic device, and a storage medium. Background Technology

[0002] In recent years, among the various branches of artificial intelligence, the progress in natural language processing has been particularly noteworthy. With the advent of various large-scale natural language models, machines' ability to understand and generate human language has reached a high level. However, although these large models perform well in many general tasks, in certain specific domains, such as medicine and science, where high-quality training data is lacking, the large-scale natural language models trained often exhibit poor processing capabilities in these specific domains due to the poor quality of the training data or the low relevance of the training data to these specific domains. Summary of the Invention

[0003] This application provides a text processing method, a corpus expansion method, an electronic device, and a storage medium to at least solve the technical problem in the related art where the quality of training corpus in a specified domain is poor and its relevance to the specified domain is low, which affects the processing capability of the machine learning model trained using the training corpus in the specified domain.

[0004] According to one aspect of the embodiments of this application, a text processing method is provided, comprising: acquiring an original text set and a knowledge graph of a target domain, wherein the original text set contains original text from multiple domains, and the multiple domains contain a target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and filtering the original text based on the relevance between the original text and the target domain to obtain target text of the target domain.

[0005] According to another aspect of the embodiments of this application, a corpus expansion method is also provided, comprising: acquiring original corpus, a knowledge graph of a target domain, and initial corpus of the target domain, wherein the original corpus contains corpus of multiple domains, and the multiple domains include the target domain; matching multiple entities contained in the knowledge graph of the target domain with the corpus in the original corpus to determine the relevance between the corpus and the target domain; filtering the corpus based on the relevance between the corpus and the target domain to obtain expanded corpus of the target domain; and summarizing the initial corpus and the expanded corpus to obtain training corpus of the target domain, wherein the training corpus is used to train a machine learning model.

[0006] According to another aspect of the embodiments of this application, a text processing method is also provided, comprising: responding to an input command applied to an operation interface, displaying an original text set and a knowledge graph of a target domain on the operation interface, wherein the original text set contains original text from multiple domains, and the multiple domains contain a target domain; responding to a text processing command applied to the operation interface, displaying target text of the target domain in the original text set on the operation interface, wherein the target text is text obtained by filtering the original text based on the relevance between the original text in the original text set and the target domain, and the relevance between the original text and the target domain is the result of matching multiple entities contained in the knowledge graph of the target domain with the original text.

[0007] According to another aspect of the embodiments of this application, a text processing method is also provided, comprising: obtaining an original text set and a knowledge graph of a target domain by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter including the original text set and the knowledge graph of the target domain, the original text set containing original text of multiple domains, and the multiple domains including the target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; filtering the original text based on the relevance between the original text and the target domain to obtain target text of the target domain; and outputting the target text by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter including the target text.

[0008] According to one aspect of the embodiments of this application, a text processing apparatus is also provided, comprising: a text acquisition module, configured to acquire an original text set and a knowledge graph of a target domain, wherein the original text set contains original text from multiple domains, and the multiple domains contain the target domain; a text matching module, configured to match multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and a text filtering module, configured to filter the original text based on the relevance between the original text and the target domain to obtain target text of the target domain.

[0009] According to another aspect of the embodiments of this application, a corpus expansion device is also provided, comprising: a corpus acquisition module, configured to acquire original corpus, a knowledge graph of a target domain, and initial corpus of the target domain, wherein the original corpus contains corpus of multiple domains, and the multiple domains include the target domain; a corpus matching module, configured to match multiple entities contained in the knowledge graph of the target domain with the corpus in the original corpus to determine the relevance between the corpus and the target domain; a corpus filtering module, configured to filter the corpus based on the relevance between the corpus and the target domain to obtain expanded corpus of the target domain; and a corpus aggregation module, configured to aggregate the initial corpus and the expanded corpus to obtain training corpus of the target domain, wherein the training corpus is used to train a machine learning model.

[0010] According to another aspect of the embodiments of this application, a text processing apparatus is also provided, comprising: a first display module, configured to respond to an input command applied to an operation interface and display an original text set and a knowledge graph of a target domain on the operation interface, wherein the original text set contains original text from multiple domains, and the multiple domains contain a target domain; and a second display module, configured to respond to a text processing command applied to the operation interface and display target text of the target domain in the original text set on the operation interface, wherein the target text is text obtained by filtering the original text based on the relevance between the original text in the original text set and the target domain, and the relevance between the original text and the target domain is the result of matching multiple entities contained in the knowledge graph of the target domain with the original text.

[0011] According to another aspect of the embodiments of this application, a text processing apparatus is also provided, comprising: a set acquisition module, configured to acquire an original text set and a knowledge graph of a target domain by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter including the original text set and the knowledge graph of the target domain, the original text set containing original text of multiple domains, and the multiple domains including the target domain; a relevance determination module, configured to match multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; a text determination module, configured to filter the original text based on the relevance between the original text and the target domain to obtain target text of the target domain; and a text output module, configured to output the target text by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter including the target text.

[0012] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0017] In this embodiment, the method involves acquiring an original text set and a knowledge graph of the target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and filtering the original text based on the relevance between the original text and the target domain to obtain the target text of the target domain. By matching multiple entities contained in the knowledge graph of the target domain with the original text, the accuracy of the determined relevance between the original text and the target domain can be guaranteed. This allows the text processing system to reasonably and accurately filter the target text that conforms to the target domain from the original text based on the relevance between the two, thereby improving the processing capability of the machine learning model trained using the target text as training corpus in the target domain. This solves the technical problem in related technologies where the quality of training corpus obtained by expanding to a specified domain is poor and its relevance to the specified domain is low, which affects the processing capability of the machine learning model trained using the training corpus in the specified domain.

[0018] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1This is a schematic diagram illustrating an application scenario of a text processing method according to this application;

[0021] Figure 2 This is a flowchart illustrating a text processing method according to this application;

[0022] Figure 3 This is a flowchart illustrating a corpus expansion method according to this application;

[0023] Figure 4 This is a schematic diagram illustrating a preliminary corpus expansion process according to this application;

[0024] Figure 5 This is a schematic diagram illustrating a raw corpus screening process according to this application;

[0025] Figure 6 This is a flowchart illustrating another text processing method according to this application;

[0026] Figure 7 This is a flowchart illustrating another text processing method according to this application;

[0027] Figure 8 This is a structural block diagram of a text processing apparatus according to this application;

[0028] Figure 9 This is a structural block diagram of a corpus expansion device according to this application;

[0029] Figure 10 This is a structural block diagram of another text processing apparatus according to this application;

[0030] Figure 11 This is a structural block diagram of another text processing apparatus according to this application;

[0031] Figure 12 This is a structural block diagram of an electronic device according to this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0035] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to various tasks. For example, large models can be widely applied in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be used in computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as in NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the example of using a relevance model to filter target text matching the target domain from unlabeled original text in a text processing scenario, and then using this target text as training data to train a machine learning model's processing capabilities in the target domain, serves as an illustration.

[0036] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0037] Knowledge Graph: A knowledge graph is a technology for organizing and representing knowledge. Its core is to use a graph data structure to represent and store information and knowledge. Nodes represent entities or concepts, and edges are composed of attributes or relationships.

[0038] Deep Learning: Deep learning refers to a collection of algorithms that use various machine learning algorithms on multi-layer neural networks to solve various problems such as images and text. It can be classified as a neural network. Its core is feature learning, which aims to obtain hierarchical feature information through layered networks, thereby solving the important problem that previously required manual feature design.

[0039] Large Language Models (MLMs), often shortened to large models, are deep neural network models with a large number of parameters. Due to their massive scale, large MLMs can capture and learn complex data patterns, performing exceptionally well in various tasks such as language translation, text generation, question answering, and text summarization. Large MLMs typically require massive amounts of training data and significant computational resources to train, enabling them to exhibit low generalization error when handling large-scale information processing. Therefore, the quality, diversity, and scale of the data directly impact the performance of large MLMs.

[0040] Corpus expansion is a technique in natural language processing that aims to artificially increase the amount of data in the training set through various methods. In deep learning and data-driven model building, corpus expansion can improve model generalization ability, reduce the risk of overfitting, and enhance the model's predictive ability for rare samples.

[0041] According to an embodiment of this application, a text processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] Figure 1 This is a schematic diagram illustrating an application scenario of a text processing method according to this application. Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the text processing method provided in this application embodiment can be applied to, for example... Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0043] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device performs the steps of acquiring the original text set and the knowledge graph of the target domain, displaying the original text set, and displaying the target text; the server performs the steps of acquiring multiple entities contained in the knowledge graph of the target domain, matching the multiple entities with the original text contained in the original text set, determining the relevance between different original texts and the target domain, and filtering the different original texts according to the relevance to obtain the target text. It should be noted that, provided the client device's operating resources can meet the deployment and operation conditions of a large model, this embodiment can be performed on the client device.

[0044] Under the aforementioned operating environment, this application provides the following: Figure 2 The text processing method shown. Figure 2 This is a flowchart illustrating a text processing method according to this application. Figure 2 As shown, the method may include the following steps:

[0045] Step S202: Obtain the original text set and the knowledge graph of the target domain.

[0046] The original text set contains original text from multiple domains, and the multiple domains contain the target domain.

[0047] The aforementioned original text may refer to text without domain labeling. The aforementioned target domain may refer to the domain of the target text used in certain specified text processing scenarios. This domain can be specified by the user or obtained by the text processing system through analysis of the text processing scenario; no limitation is made here. These text processing scenarios may include, but are not limited to: dataset construction scenarios, knowledge graph construction scenarios, topic trend analysis scenarios, risk management scenarios, and model training scenarios. In these scenarios, it is usually necessary to use text from the target domain that matches the scenario to perform the corresponding text processing tasks. For example, in a topic trend analysis scenario, it is usually necessary to obtain text data matching the topic and analyze the hot issues and trend changes contained in the topic to help users understand the topic. In a model training scenario, it is usually necessary to obtain training corpus from the same domain as the machine learning model to train the machine learning model and improve its processing capabilities in that domain. The aforementioned knowledge graph can refer to a knowledge graph constructed for text in a target domain, used to represent text entities, relationships, and attributes. It can be obtained directly from the knowledge graph database by the text processing system based on the domain identifier of the target domain, or it can be constructed by the text processing system based on the entity, relationship, attribute, and other information contained in the existing text belonging to the target domain. The specific acquisition method can be determined by the user and is not limited here.

[0048] In one optional embodiment, considering that in some text processing scenarios, the lack of basic data may lead to poor results for the corresponding text processing tasks, such as in model training scenarios, in fields lacking high-quality and highly relevant training data, the training data for machine learning models may be insufficient, resulting in poor processing capabilities of the trained models in that target domain. Therefore, in order to improve the performance of text processing tasks in some text processing scenarios, the text processing system can optimize the basic text corresponding to these scenarios. For example, the text in the target domain of the text processing scenario can be expanded to obtain higher-quality, more numerous, and more relevant target text. This ensures that the text processing system has sufficient text data support when performing corresponding text processing tasks in the text processing scenario, thereby improving the performance of the corresponding text processing tasks. For example, it can improve the processing capability of machine learning models in the target domain and increase the accuracy of analyzing topic change trends.

[0049] To ensure the authenticity of text augmentation in the target domain, the text processing system can directly select the missing text from the already labeled text belonging to the target domain to augment the target text. Alternatively, it can identify and label the domain of real text that is not labeled, thereby selecting target text that can augment the text in the target domain. This improves the effectiveness of text processing tasks performed using the augmented target text in text processing scenarios, such as improving the processing power of machine learning models and increasing the accuracy of topic trend analysis results.

[0050] Considering that labeled text belonging to the target domain can be directly used to perform text processing tasks in the corresponding text processing scenario, while unlabeled text may belong to a domain that does not match the target domain, leading to incorrect execution results when using these texts to perform text processing tasks, when expanding the text in the target domain, we can first obtain a set of original texts containing multiple domains to ensure the quantity of original texts, thereby indirectly ensuring the quantity of target texts used for expansion. Furthermore, these multiple domains can include target domains related to the text processing scenario, so that the text processing system can identify and label the domain to which the original text belongs, and select target texts that match the target domain from the original text for expansion, thus avoiding expansion errors. Meanwhile, considering that a text can contain a large number of entities, and different entities may have different relationships and attributes in different scenarios, a text may belong to multiple domains at the same time. In the knowledge graph that matches the target domain, the entities, relationships, attributes and other information of the text that belongs to the target domain can be clearly represented. Therefore, in order to accurately filter out the target text that matches the target domain from the original text set, in addition to obtaining the original text set mentioned above, the text processing system can also obtain the knowledge graph that matches the target domain, so as to use the knowledge graph to determine the relationships and attributes of different entities contained in the original text in the target domain.

[0051] Step S204: Match the multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain.

[0052] In one optional embodiment, after obtaining the original text set and the knowledge graph of the target domain, the text processing system can determine the structured knowledge related to the target domain from the knowledge graph, that is, determine the multiple entities contained in the knowledge graph. Then, these multiple entities are matched with the original text in the original text set to determine the degree of matching between different original texts and the knowledge graph, thereby determining the relevance of the original text to the target domain. This allows the text processing system to accurately filter target text belonging to the target domain from the original text based on this relevance. For example, the text processing system can match the entities contained in the original text with the entities contained in the knowledge graph to determine one or more identical target entities in both sets of text, as well as the number of entities containing the target entity in the original text. Then, based on the number of entities, it determines the evaluation score corresponding to the target entity. Finally, it performs weighted processing on the evaluation scores corresponding to different target entities to determine the relevance of the original text to the target domain.

[0053] Step S206: Based on the relevance between the original text and the target domain, the original text is filtered to obtain the target text for the target domain.

[0054] In one optional embodiment, considering that identifying the original text with low relevance as the target text might significantly impact the execution result of the text processing task, leading to low accuracy, the text processing system can first determine whether different original texts belong to the target domain based on their relevance when filtering the original texts. For example, the relevance can be compared with a preset relevance threshold to initially filter out initial texts belonging to the target domain from the original texts. Then, based on the relevance of different initial texts, the target text can be filtered out from the initial texts to avoid errors in selecting target texts. Correspondingly, when filtering target texts, different initial texts can be sorted according to their relevance, and then the target texts can be determined according to a preset filtering method, such as filtering by proportion or by quantity. The specific filtering method can be determined by the user and is not limited here.

[0055] In one optional embodiment, after the target text in the target domain is obtained through filtering, the target text can be displayed to the user in a preset operation interface, and the user can determine whether the target text needs to be filtered again, and whether the target text can be applied to the text processing task, so as to ensure the rationality of the obtained target text and the effect of the text processing task performed using the target text. For example, in the model training scenario, after obtaining user confirmation, the text processing system can apply the target text as training corpus to the training of the machine learning model, so as to improve the processing ability of the trained machine learning model in the target domain.

[0056] In this embodiment, the method involves acquiring an original text set and a knowledge graph of the target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and filtering the original text based on the relevance between the original text and the target domain to obtain the target text of the target domain. By matching multiple entities contained in the knowledge graph of the target domain with the original text, the accuracy of the determined relevance between the original text and the target domain can be guaranteed. This allows the text processing system to reasonably and accurately filter the target text that conforms to the target domain from the original text based on the relevance between the two, thereby improving the processing capability of the machine learning model trained using the target text as training corpus in the target domain. This solves the technical problem in related technologies where the quality of training corpus in a specified domain is poor and the relevance to the specified domain is low, which affects the processing capability of the machine learning model trained using the training corpus in that specified domain.

[0057] In this embodiment of the application, multiple entities contained in the knowledge graph of the target domain are matched with the original text in the original text set to determine the relevance between the original text and the target domain. This includes: matching multiple entities with preset text in the target domain to determine the entity weights corresponding to the multiple entities, wherein the entity weight corresponding to any entity is used to characterize the relevance between the entity and the target domain; matching multiple entities with the original text to determine at least one target entity contained in the original text; and summarizing the entity weights corresponding to the original text containing at least one target entity to obtain the relevance between the original text and the target domain.

[0058] The aforementioned preset text can refer to text that has already been labeled and belongs to the target domain.

[0059] In one optional embodiment, considering that the relevance between entities and the target domain cannot be intuitively displayed in a knowledge graph, directly matching the entities contained in the knowledge graph with the original text may result in errors in the determined relevance between the original text and the target domain, leading to incorrect target text. Therefore, when determining the relevance between the original text and the target domain, the text processing system can first determine the relevance between different entities contained in the knowledge graph and the target domain, so as to accurately determine the entity weights corresponding to different entities based on the relevance. Considering that in the labeled text belonging to the target domain, the number, relationship, attributes, and other information of entities can accurately reflect the relevance between the entity and the target domain, the text processing system can match multiple entities contained in the knowledge graph with preset text belonging to the target domain to determine the entity weights corresponding to different entities in the knowledge graph. For example, assuming that the knowledge graph contains entities A and B, the text processing system can determine the number A and the number B of entities A and B appearing in the preset text, respectively, and then determine the entity weights corresponding to entities A and B respectively based on the ratio of the number A to the number B.

[0060] After determining the entity weights corresponding to multiple entities, the text processing system can match the multiple entities with the original text to identify at least one target entity currently contained in the original text. Then, by summarizing the entity weights corresponding to the original text containing the target entity, the relevance of the original text to the target domain can be determined. For example, suppose we have an original text A, and the knowledge graph contains entities A and B, with entity weight 'a' for entity A and entity weight 'b' for entity B. If there are k entities in the original text A, and the target entity is entity A, with entity A appearing n1 times in the original text, then the relevance between the original text A and the target domain can be determined as n1*a / k. If the target entities in the original text A are entities A and B, with entity A appearing n2 times and entity B appearing m times, then the relevance between the original text A and the target domain can be determined as n1*a / k + m*b / k. Alternatively, we can directly determine n1*a + m*b as the relevance between the original text A and the target domain, without considering the total number of entities k in the original text A. This avoids misidentifying a large number of entities unrelated to the target domain from the original text A, leading to a low relevance determination, and improves the efficiency of determining this relevance. It should be noted that the formula for determining relevance here is only an example and can be set by the user. No restrictions are imposed here.

[0061] In this embodiment of the application, matching multiple entities with preset text in the target domain and determining the entity weights corresponding to the multiple entities includes: matching multiple entities with preset text and determining the distribution results of the multiple entities in the preset text, wherein the distribution result of any entity is used to characterize the position and quantity of the entity in the preset text; and determining the entity weights corresponding to the multiple entities based on the distribution results of the multiple entities in the preset text.

[0062] In one optional embodiment, considering that entity A may contain entity B in the text, such as the word "education" containing the word "cation," which could lead to errors in the entity weights determined based on the number of entities, the text processing system can first match multiple entities in the knowledge graph with the preset text to determine the specific results of any entity in the preset text, such as determining the position and number of entities. Then, it performs a completeness verification on entities at different positions to avoid entity recognition errors. Based on the verification results, the text processing system can determine the complete entity from multiple entities. Based on the number and position of the complete entity in the preset text, it can determine the entity weight corresponding to that entity. Entity matching algorithms, such as the Aho-Corasick algorithm, Rabin-Karp algorithm, and other string search algorithms, can be used to determine the number and position of different entities. Algorithms such as BM25 (Best Matching 25 algorithm, an information retrieval ranking algorithm) and TF-IDF (Term Frequency-Inverse) can also be used. Document Frequency (a retrieval ranking algorithm based on term frequency and inverse document frequency) and other retrieval ranking algorithms determine the entity weight corresponding to the entity based on the number and position of the above entities.

[0063] In this embodiment of the application, the target domain includes multiple disciplines; determining the entity weights corresponding to multiple entities based on the distribution results of multiple entities in a preset text includes: determining the initial weights corresponding to multiple entities based on the distribution results of multiple entities in the preset text; determining the coverage corresponding to multiple disciplines based on the discipline types to which multiple entities belong, wherein the coverage corresponding to any discipline is used to characterize the proportion of entities belonging to that discipline in the preset text; adjusting the initial weights corresponding to multiple entities based on the coverage corresponding to multiple disciplines to obtain the entity weights corresponding to multiple entities, wherein the entity weight corresponding to any entity is inversely proportional to the coverage corresponding to the discipline to which the entity belongs.

[0064] The aforementioned disciplines can be used to represent the category to which an entity belongs.

[0065] In one optional solution of this embodiment, considering that in actual text processing scenarios, if there is a small amount of text corresponding to a certain category of entities, it may be impossible to accurately perform text processing tasks based on the text of that category. For example, suppose the current knowledge graph contains entity A, and the corresponding target domain is domain A. During model training, if the number of entities corresponding to subject A in the original text is small, while the number of entities corresponding to other subjects, such as subject B, is large, then when actually allocating entity weights, entities corresponding to subject A are often assigned smaller weights, while entities corresponding to subject B are assigned larger weights. In this case, the text processing system may directly classify the original text into the subject corresponding to entity B based on the weight difference between the two. However, the text contained in the subject corresponding to entity A is still small, so even if the original text contains relevant content of subject A corresponding to entity A, making the original text usable in the training process of the machine learning model, it will not be used in the actual model training process, thus failing to improve the processing capability of the machine learning model under subject A. Therefore, to improve the ability to perform corresponding text processing tasks in text processing scenarios, when determining the entity weights based on the respective results of multiple entities in the preset text, the text processing system can first determine the weights corresponding to each entity based on the distribution results of multiple entities in the preset text, i.e., the quantity and position of different entities in the preset text. This initial weight can be determined using the aforementioned BM25 algorithm. Simultaneously with determining the initial weights, the subject types of different entities in the preset text can be identified to determine the proportion of entities belonging to different subjects in the preset text, thus determining the coverage of each subject. For example, assuming there are subjects A and B, with j entities corresponding to subject A and k entities corresponding to subject B, the coverage of subject A can be determined as j / (j+k), and the coverage of subject B as k / (j+k). Then, the determined initial weights are adjusted using the determined coverage to obtain entity weights that increase the importance of text corresponding to scarce subjects in text processing tasks, thereby improving the ability to perform text processing tasks on text corresponding to scarce subjects. In order to effectively increase the importance of texts corresponding to scarce disciplines in text processing tasks, the inverse of the determined coverage can be multiplied by the initial weights of the entities contained in the disciplines corresponding to different coverages to obtain the aforementioned entity weights. Alternatively, the coverage corresponding to different disciplines can be directly determined as the entity weights of the entities contained in that discipline.

[0066] In this embodiment of the application, the original text is filtered based on the relevance between the original text and the target domain to obtain target text in the target domain. This includes: filtering the original text based on the relevance between the original text and the target domain to obtain candidate text in the target domain; filtering the candidate text to obtain filtered text, wherein the filtered text is used to represent text in the candidate text that belongs to the target domain; and filtering the filtered text to obtain target text, wherein the target text is used to represent text in the filtered text that has a quality score greater than a preset score.

[0067] In one optional solution of this embodiment, considering that although the text directly filtered based on relevance contains relevant entities in the target domain, the professional level of these entities is unknown. If the professional level is low, the text may only have some words related to the target domain, while the text itself may belong to other domains. Correspondingly, there may be situations where the text processing task cannot be accurately performed based on these filtered texts. Therefore, in order to ensure that the text processing system can effectively perform text processing tasks based on the filtered target text, when filtering target text based on the relevance between the original text and the target domain, the text processing system can first preliminarily filter out texts with a relevance greater than a preset value from the original text based on the relevance between the two, and use them as candidate texts for the target domain, and remove other texts that are obviously unrelated to the target domain. The candidate texts selected as the target domain have a certain correlation with the target domain, but the candidate texts may not actually belong to the target domain. After selecting candidate texts, the text processing system can perform a preliminary filtering to remove texts that are irrelevant to the target domain. For example, it can use a pre-trained model for filtering candidate texts and continue filtering based on the relevance mentioned above to determine the texts that have a high probability of belonging to the target domain. These are the filtered texts. Finally, the filtered texts undergo a second filtering process. For example, a text evaluation model can be used to determine whether there are high-quality, highly specialized entities among the entities contained in the filtered texts. This allows the filtered texts to be scored, and texts with quality scores greater than a preset score are selected from the filtered texts based on the scores. Alternatively, the filtered texts can be sorted by quality score and then selected according to a preset number and proportion. These selected texts are then used as the target texts, thus ensuring the quality and specialization of the determined target texts.

[0068] In this embodiment of the application, filtering candidate text to obtain filtered text includes: extracting features from candidate text to obtain first text features of candidate text; inputting the first text features into a relevance prediction model to obtain a prediction result output by the relevance prediction model, wherein the prediction result is used to characterize the probability that candidate text belongs to the target domain; and filtering candidate text based on the prediction result to obtain filtered text.

[0069] The aforementioned relevance prediction model can refer to a model used to filter out texts from candidate texts that have a high probability of belonging to the target domain. It may include, but is not limited to, models such as SVR (Support Vector Regression) regression model, linear regression model, and ridge regression model.

[0070] In one optional embodiment, in order to effectively filter out text belonging to the target domain from candidate texts, the text processing system can first extract features from the candidate texts to determine the first text features of the candidate texts, such as semantic features of keywords explicitly marked in the text, sentences containing entities, background information, etc. Then, the extracted first text features are input into a relevance prediction model to determine the probability that different candidate texts belong to the target domain. For example, the SVR regression model mentioned above can be used to perform regression calculations on the first text features to determine the probability that different candidate texts belong to the target domain, thereby obtaining the output of the relevance prediction model. Finally, based on the prediction results, the text processing system can filter the candidate texts to obtain filtered texts that are highly likely to belong to the target domain.

[0071] In this embodiment of the application, filtering the filtered text to obtain the target text includes: extracting features from the filtered text to obtain a second text feature of the filtered text; inputting the second text feature into a quality assessment model to obtain an assessment result output by the quality assessment model, wherein the assessment result is used to characterize the quality score of the filtered text; and filtering the filtered text based on the assessment result of the filtered text to obtain the target text.

[0072] In one optional embodiment, to filter out high-quality, highly specialized target text, the text processing system can further filter the filtered text, using the original text corresponding to the filtered text as a second text feature. This feature is then input into a pre-trained quality assessment model (large language model) to evaluate the quality of different filtered texts, obtaining quality scores corresponding to different filtered texts, i.e., the aforementioned evaluation results. For example, the higher the degree of semantic specialization of entities in the target domain contained in the text, the greater the number of entities, and the more evenly they appear, the higher the quality score of the text. After determining the quality scores corresponding to different filtered texts, the text processing system can filter out the aforementioned target text from the filtered text according to the text requirements corresponding to the text processing task, based on the evaluation results. The text requirements can be determined by the user and are not limited here.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0074] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0076] According to an embodiment of this application, a corpus expansion method is also provided. Figure 3 This is a flowchart illustrating a corpus expansion method according to this application, such as... Figure 3 As shown, the method may include the following steps:

[0077] Step S302: Obtain the original corpus, the knowledge graph of the target domain, and the initial corpus of the target domain.

[0078] The original corpus contains corpora from multiple domains, and these multiple domains include the target domain.

[0079] In one optional embodiment, to improve the processing capability of the machine learning model in the target domain, the corpus expansion system can expand the corpus used to train the machine learning model in the target domain. To ensure the authenticity of the expanded training corpus, the corpus expansion system can acquire multiple unlabeled original corpora and the initial corpus prepared in the target domain for training the machine learning model. In addition, to accurately select target texts that match the target domain from the original text set, the corpus expansion system can also acquire a knowledge graph that matches the target domain in addition to acquiring the original corpus. The knowledge graph can be used to determine the relationships and attributes of the initial corpus in the target domain. The original corpus can contain corpora from multiple domains, including the target domain.

[0080] Step S304: Match the multiple entities contained in the knowledge graph of the target domain with the corpus in the original corpus to determine the relevance between the corpus and the target domain.

[0081] In one optional embodiment, in order to accurately select extended corpora that conform to the target domain from the original corpus, the corpus expansion system can first acquire a knowledge graph that matches the target domain, and determine the structured knowledge related to the target domain from the knowledge graph, that is, determine the multiple entities contained in the knowledge graph. Then, these multiple entities are matched with the corpora in the original corpus to determine the degree of matching between different corpora and the knowledge graph, thereby determining the relevance of different corpora to the target domain. This allows the corpus expansion system to accurately select extended corpora belonging to the target domain from the original corpus based on the relevance.

[0082] Step S306: Based on the relevance between the corpus and the target domain, the corpus is filtered to obtain an expanded corpus for the target domain.

[0083] In one optional embodiment, considering that determining the original corpus with lower relevance as the expanded corpus might still not improve the processing capability of the machine learning model trained using the expanded corpus in the target domain, the corpus expansion system can first determine whether the corpus belongs to the target domain based on the relevance between the corpus and the target domain when filtering the original corpus. For example, the relevance can be compared with a preset relevance threshold to initially filter out corpus belonging to the target domain from the original corpus. Then, based on the relevance of the initially filtered corpus, the expanded corpus can be selected from the initially filtered corpus to avoid errors in selecting the expanded corpus. Correspondingly, when selecting the expanded corpus, different initial corpus can be sorted according to relevance, and then the expanded corpus can be determined according to a preset filtering method, such as filtering by proportion or by quantity. The specific filtering method can be determined by the user and is not limited here. After the expanded corpus is determined, the text processing system can also display the expanded corpus to the user in a preset operation interface for the user to view and confirm.

[0084] Step S308: Summarize the initial corpus and the expanded corpus to obtain the training corpus for the target domain.

[0085] The training corpus is used to train the machine learning model.

[0086] The aforementioned machine learning models may include, but are not limited to, deep learning models, large language models, and other similar models.

[0087] In one optional embodiment, after determining the expanded corpus, the text processing system can summarize the initial corpus and the expanded corpus to obtain a high-quality and large-quantity training corpus, and use the training corpus to train the machine learning model, thereby improving the processing capability of the machine learning model in the target domain.

[0088] In this embodiment of the application, multiple entities contained in the knowledge graph of the target domain are matched with the corpus in the original corpus to determine the relevance between the corpus and the target domain. This includes: matching multiple entities with the initial corpus to determine the entity weights corresponding to the multiple entities, wherein the entity weight corresponding to any entity is used to characterize the relevance between the entity and the target domain; matching multiple entities with the corpus to determine at least one target entity contained in the corpus; and summarizing the entity weights corresponding to the original text containing at least one target entity to obtain the relevance between the corpus and the target domain.

[0089] In one optional embodiment, considering that the relevance between different entities and the target domain cannot be intuitively displayed in the knowledge graph, directly matching the entities contained in the knowledge graph with the original corpus may result in errors in the determined relevance between the original corpus and the target domain, leading to incorrect expanded corpus. Therefore, when determining the relevance between different original corpora and the target domain, the corpus expansion system can first determine the relevance between different entities contained in the knowledge graph and the target domain, so as to accurately determine the entity weights corresponding to different entities based on the relevance. At the same time, multiple entities are matched with different original corpora to determine at least one target entity contained in different corpora. Finally, the entity weights corresponding to the original text containing at least one target entity are summarized to determine the relevance of different original corpora to the target domain. In particular, considering that the quantity, relationship, attributes, and other information of different entities in the labeled corpus belonging to the target domain can accurately reflect the relevance of the entity to the target domain, the corpus expansion system can match multiple entities contained in the knowledge graph with the initial corpus belonging to the target domain to determine the entity weights corresponding to different entities in the knowledge graph. For example, assuming that the knowledge graph contains entities A and B, the corpus expansion system can determine the number of times entities A and B appear in the initial corpus, A and B, respectively, and then determine the entity weights corresponding to entities A and B respectively based on the ratio of the number of A to the number of B. Correspondingly, when determining relevance based on the entity weights of target entities, assuming there is an original corpus A, and the knowledge graph contains entities A and B, with entity weight 'a' for entity A and entity weight 'b' for entity B, if there are k entities in the original corpus A, and the target entity is entity A, and the number of instances of entity A in the original corpus is n1, then the relevance between the original corpus A and the target domain can be determined as n1*a / k. If the target entities in the original corpus A are entities A and B, with n2 instances of entity A and m instances of entity B, then the relevance between the original corpus A and the target domain can be determined as n1*a / k + m*b / k. It should be noted that the formula for determining relevance here is only an example; the specific formula can be set by the user and is not limited here.

[0090] In this embodiment of the application, the corpus is screened based on the relevance between the corpus and the target domain to obtain expanded corpus of the target domain, including: screening the corpus based on the relevance between the corpus and the target domain to obtain candidate corpus of the target domain; filtering the candidate corpus to obtain filtered corpus, wherein the filtered corpus is used to represent corpus belonging to the target domain among the candidate corpus; and filtering the filtered corpus to obtain expanded corpus, wherein the expanded corpus is used to represent corpus with a quality score greater than a preset score among the filtered corpus.

[0091] In one optional embodiment, considering that although the corpus selected directly based on relevance contains relevant entities in the target domain, the level of expertise of these entities is unknown. If the level of expertise is low, the corpus may only contain some words related to the target domain, while the corpus itself may belong to other domains. Consequently, it may be impossible to accurately train a machine learning model based on these selected corpora. Therefore, in order to ensure that the corpus expansion system can effectively improve the processing capability of the machine learning model in the target domain based on the selected target corpus, when selecting target corpus based on the relevance between the original corpus and the target domain, the corpus expansion system can first preliminarily select corpus with a relevance greater than a preset value from the original corpus as candidate corpus for the target domain, and remove other corpus that is obviously unrelated to the target domain. The selected candidate corpus for the target domain has a certain correlation with the target domain, but the candidate corpus may not actually belong to the target domain. After selecting candidate corpora, in order to remove corpora that are actually irrelevant to the target domain, the corpus expansion system can perform a preliminary filtering of the candidate corpora. For example, it can use a pre-trained model for filtering candidate corpora, or continue filtering based on the above relevance, to determine the corpora that have a high probability of belonging to the target domain, i.e., the filtered corpora mentioned above. Finally, the filtered corpora are filtered a second time. For example, a corpus evaluation model can be used to determine whether there are high-quality, highly specialized entities in the filtered corpora. This is used to score the filtered corpora, and based on the score, corpora with quality scores greater than a preset score are selected from the filtered corpora, or corpora selected according to a preset number and preset proportion after sorting the filtered corpora by quality score are used as the target corpora, thereby ensuring the quality and specialization of the determined target corpora.

[0092] To facilitate understanding of the above text processing and corpus expansion processes, let's take expanding the corpus used to train the machine learning model and improving its processing capabilities in the target domain as an example. Figure 4 This is a schematic diagram illustrating a preliminary corpus expansion process according to this application, such as... Figure 4As shown, when expanding the training corpus, we can first obtain the original corpus without domain labeling, as well as the initial corpus belonging to the target domain. Then, we match multiple entities contained in the knowledge graph corresponding to the target domain with the entities contained in the original corpus. We identify and verify the completeness of the entities contained in the original corpus to avoid entity matching errors. Based on the matching results, we can identify the target entities that are the same as the entities contained in the knowledge graph from the entities in the original corpus. Then, we match multiple entities contained in the knowledge graph with the entities contained in the initial corpus to determine the entity weights corresponding to different entities. Finally, we determine the entity weight corresponding to the target entity from the entity weights corresponding to different entities. By summarizing the entity weights corresponding to the original text containing at least one target entity, we can obtain the relevance between the original corpus and the target domain. Based on this relevance, we can filter out the expanded corpus belonging to the target domain from the original corpus.

[0093] Figure 5 This is a schematic diagram illustrating a raw corpus screening process as shown in this application, such as... Figure 5 As shown, to ensure the quality of the selected expanded corpus, we can first filter out the first corpus, which has a certain correlation with the target domain, from the original corpus based on the relevance determined above. We then extract features from the first corpus to obtain its features. These features are input into a pre-trained relevance prediction model to determine the probability that different first corpora belong to the target domain. By selecting corpora with probabilities greater than a preset value from the first corpus, we obtain the second corpus belonging to the target domain. After determining the second corpus, we can extract features again to obtain its features. Using a pre-trained quality assessment model to process these features, we can obtain quality scores for different second corpora. These quality scores reflect the quality and specialization of the second corpus. Finally, we filter the second corpus based on these quality scores to obtain high-quality expanded corpus. Using this expanded corpus and the initial corpus to train the machine learning model can effectively improve the model's processing capability in the target domain.

[0094] According to embodiments of this application, a text processing method is also provided. Figure 6 This is a flowchart illustrating another text processing method according to this application, such as... Figure 6 As shown, the method may include the following steps:

[0095] Step S602: In response to the input command applied to the operation interface, display the original text set and the knowledge graph of the target domain on the operation interface.

[0096] The original text set contains original text from multiple domains, and the multiple domains contain the target domain.

[0097] The aforementioned input instructions may refer to instructions used to obtain the original text set and the knowledge graph of the target domain.

[0098] In one optional embodiment, in order to ensure the authenticity of the determined target text, when the input instruction is received, the text processing system can first obtain the original text set containing original texts from multiple fields, as well as the knowledge graph matching the target field, and display the original text set and knowledge graph on the operation interface for the user's convenience. Specifically, it can be as shown in the aforementioned step S202, which will not be repeated here.

[0099] Step S604: In response to the text processing command applied to the operation interface, display the target text of the target domain in the original text set on the operation interface.

[0100] The target text is a text obtained by filtering the original text based on the relevance between the original text in the original text set and the target domain. The relevance between the original text and the target domain is the result of matching multiple entities contained in the knowledge graph of the target domain with the original text.

[0101] The aforementioned text processing instructions may refer to instructions used to process the original text set.

[0102] In one optional embodiment, after detecting a text processing instruction applied to the operation interface, the text processing system can filter the original text in the original text set to determine the target text belonging to the target domain, and display the target text in the operation interface for the user's convenience. Specifically, it can be as described in steps S204 and S206 above, which will not be repeated here.

[0103] According to embodiments of this application, a text processing method is also provided. Figure 7 This is a flowchart illustrating another text processing method according to this application, such as... Figure 7 As shown, the method may include the following steps:

[0104] Step S702: Obtain the original text set and the knowledge graph of the target domain by calling the first interface.

[0105] The first interface includes a first parameter, the value of which includes a set of original texts, which contains original texts from multiple domains, and the multiple domains include the target domain.

[0106] The aforementioned first interface may refer to the interface used to obtain the first parameter, namely, to obtain the original text set and the knowledge graph of the target domain.

[0107] In one optional embodiment, in order to ensure the accuracy when expanding the text in the target domain, the text processing system can first call the first interface mentioned above to obtain the corresponding first parameters, that is, obtain the original text set containing original texts from multiple domains and the knowledge graph of the target domain. Specifically, it can be as shown in the aforementioned step S202, which will not be repeated here.

[0108] Step S704: Match the multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain.

[0109] In one optional embodiment, after obtaining the original text set and the knowledge graph of the target domain, the text processing system can first match the multiple entities contained in the knowledge graph with the original text in the original text set to determine the degree of matching between the original text and the knowledge graph, thereby determining the relevance between the original text and the target domain. Specifically, it can be as shown in the aforementioned step S204, which will not be repeated here.

[0110] Step S706: Based on the relevance between the original text and the target domain, the original text is filtered to obtain the target text for the target domain.

[0111] In one optional embodiment, after determining the relevance between the original text and the target domain, the original text contained in the original text set can be filtered using the relevance to obtain the target text of the target domain. Specifically, it can be as shown in the aforementioned step S206, which will not be repeated here.

[0112] Step S708: Output the target text by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target text.

[0113] In one optional embodiment, after filtering out the target text in the target domain, the second interface described above can be called to output the second parameter, that is, to output the filtered target text. For example, the target text can be displayed to the user in a preset operation interface for the user's convenience.

[0114] According to embodiments of this application, a text processing apparatus is also provided, corresponding to the text processing method described above. Figure 8 This is a structural block diagram of a text processing apparatus according to this application, such as... Figure 8 As shown, the device may include: a text acquisition module 802, a text matching module 804, and a text filtering module 806.

[0115] The text acquisition module 802 is used to acquire the original text set and the knowledge graph of the target domain. The original text set contains original text from multiple domains, and the multiple domains contain the target domain. The text matching module 804 is used to match multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain. The text filtering module 806 is used to filter the original text based on the relevance between the original text and the target domain to obtain the target text of the target domain.

[0116] In this embodiment, the text matching module 804 includes: a first matching unit, configured to match multiple entities with preset text in the target domain and determine the entity weights corresponding to the multiple entities, wherein the entity weight corresponding to any one entity is used to characterize the relevance between the entity and the target domain; a second matching unit, configured to match multiple entities with original text and determine at least one target entity contained in the original text; and a first summarizing unit, configured to summarize the entity weights corresponding to the original text containing at least one target entity to obtain the relevance between the original text and the target domain.

[0117] In this embodiment of the application, the first matching unit is further configured to: match multiple entities with preset text, determine the distribution results of multiple entities in the preset text, wherein the distribution result of any entity is used to characterize the position and quantity of the entity in the preset text; and determine the entity weights corresponding to the multiple entities based on the distribution results of multiple entities in the preset text.

[0118] In this embodiment of the application, the target domain includes multiple disciplines, and the first matching unit is further configured to: determine the initial weights corresponding to multiple entities based on the distribution results of multiple entities in a preset text; determine the coverage corresponding to multiple disciplines based on the discipline types to which the multiple entities belong, wherein the coverage corresponding to any discipline is used to characterize the proportion of entities belonging to that discipline in the preset text; adjust the initial weights corresponding to multiple entities based on the coverage corresponding to multiple disciplines to obtain the entity weights corresponding to multiple entities, wherein the entity weight corresponding to any entity is inversely proportional to the coverage corresponding to the discipline to which the entity belongs.

[0119] In this embodiment, the text filtering module 806 includes: a text filtering unit, used to filter the original text based on the relevance between the original text and the target domain to obtain candidate texts in the target domain; a first filtering unit, used to filter the candidate texts to obtain filtered texts, wherein the filtered texts are used to characterize texts in the candidate texts that belong to the target domain; and a second filtering unit, used to filter the filtered texts to obtain target texts, wherein the target texts are used to characterize texts in the filtered texts that have a quality score greater than a preset score.

[0120] In this embodiment of the application, the first filtering unit is further configured to: extract features from the candidate text to obtain the first text features of the candidate text; input the first text features into the relevance prediction model to obtain the prediction result output by the relevance prediction model, wherein the prediction result is used to characterize the probability that the candidate text belongs to the target domain; and filter the candidate text based on the prediction result to obtain the filtered text.

[0121] In this embodiment of the application, the second filtering unit is further configured to: extract features from the filtered text to obtain second text features of the filtered text; input the second text features into a quality assessment model to obtain an assessment result output by the quality assessment model, wherein the assessment result is used to characterize the quality score of the filtered text; and filter the filtered text based on the assessment result of the filtered text to obtain the target text.

[0122] It should be noted that the text acquisition module 802, text matching module 804, and text filtering module 806 correspond to steps S202 to S206. The instances and application scenarios implemented by these three modules and their corresponding steps are the same, but not limited to the content disclosed above. It should also be noted that the modules or units described above can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of the device within the provided server 10.

[0123] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided above, but are not limited to the solutions provided above.

[0124] According to an embodiment of this application, corresponding to the above-described corpus expansion method, a corpus expansion device is also provided. Figure 9 This is a structural block diagram of a corpus expansion device shown in this application, such as... Figure 9 As shown, the device may include: a corpus acquisition module 902, a corpus matching module 904, a corpus filtering module 906, and a corpus summarization module 908.

[0125] The corpus acquisition module 902 is used to acquire the original corpus, the knowledge graph of the target domain, and the initial corpus of the target domain. The original corpus contains corpus from multiple domains, and the multiple domains contain the target domain. The corpus matching module 904 is used to match multiple entities contained in the knowledge graph of the target domain with the corpus in the original corpus to determine the relevance between the corpus and the target domain. The corpus filtering module 906 is used to filter the corpus based on the relevance between the corpus and the target domain to obtain the expanded corpus of the target domain. The corpus summarization module 908 is used to summarize the initial corpus and the expanded corpus to obtain the training corpus of the target domain. The training corpus is used to train the machine learning model.

[0126] In this embodiment, the corpus matching module 904 includes: a third matching unit, used to match multiple entities with an initial corpus and determine the entity weights corresponding to the multiple entities, wherein the entity weight corresponding to any one entity is used to characterize the relevance between the entity and the target domain; a fourth matching unit, used to match multiple entities with the corpus and determine at least one target entity contained in the corpus; and a second summarizing unit, used to summarize the entity weights corresponding to the original text containing at least one target entity to obtain the relevance between the corpus and the target domain.

[0127] In this embodiment, the corpus screening module 906 includes: a corpus screening unit, used to screen the corpus based on the relevance between the corpus and the target domain to obtain candidate corpus of the target domain; a third filtering unit, used to filter the candidate corpus to obtain filtered corpus, wherein the filtered corpus is used to represent the corpus of the candidate corpus belonging to the target domain; and a fourth filtering unit, used to filter the filtered corpus to obtain expanded corpus, wherein the expanded corpus is used to represent the corpus of the filtered corpus whose quality score is greater than a preset score.

[0128] It should be noted that the corpus acquisition module 902, corpus matching module 904, corpus filtering module 906, and corpus summarization module 908 correspond to steps S302 to S308. The four modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed above. It should also be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of the device within the provided server 10.

[0129] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided above, but are not limited to the solutions provided above.

[0130] According to embodiments of this application, a text processing apparatus is also provided, corresponding to the text processing method described above. Figure 10 This is a structural block diagram of another text processing apparatus shown in this application, such as... Figure 10 As shown, the device may include: a first display module 1002 and a second display module 1004.

[0131] The first display module 1002 is used to respond to input commands applied to the operation interface and display the original text set and the knowledge graph of the target domain on the operation interface. The original text set contains original text from multiple domains, and the multiple domains contain the target domain. The second display module 1004 is used to respond to text processing commands applied to the operation interface and display the target text of the target domain in the original text set on the operation interface. The target text is the text obtained by filtering the original text based on the relevance between the original text in the original text set and the target domain. The relevance between the original text and the target domain is the result of matching multiple entities contained in the knowledge graph of the target domain with the original text.

[0132] It should be noted that the first display module 1002 and the second display module 1004 mentioned above correspond to steps S602 to S604. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the server 10 provided above.

[0133] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided above, but are not limited to the solutions provided above.

[0134] According to embodiments of this application, a text processing apparatus is also provided, corresponding to the text processing method described above. Figure 11 This is a structural block diagram of another text processing apparatus shown in this application, such as... Figure 11 As shown, the device may include: a set acquisition module 1102, a relevance determination module 1104, a text determination module 1106, and a text output module 1108.

[0135] The set acquisition module 1102 is used to acquire an original text set and a knowledge graph of the target domain by calling a first interface. The first interface includes a first parameter, the value of which includes the original text set. The original text set contains original text from multiple domains, and the multiple domains include the target domain. The relevance determination module 1104 is used to match multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain. The text determination module 1106 is used to filter the original text based on the relevance between the original text and the target domain to obtain the target text of the target domain. The text output module 1108 is used to output the target text by calling a second interface. The second interface includes a second parameter, the value of which includes the target text.

[0136] It should be noted that the aforementioned set acquisition module 1102, relevance determination module 1104, text determination module 1106, and text output module 1108 correspond to steps S702 to S708. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. It should also be noted that the aforementioned modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of the device within the provided server 10.

[0137] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided above, but are not limited to the solutions provided above.

[0138] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0139] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0140] In this embodiment, the electronic device described above can execute the program code in the method.

[0141] Optionally, Figure 12 This is a structural block diagram of an electronic device according to this application. Figure 12 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1202, memory 1204, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to the radio frequency module, the audio module, and the display.

[0142] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0143] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: acquiring an original text set and a knowledge graph of the target domain, wherein the original text set contains original text from multiple domains, and the multiple domains contain the target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and filtering the original text based on the relevance between the original text and the target domain to obtain the target text of the target domain.

[0144] Those skilled in the art will understand that, Figure 12 The structure shown is for illustrative purposes only; the electronic device can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile internet device (MID), PAD, and other terminal devices. Figure 12 This does not limit the structure of the aforementioned electronic device. For example, electronic device A may include more or fewer components (such as network interfaces, display devices, etc.) than those shown in the figure, or have the same characteristics as the aforementioned electronic device. Figure 12 The configuration shown.

[0145] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0146] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0147] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0148] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining an original text set and a knowledge graph of the target domain, wherein the original text set contains original text from multiple domains, and the multiple domains contain the target domain; matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain; and filtering different original texts based on the relevance between different original texts and the target domain to obtain the target text of the target domain.

[0149] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0150] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0151] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0152] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0158] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text processing method, characterized in that, include: Obtain an original text set and a knowledge graph of the target domain, wherein the original text set contains original text from multiple domains, and the multiple domains include the target domain; The knowledge graph of the target domain is matched with the original text in the original text set to determine the relevance between the original text and the target domain. Based on the relevance between the original text and the target domain, the original text is filtered to obtain the target text for the target domain.

2. The method according to claim 1, characterized in that, The step of matching multiple entities contained in the knowledge graph of the target domain with the original text in the original text set to determine the relevance between the original text and the target domain includes: The plurality of entities are matched with preset text in the target domain to determine the entity weights corresponding to the plurality of entities, wherein the entity weight of any entity is used to characterize the relevance of the entity to the target domain; Match the plurality of entities with the original text to determine at least one target entity contained in the original text; The entity weights corresponding to the original text containing at least one target entity are summarized to obtain the relevance between the original text and the target domain.

3. The method according to claim 2, characterized in that, The step of matching the plurality of entities with preset text in the target domain to determine the entity weights corresponding to the plurality of entities includes: The plurality of entities are matched with the preset text to determine the distribution results of the plurality of entities in the preset text, wherein the distribution result of any one entity is used to characterize the position and quantity of the entity in the preset text; Based on the distribution of the multiple entities in the preset text, the entity weights corresponding to the multiple entities are determined.

4. The method according to claim 3, characterized in that, The target domain includes multiple disciplines; determining the entity weights corresponding to the multiple entities based on their distribution in the preset text includes: Based on the distribution results of the multiple entities in the preset text, the initial weights corresponding to the multiple entities are determined; Based on the subject types to which the multiple entities belong, the coverage corresponding to the multiple subjects is determined, wherein the coverage corresponding to any one subject is used to characterize the proportion of entities belonging to that subject in the preset text; The initial weights of the multiple entities are adjusted based on the coverage of the multiple disciplines to obtain the entity weights of the multiple entities. The entity weight of any entity is inversely proportional to the coverage of the discipline to which the entity belongs.

5. The method according to any one of claims 1 to 4, characterized in that, The step of filtering the original text based on the relevance between the original text and the target domain to obtain target text for the target domain includes: Based on the relevance between the original text and the target domain, the original text is filtered to obtain candidate texts for the target domain; The candidate texts are filtered to obtain filtered texts, wherein the filtered texts are used to characterize the texts in the candidate texts that belong to the target domain; The filtered text is then filtered to obtain the target text, wherein the target text is used to characterize the text in the filtered text whose quality score is greater than a preset score.

6. The method according to claim 5, characterized in that, The filtering of the candidate text to obtain the filtered text includes: Feature extraction is performed on the candidate text to obtain the first text feature of the candidate text; The first text feature is input into the relevance prediction model to obtain the prediction result output by the relevance prediction model, wherein the prediction result is used to characterize the probability that the candidate text belongs to the target domain; The candidate text is filtered based on the prediction results to obtain the filtered text.

7. The method according to claim 5, characterized in that, The step of filtering the filtered text to obtain the target text includes: Feature extraction is performed on the filtered text to obtain the second text feature of the filtered text; The second text feature is input into the quality assessment model to obtain the assessment result output by the quality assessment model, wherein the assessment result is used to characterize the quality score of the filtered text; The filtered text is then filtered again based on the evaluation results of the filtered text to obtain the target text.

8. A corpus expansion method, characterized in that, include: Obtain the original corpus, the knowledge graph of the target domain, and the initial corpus of the target domain, wherein the original corpus contains corpus from multiple domains, and the multiple domains include the target domain; The knowledge graph of the target domain is matched with the corpus in the original corpus to determine the relevance between the corpus and the target domain. Based on the relevance between the corpus and the target domain, the corpus is filtered to obtain an expanded corpus for the target domain; The initial corpus and the expanded corpus are combined to obtain the training corpus for the target domain, wherein the training corpus is used to train the machine learning model.

9. The method according to claim 8, characterized in that, The step of matching multiple entities contained in the knowledge graph of the target domain with the corpus in the original corpus to determine the relevance between the corpus and the target domain includes: The plurality of entities are matched with the initial corpus to determine the entity weights corresponding to the plurality of entities, wherein the entity weight corresponding to any entity is used to characterize the relevance between the entity and the target domain; The plurality of entities are matched with the corpus to determine at least one target entity contained in the corpus; The entity weights corresponding to the original text containing at least one target entity are summarized to obtain the relevance between the corpus and the target domain.

10. The method according to claim 8 or 9, characterized in that, The process of filtering the corpus based on its relevance to the target domain to obtain an expanded corpus for the target domain includes: Based on the relevance between the corpus and the target domain, the corpus is filtered to obtain candidate corpus for the target domain; The candidate corpus is filtered to obtain filtered corpus, wherein the filtered corpus is used to characterize the corpus belonging to the target domain among the candidate corpus; The filtered corpus is filtered to obtain the expanded corpus, wherein the expanded corpus is used to characterize the corpus in the filtered corpus whose quality score is greater than a preset score.

11. A text processing method, characterized in that, include: The original text set and the knowledge graph of the target domain are obtained by calling the first interface. The first interface includes a first parameter, the value of which includes the original text set and the knowledge graph of the target domain. The original text set contains original text from multiple domains, and the multiple domains include the target domain. The knowledge graph of the target domain is matched with the original text in the original text set to determine the relevance between the original text and the target domain. Based on the relevance between the original text and the target domain, the original text is filtered to obtain the target text for the target domain; The target text is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target text.

12. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 11.

Images