Text data processing method and device and computer equipment

Abstract

The invention relates to a text data processing method and device, computer equipment and a computer readable storage medium, which can be used for data processing in the financial field. The method comprises the steps of obtaining unstructured text data related to a business scene, performing word segmentation to determine entity objects and an entity sequence, performing statistics on frequency values of the entity objects, filtering the entity objects in the entity sequence according to a preset frequency threshold value, and obtaining an initial sequence after filtering, the feature vectors corresponding to the entity objects are constructed and input into the preset extraction model to obtain the entity relationship, and then the corresponding knowledge graph can be generated according to the entity objects and the entity relationship. Therefore, the knowledge graph in the business scene can be obtained by processing the text data related to the business scene, the data volume of the business scene is simplified, more valuable information in the text data is reserved, the availability of the text data is improved, the learning difficulty is reduced, and the data utilization efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a text data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of computer technology, more and more business scenarios need to rely on business systems in the computer environment. As business scenarios continue to enrich and expand, the processes in business scenarios are becoming more and more complex, which not only increases the operational difficulty and learning cost of business personnel, but also increases the training difficulty and cost of some processes that can be replaced by artificial intelligence or models.

[0003] Knowledge Graph (KG) is essentially a large-scale semantic network that contains entities, concepts and various semantic relationships between them. It has become the cornerstone of cognitive intelligence. Knowledge graphs can not only improve the ability of artificial intelligence machine learning, but also better help business personnel master complex business operation processes and reduce learning costs. However, the current application of knowledge graphs is mainly limited to general fields, and cannot adapt to the needs of some business fields. It is impossible to use text data related to business scenarios to solve problems that are too complex in business scenarios. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a text data processing method, device, computer equipment, computer readable storage medium and computer program product are provided. The technical solution of the present disclosure is as follows:

[0005] According to one aspect of an embodiment of the present disclosure, a method for processing text data is provided, including:

[0006] Obtain unstructured text data related to business scenarios;

[0007] Based on a preset polynomial grammar model, the unstructured text data is segmented, and an entity sequence is determined according to the segmentation result; the entity sequence includes a plurality of entity objects;

[0008] Counting the frequency values ​​of the entity objects, and filtering the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold to obtain a filtered initial sequence;

[0009] Constructing a feature vector based on the entity objects in the initial sequence, inputting the feature vector and a preset data set into a preset extraction model, and outputting the entity relationship between the entity objects through the extraction model;

[0010] A knowledge graph corresponding to the business scenario is generated based on the entity objects and the entity relationships.

[0011] In one embodiment, after determining the entity sequence according to the word segmentation result, the method further includes:

[0012] Calculating association probabilities between a plurality of entity objects according to the unstructured text data;

[0013] When the association probability reaches a preset probability threshold, a new entity object is generated according to a plurality of entity objects corresponding to the association probability, and the new entity object is added to the entity sequence.

[0014] In one embodiment, after constructing the feature vector based on the entity objects in the initial sequence, the method further includes:

[0015] Calculate the similarity between the feature vectors according to a preset algorithm, and select feature vectors whose similarity is higher than a preset threshold;

[0016] Determine the screening threshold of similar words based on the screened feature vectors;

[0017] An entity object is selected from the entity sequence according to the screening threshold, and the selected entity object is added to the initial sequence.

[0018] In one embodiment, before generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes:

[0019] Calculating the occurrence frequency of the entity object corresponding to the entity relationship according to a preset corpus;

[0020] When the occurrence frequency is lower than a preset frequency threshold, the entity relationship is deleted.

[0021] In one embodiment, after generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes:

[0022] Determine a node set for the nodes in the knowledge graph according to a graph group algorithm, and perform cluster analysis;

[0023] Calculating the in-degree of the node set according to the result of the cluster analysis;

[0024] The node is optimized according to the in-degree.

[0025] In one embodiment, after generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes:

[0026] Determine the subject keywords of the unstructured text data according to a preset subject mining model;

[0027] The weight of the entity object is determined according to the subject keywords and the knowledge graph is optimized.

[0028] According to another aspect of an embodiment of the present disclosure, there is provided a device for processing text data, including:

[0029] Data acquisition module, used to acquire unstructured text data related to business scenarios;

[0030] A sequence determination module, used to segment the unstructured text data based on a preset polynomial grammar model, and determine an entity sequence according to the segmentation result; the entity sequence includes a plurality of entity objects;

[0031] A sequence filtering module, used to count the frequency values ​​of the entity objects, and filter the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold to obtain a filtered initial sequence;

[0032] A relationship extraction module, used for constructing a feature vector based on the entity objects in the initial sequence, inputting the feature vector and a preset data set into a preset extraction model, and outputting the entity relationship between the entity objects through the extraction model;

[0033] A graph generation module is used to generate a knowledge graph corresponding to the business scenario based on the entity objects and the entity relationships.

[0034] According to another aspect of an embodiment of the present disclosure, a computer device is further provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, including a computer program, wherein the steps of the above method are implemented when the computer program is executed by a processor.

[0037] In the technical solution provided by the embodiment of the present disclosure, unstructured text data related to the business scenario can be obtained, the text data can be segmented and the entity objects and entity sequences can be determined through a multivariate grammar model, and then the frequency values ​​of the entity objects are counted, and the entity objects in the entity sequence are filtered according to a preset frequency threshold. After filtering, an initial sequence is obtained, and a feature vector corresponding to the entity object can be constructed based on the initial sequence. The feature vector is input into a preset extraction model to obtain the entity relationship between the entity objects, and then the corresponding knowledge graph can be generated according to the entity objects and the entity relationship. In this way, the knowledge graph under the business scenario can be obtained by processing the text data related to the business scenario, which simplifies the data volume of the business scenario, retains more valuable information in the text data, improves the availability of text data, reduces the difficulty of learning, and improves data utilization efficiency.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the implementation methods of this specification or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] FIG1 is a schematic flow chart of a method for processing text data in one embodiment;

[0041] Figure 2 is a schematic flow chart of a method for expanding a solid sequence in an embodiment;

[0042] Figure 3 is a schematic flow chart of a method for expanding an initial sequence in an embodiment;

[0043] Figure 4 is a flowchart of a method for processing text data in another embodiment;

[0044] Figure 5 is a flowchart of a method for processing text data in another embodiment;

[0045] Figure 6 is a flowchart of a method for processing text data in another embodiment;

[0046] Figure 7 is a schematic diagram of the structure of a text data processing device in an embodiment;

[0047] Figure 8 The figure is a schematic diagram of the internal structure of a computer device in an embodiment. DETAILED DESCRIPTION

[0048] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0050] The terms "vertical", "horizontal", "left", "right", "up", "down", "front", "rear", "circumferential", "direction of travel" and similar expressions used in this document are based on the orientation or position relationship shown in the drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0051] Unless otherwise defined, the technical and scientific terms used herein may have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The terms "and / or", "and / or", "at least one of..." used herein include any and all combinations of one or more related listed items. It should be noted that the connection, connection, etc. described in the present disclosure may be a direct connection through an interface or pin between devices, may be a connection through a lead, or may be a wireless connection (communication connection).

[0052] A knowledge graph is a relational network that connects different types of information. It provides the ability to analyze problems from a "relationship" perspective. The knowledge graph can display complex knowledge fields through data mining, information processing, knowledge measurement, and graphics, revealing the dynamic development laws of knowledge fields. It can be applied to scenarios such as intelligent search, text analysis, machine reading comprehension, anomaly monitoring, and risk control. Knowledge graphs can generally be divided into two types: comprehensive knowledge graphs and professional knowledge graphs. Professional knowledge graphs mainly reveal the relationship between information based on specific professional fields.

[0053] Nowadays, the architecture of business systems is becoming increasingly complex, and it is extremely necessary to effectively play the role of knowledge graphs. For example, in the financial system, the API gateway provides a unified access portal for services in the microservice architecture, and the client accesses related services through the API gateway. At present, the knowledge acquisition and learning methods of banking gateway products are still mainly based on the establishment of technical courses such as text-type data and code modules. However, the API gateway module has many functions, the relevant personnel have a large amount of tasks, the number of documents is large, and the technical threshold is high. Therefore, in a short period of time, it is difficult to fully understand the gateway product, sort out the connection between the architectures of each module, and the specific technical overview. Relying solely on documents and existing codes requires a long learning cost and cycle to complete.

[0054] Based on the above technical problems, the inventors have conducted research and developed a method for processing text data in specific business scenarios or business fields using big data and artificial intelligence technologies. This method can more efficiently extract valuable information from text data, and can extract and display the relationship between this information based on knowledge graph technology, thereby reducing the learning costs of relevant staff and improving work efficiency.

[0055] In one embodiment, as shown in FIG1 , a method for processing text data is provided, comprising the following steps:

[0056] Step S210, obtaining unstructured text data related to the business scenario.

[0057] The business scenario may be a specific application scenario in which a business system processes a business process, such as a gateway module of a bank or other financial institution. Unstructured text data may be text data extracted from documents such as introductory articles, instructions, web pages, and user manuals in fields related to the business scenario.

[0058] Specifically, text data can be extracted from web pages, electronic documents and other texts related to the business scenario through big data algorithms such as data extraction. For example, if the business scenario is a banking gateway operation scenario, data can be extracted from text data information such as the architecture document, interface information, technical description (such as filter description documents in the project), and front-end and back-end product document data of the gateway product to obtain unstructured text data related to the gateway field.

[0059] Step S220: segment the unstructured text data based on a preset polynomial grammar model, and determine an entity sequence according to the segmentation result.

[0060] The entity sequence includes several entity objects. The preset multi-gram model can be a model trained by using the N-gram algorithm. N-Gram is a language model commonly used in large vocabulary continuous speech recognition, which can be used for sentence similarity comparison, fuzzy query, sentence rationality, sentence correction, etc. The more commonly used ones are the binary Bi-Gram and the ternary Tri-Gram.

[0061] Specifically, the N-grams statistical language model can be used to perform word segmentation on a large amount of unstructured text data, entity features can be extracted based on the word segmentation results, entity objects can be generated based on the extracted entity features, and entity sequences can be composed of entity objects. In some specific implementations, the hash structure can also be used to iterate the initial number of entities.

[0062] For the problem of identifying new keywords in a specific field, the initial word segmentation algorithm of the initial word library can adopt the word segmentation method of adding HMM (Hidden Markov Model, which can be used to describe a Markov process with hidden unknown parameters) statistical algorithm, and add manually collected data as the initial dictionary based on some open source Chinese word dictionary libraries. For example, the word segmentation can adopt the hybrid word segmentation of jieba (an open source Chinese word segmentation third-party library) and NLTK (Natural Language Toolkit, natural language processing toolkit), clean the initial text data and calculate the word segmentation to obtain the initial word sequence, and remove the stop words from the initial word library to construct the entity object of the head candidate. The specific method is as follows:

[0063] (1)

[0064] (2)

[0065] In the formula, Represents the initial constructed entity object sequence, is the stop word sequence, is the initial word sequence. Since Chinese stop words are highly flexible, the judgment requirements in the word segmentation process in different texts are uncertain. Since the construction demand domain is still less sensitive to the expansion sequence of words such as verbs and common personal names, the sequence is iteratively improved and expanded.

[0066] For the specific stop word expansion method, It represents the expanded stop word group of a stop word. As New expansion words based on words do not require strong matching of words. For the entity extraction task of specified HPC (Highperformance computing), you can design Indicates the length of the word. For word quantity, To set the threshold.

[0067] In some other embodiments, if the unstructured text data is in HTML format, all text data in tags such as title and context can be extracted after parsing by a parsing library or algorithm such as beautifulsoup4 as the basis for word segmentation. In some other embodiments, the word frequency and probability distribution after word segmentation can also be calculated by an N-gram model, and the entity objects in the sequence can be determined based on the word frequency and / or probability.

[0068] Step S230 , counting the frequency values ​​of the entity objects, and filtering the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold, to obtain a filtered initial sequence.

[0069] Specifically, the frequency value of entity objects in the entity sequence can be calculated through the N-gram model, and the frequency value of each entity object can be compared with a preset frequency threshold. The entity objects with frequency values ​​lower than the frequency threshold can be deleted from the entity sequence, and the entity sequence after deletion is used as the initial sequence.

[0070] Step S240, constructing a feature vector based on the entity objects in the initial sequence, inputting the feature vector and a preset data set into a preset extraction model, and outputting the entity relationship between the entity objects through the extraction model.

[0071] The preset data set includes a large number of pre-acquired sentences, chapters and other corpora, which may be the unstructured text data acquired in step S210, or a corpus acquired in advance by other means. The preset extraction model may be pre-trained based on the "Word2Vec (word to vector)" model, and may be used to extract the association between different word vectors from the input word vectors.

[0072] Specifically, based on the target HPC high-performance computing NER (Named Entity Recognition) task, Word2vec's auto-encoder (a neural network used in semi-supervised learning and unsupervised learning) can be used for unsupervised pre-training to obtain the word vectors of each entity object in the form of word embedding after the original unstructured text is segmented. These word vectors are used as feature vectors to input the extraction model with the preset data set, and the extraction model outputs the entity relationship between the entity objects.

[0073] In some other implementations, gensim (generate similarity, an open source library for natural language processing) can be used for initialization to randomly generate n-dimensional vectors for words. Stored in hash form. To solve the problem that the initial corpus keywords are low-frequency words, define Threshold parameter, is the dictionary size. When the following formula (3) is satisfied, delete The frequency of occurrence of words.

[0074] (3)

[0075] High-frequency words can be treated with a value balancing operation similar to that used in statistics to adjust the counts of useless high-frequency words.

[0076] (4)

[0077] In the above formula, D represents the corpus after word segmentation, v represents the current calculated word, and counter is the word frequency calculation function. g(v) can represent the frequency of word v. For whether to discard word v, refer to the following formula (5).

[0078] (5)

[0079] In the above formula, λ is the word frequency threshold parameter, and the value in the brackets is the migration value. You can also use a random real value in (0,1) to compare with the migration value. If it is greater than the migration value, v is discarded. prob is a function that can be used to represent the corresponding probability of a value falling within a specified interval.

[0080] Step S250: Generate a knowledge graph corresponding to the business scenario based on the entity objects and the entity relationships.

[0081] Specifically, a knowledge graph related to the business scenario can be generated based on the extracted entity objects and the relationships between different entity objects. In some other implementations, the knowledge graph can also be optimized by a graph group detection algorithm and / or a clustering algorithm of an LDA (Latent Dirichlet Allocation) topic model.

[0082] In the technical solution provided by the embodiment of the present disclosure, unstructured text data related to the business scenario can be obtained, the text data can be segmented and the entity objects and entity sequences can be determined through a multivariate grammar model, and then the frequency values ​​of the entity objects are counted, and the entity objects in the entity sequence are filtered according to a preset frequency threshold. After filtering, an initial sequence is obtained, and a feature vector corresponding to the entity object can be constructed based on the initial sequence. The feature vector is input into a preset extraction model to obtain the entity relationship between the entity objects, and then the corresponding knowledge graph can be generated according to the entity objects and the entity relationship. In this way, the knowledge graph under the business scenario can be obtained by processing the text data related to the business scenario, which simplifies the data volume of the business scenario, retains more valuable information in the text data, improves the availability of text data, reduces the difficulty of learning, and improves data utilization efficiency.

[0083] In one embodiment, Figure 2 As shown, after determining the entity sequence according to the word segmentation results, it also includes:

[0084] Step S2202: Calculate association probabilities between the plurality of entity objects based on the unstructured text data.

[0085] Specifically, the Bi-Gram algorithm in N-gram can be used to calculate the probability that two entity objects appear simultaneously in a specific interval in the unstructured text data. In some other implementations, the specific interval can include a sentence, a paragraph, a document, etc. in the unstructured text data.

[0086] Step S2204: when the association probability reaches a preset probability threshold, a new entity object is generated according to a plurality of entity objects corresponding to the association probability, and the new entity object is added to the entity sequence.

[0087] Specifically, if the association probability of multiple entity objects reaches a preset probability threshold, these entity objects can be combined into a new entity and added to the entity sequence. For example, if two entity objects are "storage" and "space", and the probability that these two entity objects appear in a sentence at the same time exceeds the probability threshold, a new entity object "storage space" can be generated based on these two entity objects, and "storage space" can be added to the entity sequence.

[0088] In the above embodiment, new entity objects may be generated according to the association probability between entity objects, thereby expanding the entity sequence, adding more entities with strong association, and improving data quality.

[0089] In one embodiment, Figure 3 As shown, after constructing the feature vector based on the entity objects in the initial sequence, it also includes:

[0090] Step S2402, calculating the similarity between the feature vectors according to a preset algorithm, and screening out feature vectors with similarity higher than a preset threshold.

[0091] Among them, the preset algorithm may include a Skip-gram algorithm.

[0092] Specifically, after constructing the feature vector corresponding to the entity object, the similarity between the vectors can be calculated through an algorithm, and the feature vectors with higher similarity can be screened out through a preset threshold.

[0093] Step S2404: Determine a screening threshold for similar words based on the screened feature vectors.

[0094] Specifically, the filtered feature vectors can be analyzed through an algorithm or model to determine the screening criteria for similar words and the weight ratio of each criterion, and generate a screening threshold for screening similar words.

[0095] Step S2406: selecting entity objects from the entity sequence according to the screening threshold, and adding the selected entity objects to the initial sequence.

[0096] Specifically, the entity sequence obtained in step S220 can be screened according to the screening threshold. For example, an entity object can be selected from the initial sequence and the entity sequence respectively, and the similarity of the two entity objects can be calculated. If the similarity meets the screening threshold, the entity object selected from the entity sequence can be added to the initial sequence. It should be noted that the entity object selected from the entity sequence for calculating the similarity should not be in the initial sequence.

[0097] In the above embodiment, the screening threshold of similar words can be determined based on the similarity relationship of entity objects in the initial sequence, and entity objects with higher similarity can be extracted from the entity sequence based on the threshold and added to the initial sequence. This can avoid filtering out certain entities with higher value due to too low frequency values, reduce the possibility of data omissions, and improve data quality.

[0098] In one embodiment, Figure 4 As shown, before generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, it also includes:

[0099] Step S242, calculating the occurrence frequency of the entity object corresponding to the entity relationship according to a preset corpus.

[0100] Among them, an entity relationship can correspond to two entity objects, or more than two entity objects.

[0101] Specifically, several sentences and / or several pieces of corpus can be selected from a preset corpus. If a certain entity relationship corresponds to two entity objects, the frequency of the two entity objects appearing simultaneously in a sentence and / or a piece of corpus can be calculated.

[0102] Step S244: when the occurrence frequency is lower than a preset frequency threshold, delete the entity relationship.

[0103] Specifically, if the occurrence frequency of two or more entity objects corresponding to a certain entity relationship is lower than a preset frequency threshold, the entity relationship may be deleted.

[0104] The above steps can be implemented by the following algorithm:

[0105] S2422, get entity objects (w1, w2, ..., w N ) and the corpus sequence (C1, C2, ..., C k ).

[0106] S2424, regular sentence segmentation is performed on the corpus in the corpus sequence using a preset regular expression to obtain a sentence sequence (S1, S2, ..., S m ).

[0107] S2426, set the algorithm loop, let i=1,2, ...,N-1; j=i+1,i+2, ...,N.

[0108] S2428, calculate the function contentLinks(w i , w j , C k ) value, sLinks[w i , w j ,( S1,S2,……,S m )]value.

[0109] S2442, contentLinks(w i , w j , C k ) is compared with λ1, sLinks[w i , w j ,( S1,S2,……,S m )] will be compared with λ2.

[0110] S2444, word frequency (w i , w j ) weighted, and decide whether to keep w by the weight i With w j The corresponding entity relationship.

[0111] Among them, λ1 can be the frequency threshold of the article unit corpus, λ2 can be the frequency threshold of the sentence unit, contentLinks can be expressed as the word sequence frequency statistics of a text corpus, and sLinks can be expressed as the word sequence frequency statistics of a single sentence. In some specific applications, a lower weight can be assigned to the article unit frequency and a higher weight can be assigned to the sentence unit frequency.

[0112] In the above embodiment, the entity relationship can be deleted based on the frequency of occurrence of the entity object in the preset corpus. If the entity object corresponding to the entity relationship appears less frequently in the corpus, the entity relationship can be deleted. In this way, by optimizing the entity relationship, it is possible to avoid data redundancy caused by too many entity relationships and reduce the difficulty of data processing.

[0113] In one embodiment, Figure 5 As shown, after generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, it also includes:

[0114] Step S262, determine the node set for the nodes in the knowledge graph according to the graph community algorithm, and perform cluster analysis.

[0115] Among them, the Graph Community Detection algorithm (also known as the Graph Community Detection algorithm) is a clustering method. A node can be a point in the knowledge graph used to represent an entity object.

[0116] Specifically, the graph group algorithm can be used to calculate the closely related sets based on the nodes in the knowledge graph for cluster analysis, and the association information between entities can be obtained through cluster analysis.

[0117] Step S264, calculating the in-degree of the node set according to the result of the cluster analysis.

[0118] Among them, the node is the data element in the knowledge graph.

[0119] Specifically, the graph group algorithm can be used to calculate the in-degree and out-degree of the node. The calculation formula is as follows:

[0120] (6)

[0121] In the formula, represents the number of relations contained in the L graph, N is the number of vertices, and k aRepresented as the degree of node a, A a,b is the value in the adjacency matrix of nodes a and b, c a It is represented as the clustering of node a; δ is represented as the Kronecker function, which returns 1 if equal and 0 if not equal. k a and k b represents the in-degree of two vertices, Inferred as the expected number of edges in the graph. In some other embodiments, the module value can be increased by the difference between the a and b adjacency matrix values ​​and the expected values.

[0122] Step S266: Optimize the node according to the in-degree.

[0123] Specifically, the entity nodes may be adjusted according to the in-degree. For example, a threshold may be preset to remove entity relationships with in-degrees lower than the threshold, or entity relationships with in-degrees lower than the threshold may be expanded.

[0124] In the above embodiment, the entities and entity relationships in the knowledge graph can be clustered and analyzed by the graph group algorithm, and the entities and entity relationships can be further optimized by calculating the in-degree and out-degree to improve the accuracy of the knowledge graph.

[0125] In one embodiment, Figure 6 As shown, after generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, it also includes:

[0126] Step S272, determining the topic keywords of the unstructured text data according to a preset topic mining model.

[0127] The topic mining model may be a model trained using an LDA (Latent Dirichlet Allocation) algorithm.

[0128] Specifically, we can input the entity set {e1, e1,…, e i ,……, e n} and the corresponding weights ; Perform the mean operation on the weight values, which are [0.1, 1] ten weight levels, and average the weights of the entities extracted by LDA topics to 1; Determine the relationship between the number of entities in α and the set threshold λα, that is, if n≤λα, no entity granularity trimming is performed, and if n>λα, the entity is discarded. The weight levels after averaging are discarded from 0.1 until n≤λα is satisfied. Among them, different λ can be set according to different α.

[0129] Step S274, determine the weight of the entity object according to the subject keyword and optimize the knowledge graph.

[0130] Specifically, the entity relationships with in-degrees within a preset range can be pruned through a preset relationship granularity balancing algorithm, and entity relationships with lower in-degrees can be isolated or expanded. The specific process can be expressed by the following formula:

[0131] (7)

[0132] In formula (7), A i Can represent e i The sum of the relationship between knowledge point entity nodes. n Can represent e i With e n The number of relationships between entities. R can represent the relationship threshold. α can represent the cumulative number of relationships between entities that do not meet the threshold. β can represent the relationship overflow reduction value.

[0133] In some specific implementations, the idea of ​​graph group detection can be used to calculate a k-like a The in-and-out degrees of the graph entity nodes of k, that is, the sum of the relationship quantities, are different R values ​​designed for different types of graphs, and multiple iterative calculations are required for a graph.

[0134] In some other implementations, because the relationship quantity is different from the entity quantity, one relationship granularity involves two entities. The clipping value α and the relationship quantity increase value β when determining the overflow of the relationship quantity need to further determine the characteristics and quantity value of the knowledge relationship. The relationship weight is defined as follows:

[0135] (8)

[0136] In formula (8), Represented as e i With e j The relationship weights are the sum of the comprehensive weights of the two entities and the comprehensive weight of the relationship. The specific steps are as follows:

[0137] S2742, according to The screening condition determines the entity point e that needs to be overflow trimmed i , Statistics and e i A collection of entities linked with existence relationships (including coarse-grained and fine-grained relationships) The relationship between each entity knowledge point .

[0138] S2744, α value from Entities e from largest to smallest in descending order of relational quantity j Priority selection means that entities with more relationships are selected for pruning first. The specific pruning edges are determined based on the W weight, that is, knowledge relationships with smaller weights are cut first.

[0139] S2746, according to The filter condition determines the entity e that needs to be added to the relationship k , the addition relationship of β value is still based on the comprehensive weight of the entity. The value is larger, then β chooses The entities with smaller values ​​are balanced, and the knowledge points with larger weights are selected.

[0140] In the above embodiment, after the knowledge graph is preliminarily generated based on entities and entity relationships, the LDA topic model can be used to extract topic keywords in unstructured data, and based on the topic keywords, the weight of each entity object in the knowledge graph can be determined, and the weight of entity objects with a higher correlation with the keywords can be increased, thereby optimizing the knowledge graph and improving the correlation between the knowledge graph and the business scenario theme.

[0141] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] According to another aspect of the present disclosure, Figure 7 As shown, a text data processing device is also provided, comprising:

[0143] The data acquisition module 310 is used to acquire unstructured text data related to the business scenario;

[0144] A sequence determination module 320 is used to segment the unstructured text data based on a preset polynomial grammar model, and determine an entity sequence according to the segmentation result; the entity sequence includes a plurality of entity objects;

[0145] A sequence filtering module 330 is used to count the frequency values ​​of the entity objects, and filter the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold to obtain a filtered initial sequence;

[0146] A relationship extraction module 340 is used to construct a feature vector based on the entity objects in the initial sequence, input the feature vector and a preset data set into a preset extraction model, and output the entity relationship between the entity objects through the extraction model;

[0147] The graph generation module 350 is used to generate a knowledge graph corresponding to the business scenario based on the entity objects and the entity relationships.

[0148] For the specific definition of the above-mentioned processing device, please refer to the definition of the above-mentioned processing method, which will not be repeated here. According to the above-mentioned processing method, the processing device can add a first module, a second module, etc. to implement the steps in the corresponding method embodiment. Each module in the above-mentioned processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] It should be noted that the text data processing method and device of the present invention can be used for data processing in the financial field, and can also be used in any field other than the financial field. The application field of the method and device of the present invention is not limited.

[0150] According to another aspect of the embodiment of the present disclosure, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the above processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0151] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0153] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited thereto.

[0155] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0157] It will be understood that the present disclosure is not limited to the exact construction that has been described and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for processing text data, characterized in that: include: Obtain unstructured text data related to business scenarios; Based on a preset polynomial grammar model, the unstructured text data is segmented, and an entity sequence is determined according to the segmentation result; the entity sequence includes a plurality of entity objects; Counting the frequency values ​​of the entity objects, and filtering the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold to obtain a filtered initial sequence; Constructing a feature vector based on the entity objects in the initial sequence, inputting the feature vector and a preset data set into a preset extraction model, and outputting the entity relationship between the entity objects through the extraction model; A knowledge graph corresponding to the business scenario is generated based on the entity objects and the entity relationships.

2. The method according to claim 1, characterized in that After determining the entity sequence based on the word segmentation results, it also includes: Calculating association probabilities between a plurality of entity objects according to the unstructured text data; When the association probability reaches a preset probability threshold, a new entity object is generated according to a plurality of entity objects corresponding to the association probability, and the new entity object is added to the entity sequence.

3. The method according to claim 1, characterized in that After constructing the feature vector based on the entity objects in the initial sequence, the method further includes: Calculate the similarity between the feature vectors according to a preset algorithm, and select feature vectors whose similarity is higher than a preset threshold; Determine the screening threshold of similar words based on the screened feature vectors; An entity object is selected from the entity sequence according to the screening threshold, and the selected entity object is added to the initial sequence.

4. The method according to claim 1, characterized in that: Before generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes: Calculating the occurrence frequency of the entity object corresponding to the entity relationship according to a preset corpus; When the occurrence frequency is lower than a preset frequency threshold, the entity relationship is deleted.

5. The method according to claim 1, characterized in that After generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes: Determine a node set for the nodes in the knowledge graph according to a graph group algorithm, and perform cluster analysis; Calculating the in-degree of the node set according to the result of the cluster analysis; The node is optimized according to the in-degree.

6. The method according to claim 1, characterized in that After generating the knowledge graph corresponding to the business scenario according to the entity object and the entity relationship, the method further includes: Determine the subject keywords of the unstructured text data according to a preset subject mining model; The weight of the entity object is determined according to the subject keywords and the knowledge graph is optimized.

7. A text data processing device, characterized in that: include: Data acquisition module, used to acquire unstructured text data related to business scenarios; A sequence determination module, used to segment the unstructured text data based on a preset polynomial grammar model, and determine an entity sequence according to the segmentation result; the entity sequence includes a plurality of entity objects; A sequence filtering module, used to count the frequency values ​​of the entity objects, and filter the entity objects in the entity sequence according to the frequency values ​​and a preset frequency threshold to obtain a filtered initial sequence; A relationship extraction module, used for constructing a feature vector based on the entity objects in the initial sequence, inputting the feature vector and a preset data set into a preset extraction model, and outputting the entity relationship between the entity objects through the extraction model; A graph generation module is used to generate a knowledge graph corresponding to the business scenario based on the entity objects and the entity relationships.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images