Data extraction method and system, electronic equipment and storage medium

Abstract

The invention discloses a data extraction method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of documents to be detected, and constructing an entity relationship set of a target service; wherein the entity relationship set comprises a plurality of triads, and each triad comprises two entities related to the target business and a corresponding relationship between the two entities; respectively carrying out characteristic vector extraction on each to-be-detected document to obtain a first characteristic vector set of each to-be-detected document; obtaining an association relationship between each to-be-detected document and the entity relationship set by using the first feature vector set, and selecting a target document of a target service from the plurality of to-be-detected documents; and dividing the target document into a plurality of associated fragments, and performing relation mapping extraction on the associated fragments according to each triple of the entity relation set to obtain a plurality of fragment data associated with each triple, thereby obtaining a fragment data set of the target business. The technical problem that document relation extraction is low in efficiency and accuracy can be solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data extraction method, system, electronic device and storage medium. Background Art

[0002] With the development of digital construction and artificial intelligence models, it is becoming increasingly important to quickly build a domain-based knowledge base by extracting entity relationships from documents. The current mainstream technology includes methods based on template matching.

[0003] However, simple template-based matching requires a lot of manpower to enter relevant content according to a fixed extraction template; at the same time, if the content to be extracted is relatively messy, it may be difficult to identify the desired extraction content, resulting in low efficiency and accuracy of document relationship extraction. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a data extraction method, system, electronic device and storage medium, which can solve the technical problems of low efficiency and accuracy of document relationship extraction.

[0005] In order to solve the above technical problems, a technical solution adopted in this application is: the method comprises:

[0006] Acquire multiple documents to be detected, and construct an entity relationship set of a target business; wherein the entity relationship set includes multiple triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities;

[0007] Extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected;

[0008] Using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, a target document of the target business is selected from the multiple documents to be detected;

[0009] The target document is divided into a plurality of associated segments, and relationship mapping is extracted for the associated segments according to each triple of the entity relationship set to obtain a plurality of segment data associated with each triple, thereby obtaining a segment data set of the target business.

[0010] Optionally, the step of obtaining the association relationship between each of the documents to be detected and the entity relationship set by using the first feature vector set to select the target document of the target business from the multiple documents to be detected includes:

[0011] Performing feature extraction on each triple of the entity relationship set to obtain a second feature vector set of the entity relationship set;

[0012] Using the first feature vector set and the second feature vector set, obtaining the similarity between each document to be detected and the entity relationship set;

[0013] The document to be detected whose similarity is higher than a set threshold is determined as the target document.

[0014] Optionally, the step of obtaining the similarity between each document to be detected and the entity relationship set using the first feature vector set and the second feature vector set includes:

[0015] Using the vector angle between the first feature vector set and the second feature vector set of the document to be detected;

[0016] The similarity between the document to be detected and the entity relationship set is determined based on the vector angle; wherein the similarity is negatively correlated with the vector angle.

[0017] Optionally, the step of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining the segment data set of the target business includes:

[0018] Dividing the target document into a plurality of associated segments, and obtaining segment vectors of the associated segments;

[0019] Relationship mapping is performed between the fragment vectors and the triples respectively to obtain a plurality of fragment data associated with the triples, thereby obtaining a fragment data set of the target service.

[0020] Optionally, the step of converting the associated fragment into a fragment vector comprises:

[0021] Dividing the associated fragment into a plurality of sentences, and converting each sentence into a sentence vector respectively;

[0022] The sentence vector is used to perform semantic recognition on the sentence, and context sentence vectors with associated relationships are spliced ​​into the segment vector according to the semantic recognition result.

[0023] Optionally, after the step of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining the segment data set of the target business, the method further includes:

[0024] Obtaining a data extraction instruction for at least one set target of the target business; wherein the extraction instruction carries an extraction condition for the set target;

[0025] The specific data set corresponding to the set target is extracted from the segment data set using the extraction condition.

[0026] Optionally, the extraction condition includes a keyword, and a frequency threshold of the number of occurrences of the keyword and / or a probability threshold of the probability of the keyword appearing in each of the fragment data;

[0027] The step of extracting a specific data set corresponding to a set target from the segment data set by using the extraction condition comprises:

[0028] Select from the fragment data set the fragment data in which the number of occurrences of the keyword exceeds the number threshold or / and select the fragment data in which the probability of the keyword appearing in the fragment data set is greater than the set probability, and determine the combination of the fragment data as the specific data set.

[0029] Optionally, the step of obtaining the probability of the keyword appearing in the segment data set includes:

[0030] Segmenting the fragment data to obtain word combinations of the fragment;

[0031] The number of occurrences and / or the probability of occurrence of the keyword in the word combination are counted to obtain the number of occurrences of the keyword in the segment data set and / or the probability of occurrence of the keyword in the segment data.

[0032] In order to solve the above technical problems, another technical solution adopted by the present application is: a data extraction system, the data extraction system comprising:

[0033] A document extraction module is used to obtain a plurality of documents to be detected and to construct an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities; feature vectors are extracted from each of the documents to be detected to obtain a first feature vector set of each of the documents to be detected;

[0034] A relationship matching module, used to obtain the association relationship between each of the documents to be detected and the entity relationship set by using the first feature vector set, and select a target document of the target business from the multiple documents to be detected;

[0035] The extraction and mapping module is used to divide the target document into multiple related fragments, perform relationship mapping extraction on the related fragments according to each triple of the entity relationship set, obtain multiple fragment data associated with each triple, and obtain the fragment data set of the target business.

[0036] In order to solve the above technical problems, another technical solution adopted by the present application is: an electronic device, the electronic device comprising:

[0037] A memory for storing executable program codes;

[0038] The processor is used to call and run the executable program code from the memory, so that the electronic device executes the data extraction method as described in any one of the above.

[0039] To solve the above technical problems, another technical solution adopted in the present application is: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the data extraction method as described in any one of the above items is implemented.

[0040] Different from the prior art, an embodiment of the present application provides a data extraction method, which includes: obtaining multiple documents to be detected, and constructing an entity relationship set of a target business; wherein the entity relationship set includes multiple triples, each of which includes two entities related to the target business and the corresponding relationship between the two entities; performing feature vector extraction on each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected; using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, and selecting a target document of the target business from the multiple documents to be detected; dividing the target document into multiple associated fragments, performing relationship mapping extraction on the associated fragments according to each triple of the entity relationship set, obtaining multiple fragment data associated with each of the triples, and obtaining a fragment data set of the target business.

[0041] The beneficial effect of the embodiment of the present application is: by using the entity relationship set of the target business to perform a preliminary screening of the feature vector of the document before extracting the document to be detected, the target document is determined from the messy documents, and then the target document and the entity relationship set are subjected to relationship mapping and extraction to obtain a fragment data set of the target business, which can effectively improve the efficiency and accuracy of document relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of the first embodiment of the data extraction method of the present application;

[0043] Figure 2 It is a flow chart of the second embodiment of the data extraction method of the present application;

[0044] Figure 3 It is a flowchart of the third embodiment of the data extraction method of the present application;

[0045] Figure 4It is a flowchart of the fourth embodiment of the data extraction method of the present application;

[0046] Figure 5 It is an exemplary structural block diagram of a data extraction system of the data extraction method of the present application;

[0047] Figure 6 It is another exemplary structural block diagram of the data extraction system of the data extraction method of the present application;

[0048] Figure 7 is an exemplary structural block diagram of an electronic device of the data extraction method of the present application;

[0049] Figure 8 It is an exemplary structural block diagram of a computer-readable storage medium of the data extraction method of the present application. DETAILED DESCRIPTION

[0050] The present application is described in detail below with reference to the accompanying drawings and implementation methods.

[0051] See also Figure 1 , Figure 1 It is a flow chart of the first embodiment of the data extraction method of the present application. It should be noted that if there are substantially the same results, the method of the present application is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, the method includes:

[0052] Step S110, obtaining a plurality of documents to be detected, and constructing an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities;

[0053] This embodiment takes into account that with the development and advancement of digital government construction and artificial intelligence large models, it is increasingly important to quickly build a domain-based knowledge base. Taking government service-provident fund withdrawal as an example, to support the business system to provide provident fund withdrawal services to the people, it is necessary to first structurally convert the government's relevant policy documents, service guidelines, etc. (such as item names, application conditions, application materials, handling procedures, handling time limits, charging standards, common problems, etc.), and build a knowledge base in the field of government services as required. However, this requires a large amount of manual service and file import extraction to extract conversion and verification. When the number of files to be extracted is relatively large and the content is mixed and disordered, the extraction efficiency and service experience are relatively poor, which increases the cost of building the system and special knowledge base.

[0054] Therefore, this embodiment proposes a data extraction method based on the knowledge field. Before extracting the document to be detected, the feature vector of the document is preliminarily screened using the entity relationship set of the target business to determine the target document from the messy documents, and then the target document and the entity relationship set are subjected to relationship mapping extraction to obtain a fragment data set of the target business, which can effectively improve the efficiency and accuracy of document relationship extraction.

[0055] Specifically, the multiple documents to be detected may be a set of original documents whose relationships need to be extracted. The target business may refer to the main business or task that needs to be processed or focused on in a specific business scenario. The target entity relationship set may include multiple triples related to the target business, which are used to define the target knowledge that needs to be identified and extracted. Each triple may consist of two entities and the relationship between the entities, for example (entity A, entity relationship, entity B).

[0056] For example, the target business may be to provide housing fund withdrawal services, and the multiple documents to be tested may be policy documents, service guides, and other documents related to housing fund withdrawal services. The entity relationship set may be used to define key entities related to housing fund withdrawal services and the relationships between key entities. For example, entities may include declaration conditions and handling procedures, and relationships may be conditions that need to be met and / or steps involved.

[0057] Step S120, extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected;

[0058] Specifically, the first feature vector set can be a set of feature vectors corresponding to each document to be detected. By taking the feature vectors of each document to be detected as a vector set, and then using this first feature vector set to select target documents that have an association relationship with the entity relationship set from each document to be detected, a preliminary extraction can be performed before extracting the document, and at the same time, it can avoid detecting individual feature vectors separately, thereby improving the efficiency of extraction.

[0059] Among them, the text data can be converted into a computable form, for example, using natural language processing technology to extract keywords, syntactic structures, etc. for feature vector extraction.

[0060] Step S130, using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, and selecting a target document of the target business from the multiple documents to be detected;

[0061] Specifically, in order to reduce the redundancy of document processing and analysis and improve the extraction efficiency, at least one target document can be identified from each document to be detected by calculating the correlation between the first feature vector set corresponding to each document to be detected and the entity relationship set. The correlation relationship can be that the relationship between two or more entities in the target document is related to the entity relationship in any triple. These connections can be explicit, such as explicitly stated relationships, and / or implicit, such as relationships inferred from context.

[0062] For example, the target document most relevant to the housing provident fund withdrawal business may be selected from the documents to be detected. For example, the target document describes the application materials and / or the handling process in detail.

[0063] Step S140, dividing the target document into multiple related segments, performing relationship mapping extraction on the related segments according to each triple of the entity relationship set, obtaining multiple segment data associated with each triple, and obtaining a segment data set of the target business.

[0064] Specifically, in order to identify the relationship instances in the target document that may be related to the target entity relationship set, the target document can be further split into multiple related segments, which can be sentences or paragraphs and can contain semantic information similar to or identical to at least one triple. Then, in each related segment, the relationship and entity corresponding to the triple are identified and extracted to obtain multiple segment data related to the triple, so that each segment data can centrally describe the entity relationship in the triple, thereby obtaining a segment data set corresponding to the target business.

[0065] For example, the specific conditions and steps related to the application materials can be found in the associated fragments to form structured data. All the paragraph data corresponding to the triples are collected to form a complete fragment data set. This fragment data set can contain all the key provident fund withdrawal business information, which is convenient for system automation processing, user query and service provision.

[0066] This embodiment uses the entity relationship set of the target business to perform a preliminary screening of the feature vectors of the document before extracting the document to be detected, thereby determining the target document from the messy documents, and then performing relationship mapping and extraction between the target document and the entity relationship set to obtain a fragment data set of the target business, which can effectively improve the efficiency and accuracy of document relationship extraction.

[0067] See also Figure 2 , Figure 2 It is a flow chart of the second embodiment of the data extraction method of the present application.

[0068] In this embodiment, the method includes the following steps:

[0069] Step S210, obtaining a plurality of documents to be detected, and constructing an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities;

[0070] Specifically, the above step S110 may be performed, which will not be described in detail herein.

[0071] Step S220, extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected;

[0072] Specifically, feature vector extraction can use methods such as term frequency-inverse document frequency (TF / IDF), tape model, word embedding, sentence vector, deep learning model, principal component analysis, topic model, etc. to extract feature vectors from data in multiple documents to be detected, which is not limited in this embodiment.

[0073] Step S230, performing feature extraction on each triple of the entity relationship set to obtain a second feature vector set of the entity relationship set;

[0074] Specifically, in order to effectively extract and analyze business-related information, simplify similarity calculations, and make complex business relationship processing more intuitive and efficient, the target entity relationship set can be treated as a document to obtain the feature vectors of each triple in the target entity relationship set to obtain a second feature vector set. The first feature vector set and the second feature vector set can then be used to calculate the similarity between the target entity relationship set and the document to be detected.

[0075] Step S240, using the first feature vector set and the second feature vector set to obtain the similarity between each document to be detected and the entity relationship set;

[0076] Specifically, in order to effectively enhance the extraction effect, the second feature vector of the target entity relationship set can be used to obtain the relevance between the document and the target business by calculating the similarity between the feature vector of the document to be detected and the feature vector of the target entity relationship set. The similarity calculation method may include cosine similarity, Euclidean distance, etc.

[0077] Furthermore, the step S240 includes:

[0078] The vector angle between the first feature vector set of the document to be detected and the second feature vector set is obtained; based on the vector angle, the similarity between the document to be detected and the entity relationship set is determined; wherein the similarity is negatively correlated with the vector angle.

[0079] Specifically, this embodiment uses the cosine method to obtain the similarity between two documents by calculating the feature vector sets corresponding to the two documents. The similarity is negatively correlated with the vector angle, that is, the smaller the vector angle, the higher the similarity; the larger the vector angle, the lower the similarity. Please refer to formula (1):

[0080]

[0081] In the formula, SC(d i , d j ) represents the similarity, cos(d i , d j ) represents the angle, d i represents the i-th document; d j represents the jth document; w ki represents the weight of feature item k in document i; w kj Represents the weight of feature item k in document j.

[0082] Step S250: determine the document to be detected whose similarity is higher than a set threshold as the target document.

[0083] Specifically, in order to screen out documents that are highly relevant to the target business, documents with similarities higher than a set threshold can be determined as target documents based on the similarity results of the previous step. Furthermore, when calculating the similarity between the current document to be detected and the target entity relationship set by calculating some feature vectors in the first feature vector set and some feature vectors in the second feature vector set, if the similarity between some feature vectors in the first feature vector set and some feature vectors in the second feature vector set has met the set threshold, the current document to be detected is determined as the target document, so that the other feature vectors in the first feature vector set of the current document to be detected, except for the above-mentioned partial feature vectors, do not need to be further calculated and compared, thereby improving the efficiency of the preliminary screening of the document to be detected.

[0084] The threshold value can be a pre-set demarcation value used to filter documents that are not related to the target entity relationship set. The similarity scores SC (0, 1) are obtained from high to low, and the target documents are filtered according to whether the similarity scores meet the similarity threshold. Taking the similarity threshold D = 0.8 as an example, if the documents with similarity scores higher than or equal to the similarity threshold are determined as target documents, and documents lower than the similarity threshold are ignored, the efficiency and accuracy of knowledge extraction can be improved.

[0085] Step S260, dividing the target document into a plurality of associated segments, and obtaining segment vectors of the associated segments;

[0086] Specifically, in order to facilitate subsequent calculations and comparisons, by vectorizing the associated fragments, the semantic information of the text can be converted into numerical form to obtain the fragment vectors corresponding to each associated fragment, so that these fragment vectors can be used to describe various characteristics and attributes of the text fragments.

[0087] Further, step S260 includes:

[0088] The target document is divided into a plurality of related segments, the related segments are divided into a plurality of sentences, and each sentence is converted into a sentence vector; semantic recognition is performed on the sentences to obtain semantic recognition results, and according to the semantic recognition results, the sentence vectors of the contexts with related relationships are spliced ​​into the segment vectors.

[0089] Specifically, in order to refine the content of the document and make processing and analysis more accurate, the associated fragments can be segmented according to the boundaries of the sentences, and the text information can be converted into numerical representations, so that the system can use the converted sentence vectors to identify and understand the semantic content of the sentences to determine the relevance between the sentences. Semantic recognition can include identifying entities, relationships, emotions, etc. in sentences.

[0090] Then, according to the result of semantic recognition, the context sentences with related relationships can be concatenated into fragment vectors. The context sentence vector can refer to semantically related data that can jointly describe a complete situation or event, thereby retaining the complete semantics and context, and concatenated with the current sentence vector to form a new fragment vector.

[0091] Step S270 , mapping the fragment vectors to the triples respectively, obtaining a plurality of fragment data associated with the triples, and obtaining a fragment data set of the target business.

[0092] Specifically, in order to integrate scattered fragment data and form more comprehensive and structured information, the fragment data and triples can be relationally mapped, and then the semantic association between the fragment vector and the triple can be extracted to ensure that the extracted fragment data is related to the entity relationship of the target business, thereby forming a fragment data set corresponding to the target business, so that the fragment data set can be used to describe various entity relationships and specific content related to the target business, thereby enriching the information in the knowledge graph or business model and improving its accuracy and coverage.

[0093] In this embodiment, by extracting features from each triple of the entity relationship set and calculating the similarity using the feature vector of the document to be detected and the feature vector of the entity relationship set, documents related to the entity relationship set can be identified more accurately; at the same time, by calculating the feature vector set, the similarity between the document and the entity relationship set can be quickly evaluated without the need to perform full-text analysis one by one.

[0094] See also Figure 3 , Figure 3 It is a flowchart of the third embodiment of the data extraction method of the present application.

[0095] In this embodiment, the method includes the following steps:

[0096] Step S310, obtaining a plurality of documents to be detected, and constructing an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities;

[0097] Specifically, the above step S110 may be performed, which will not be described in detail in this embodiment.

[0098] Step S320, extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected;

[0099] Specifically, the above step S120 may be performed, which will not be described in detail in this embodiment.

[0100] Step S330, using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, and selecting a target document of the target business from the multiple documents to be detected;

[0101] Specifically, the above step S130 may be performed, which will not be described in detail in this embodiment.

[0102] Step S340, dividing the target document into multiple related segments, performing relationship mapping extraction on the related segments according to each triple of the entity relationship set, obtaining multiple segment data associated with each triple, and obtaining a segment data set of the target business.

[0103] Specifically, the above step S140 may be performed, which will not be described in detail in this embodiment.

[0104] Step S350, obtaining a data extraction instruction of at least one set target of the target business; wherein the extraction instruction carries an extraction condition of the set target;

[0105] This embodiment takes into account that different roles have different needs when handling target businesses. For example, when the target business is to handle provident fund, individuals can obtain the specific information and procedures required for handling; enterprises can understand the specific requirements and required documents for enterprise handling; and staff can obtain the latest policies and operating specifications. Therefore, through user-defined data extraction instructions, further role division can improve information accuracy and relevance.

[0106] Specifically, the specific target can be different target roles, such as individuals, enterprises, and office personnel. The extraction condition can be used to screen and filter the data in the segment data set that meets the set target. The user can customize the specific target and the corresponding extraction condition through the data extraction instruction, so that when the system obtains the data extraction instruction of at least one set target of the target business, the extraction condition carried by the data extraction instruction is obtained to extract relevant data from the data set.

[0107] Step S360: extracting a specific data set corresponding to a set target from the segment data set using the extraction condition.

[0108] Specifically, when obtaining the extraction conditions, the data associated with the set target can be extracted from the fragment data set using the extraction conditions to obtain a specific data set. For example, an SQL query statement can be used to filter out records that meet the conditions from the database; data processing tools, such as the Pandas library in Python, can also be used to extract corresponding data from the data frame through conditional filtering.

[0109] In this embodiment, the user uses data extraction instructions to customize the setting target and the corresponding extraction conditions, which can improve the flexibility of relationship extraction and enable different roles to access relevant information, thereby improving the security of information data.

[0110] For further information, see Figure 4 , Figure 4 1 is a flow chart of the fourth embodiment of the data extraction method of the present application. This embodiment is a refinement of steps S350 and S360 in the third embodiment above. Step S350 includes:

[0111] Step S410, obtaining a data extraction instruction for at least one set target of the target business; wherein the extraction instruction carries an extraction condition for the set target, and the extraction condition includes a keyword, and a number threshold of the number of occurrences of the keyword and / or a probability threshold of the probability of the keyword appearing in each of the fragment data;

[0112] Step S360 includes:

[0113] Step S420, selecting from the fragment data set the fragment data in which the number of occurrences of the keyword exceeds the number threshold or / and the probability of the keyword appearing in the fragment data is greater than the set probability, and determining the combination of the fragment data as the specific data set.

[0114] Specifically, keywords can be specific central words that need to be focused on in the fragment data. These keywords can be related to the role level of a specific target. The number of occurrences of a keyword can be the total number of times a keyword appears in a specific fragment data, which is used to evaluate the prominence of the keyword in the fragment. If a keyword appears more times in a fragment, it can be said that the fragment has a high correlation with the keyword. The probability of a keyword appearing in the fragment data can be the frequency of a keyword appearing in all fragment data, which can reflect the prevalence of the keyword and the overall relevance of the data set.

[0115] The frequency threshold can be the minimum number of times a keyword appears in the fragment data. For example, if the threshold is set to 10, then the fragment data that contains the keyword at least 10 times can be focused on. The probability threshold can be the probability of a keyword appearing in the fragment data. For example, if the threshold is set to 0.5, then the fragment data in which the keyword appears more than 50% in the fragment data can be focused on. In this way, the filtered fragment data can be combined to form a specific data set.

[0116] Furthermore, the step of obtaining the probability of the keyword appearing in the segment data set includes:

[0117] Segmenting the fragment data to obtain word combinations of the fragment;

[0118] Specifically, word segmentation can be the process of breaking a piece of text into independent words or phrases. For example, irrelevant characters and symbols, such as punctuation and special characters, can be removed; all text can also be converted to uniform uppercase and lowercase to reduce the interference of repeated words. Word combinations can be all individual words or phrases extracted from the fragment data, and these word combinations can be used for further analysis.

[0119] The number of occurrences and / or the probability of occurrence of the keyword in the word combination are counted to obtain the number of occurrences of the keyword in the segment data set and / or the probability of occurrence of the keyword in the segment data.

[0120] Specifically, regarding the calculation of the number of occurrences of the keyword in the segment data set or / and the probability P(c|w; θ) of the keyword appearing in the segment data, P(c|w; θ) is required to meet two conditions: P(c|w; θ) is within the range of 0 and 1; for the segment data set, the sum of P(c|w; θ) is 1. This embodiment adopts the following formula (2):

[0121]

[0122] In the formula, U represents the fragment dataset, c represents the words in the context, c' represents the words in the fragment dataset, and u c is the vector representation of c; v w is the vector representation of the keyword; u c’ is the vector representation of c'.

[0123] This embodiment can not only improve the efficiency and accuracy of data processing, but also provide strong support for data analysis and business decision-making by selecting fragment data in which the number of keyword appearances exceeds the set number of times from the fragment data set and / or selecting fragment data in which the probability of the keyword appearing in the fragment data set is greater than the set probability.

[0124] See also Figure 5 , Figure 5 This is an exemplary structural block diagram of a data extraction system of the data extraction method of the present application. Figure 5 As shown, the data extraction system 500 of the present application may include a document extraction module 501 , a relationship matching module 502 , and an extraction mapping module 503 .

[0125] The document extraction module 501 obtains a plurality of documents to be detected and constructs an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities; extracts a feature vector from each of the documents to be detected to obtain a first feature vector set of each of the documents to be detected;

[0126] A relationship matching module 502, using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, and selects a target document of the target business from the multiple documents to be detected;

[0127] The extraction and mapping module 503 divides the target document into multiple related segments, performs relationship mapping extraction on the related segments according to each triple of the entity relationship set, obtains multiple segment data associated with each triple, and obtains the segment data set of the target business.

[0128] This embodiment uses the entity relationship set of the target business to perform a preliminary screening of the feature vectors of the document before extracting the document to be detected, thereby determining the target document from the messy documents, and then performing relationship mapping and extraction between the target document and the entity relationship set to obtain a fragment data set of the target business, which can effectively improve the efficiency and accuracy of document relationship extraction.

[0129] Furthermore, the extraction mapping module 503 is also used to obtain a data extraction instruction for at least one set target of the target business; wherein the extraction instruction carries an extraction condition for the set target; and uses the extraction condition to extract a specific data set corresponding to the set target from the fragment data set.

[0130] For further information, see Figure 6 , Figure 6 This is another exemplary structural block diagram of the data extraction system of the data extraction method of the present application. Figure 5 The data extraction system 500 further includes:

[0131] The relation extraction processing unit 600 is connected to the document extraction module 501, the relation matching module 502, and the extraction mapping module 503 respectively;

[0132] A knowledge base 601, connected to the document extraction module 501, and connected to the relationship matching module 502;

[0133] The configuration library 602 is connected to the extraction and mapping module 503 .

[0134] Specifically, the relationship extraction processing unit 600 is connected to the document extraction module 501, the relationship matching module 502, and the extraction mapping module 503, and is used to coordinate and manage the modules to ensure smooth information transmission between the modules and avoid data loss or repeated processing.

[0135] The knowledge base 601 can provide prior knowledge of document and entity relationships, ensuring that the document extraction module 501 can obtain data such as the document to be detected and the entity relationship set, so that the document extraction module 501 can obtain previously stored document and entity relationship set information from the knowledge base 601. At the same time, the relationship matching module 502 is connected to the knowledge base 601, so that the relationship matching module 502 can obtain entity relationship information from the knowledge base 601 to calculate the association relationship.

[0136] The configuration library 602 can define mapping rules for different businesses, and the extraction mapping module 503 can use the mapping rules and parameters defined in the configuration library 602 to perform relationship mapping extraction to obtain a detailed fragment data set. In addition, the user can also flexibly configure specific targets and extraction conditions corresponding to specific targets through the configuration library 602 to further extract entity relationships at the role level from the fragment data set.

[0137] Compared with the prior art, this embodiment coordinates and manages the knowledge base 601, the configuration library 602, the document extraction module 501, the relationship matching module 502, and the extraction mapping module 503 through the relationship extraction processing unit 600, so as to enable the data extraction system 500 to run efficiently and obtain accurate relationship extraction results.

[0138] See also Figure 7 , Figure 7 is an exemplary structural block diagram of an electronic device of the data extraction method of the present application. Figure 7 As shown, the electronic device 700 of the present application may include a processor 701 and a memory 702, wherein the processor 701 and the memory 702 communicate with each other via a bus. The memory 702 stores program instructions for relationship extraction, and when the program instructions are executed by the processor 701, the method steps described in the above text in combination with the accompanying drawings are implemented:

[0139] Acquire multiple documents to be detected, and construct an entity relationship set of a target business; wherein the entity relationship set includes multiple triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities;

[0140] Extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected;

[0141] Using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, a target document of the target business is selected from the multiple documents to be detected;

[0142] The target document is divided into a plurality of associated segments, and relationship mapping is extracted for the associated segments according to each triple of the entity relationship set to obtain a plurality of segment data associated with each triple, thereby obtaining a segment data set of the target business.

[0143] Optionally, the processor 701 executes the step of using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set to select a target document of the target business from the multiple documents to be detected, including:

[0144] Performing feature extraction on each triple of the entity relationship set to obtain a second feature vector set of the entity relationship set;

[0145] Using the first feature vector set and the second feature vector set, obtaining the similarity between each document to be detected and the entity relationship set;

[0146] The document to be detected whose similarity is higher than a set threshold is determined as the target document.

[0147] Optionally, the processor 701 executes the step of obtaining the similarity between each document to be detected and the entity relationship set by using the first feature vector set and the second feature vector set, including:

[0148] Using the vector angle between the first feature vector set and the second feature vector set of the document to be detected;

[0149] The similarity between the document to be detected and the entity relationship set is determined based on the vector angle; wherein the similarity is negatively correlated with the vector angle.

[0150] Optionally, the processor 701 executes the step of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining a segment data set of the target business, including:

[0151] Dividing the target document into a plurality of associated segments, and obtaining segment vectors of the associated segments;

[0152] Relationship mapping is performed between the fragment vectors and the triples respectively to obtain a plurality of fragment data associated with the triples, thereby obtaining a fragment data set of the target service.

[0153] Optionally, the processor 701 executes the step of converting the associated fragment into a fragment vector, including:

[0154] Dividing the associated fragment into a plurality of sentences, and converting each sentence into a sentence vector respectively;

[0155] The sentence vector is used to perform semantic recognition on the sentence, and context sentence vectors with associated relationships are spliced ​​into the segment vector according to the semantic recognition result.

[0156] Optionally, after the processor 701 executes the steps of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining the segment data set of the target business, the process further includes:

[0157] Obtaining a data extraction instruction for at least one set target of the target business; wherein the extraction instruction carries an extraction condition for the set target;

[0158] The specific data set corresponding to the set target is extracted from the segment data set using the extraction condition.

[0159] Optionally, the processor 701 executes the extraction condition including a keyword, and a number threshold of the number of occurrences of the keyword and / or a probability threshold of the probability of the keyword appearing in each of the segment data;

[0160] The step of extracting a specific data set corresponding to a set target from the segment data set by using the extraction condition comprises:

[0161] Select from the fragment data set the fragment data in which the number of occurrences of the keyword exceeds the number threshold or / and select the fragment data in which the probability of the keyword appearing in the fragment data set is greater than the set probability, and determine the combination of the fragment data as the specific data set.

[0162] Optionally, the processor 701 executes a step of acquiring the probability of the keyword appearing in the segment data set, including:

[0163] Segmenting the fragment data to obtain word combinations of the fragment;

[0164] The number of occurrences and / or the probability of occurrence of the keyword in the word combination are counted to obtain the number of occurrences of the keyword in the segment data set and / or the probability of occurrence of the keyword in the segment data.

[0165] See also Figure 8 , Figure 8 is an exemplary structural block diagram of a computer-readable storage medium of the data extraction method of the present application. Figure 8 As shown, the computer-readable storage medium 800 stores a computer program 801. When the computer program 801 is executed on a computer, the computer executes the above-mentioned related method steps to implement a data extraction method in the above-mentioned embodiment.

[0166] The above scheme obtains multiple documents to be detected and constructs an entity relationship set of the target business; wherein the entity relationship set includes multiple triplets, each of which includes two entities related to the target business and the corresponding relationship between the two entities; extracts feature vectors from each of the documents to be detected to obtain the first feature vector set of each document to be detected; uses the first feature vector set to obtain the association relationship between each document to be detected and the entity relationship set, and selects the target document of the target business from the multiple documents to be detected; divides the target document into multiple associated fragments, and performs relationship mapping extraction on the associated fragments according to each triplet of the entity relationship set to obtain multiple fragment data associated with each triplet, and obtains a fragment data set of the target business. By using the entity relationship set of the target business to perform a preliminary screening of the feature vectors of the document before extracting the document to be detected, determining the target document from the messy document, and then performing relationship mapping extraction on the target document and the entity relationship set to obtain the fragment data set of the target business, the efficiency and accuracy of document relationship extraction can be effectively improved.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed methods, electronic devices and storage media can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0171] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data extraction method, characterized in that: The method comprises: Acquire multiple documents to be detected, and construct an entity relationship set of a target business; wherein the entity relationship set includes multiple triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities; Extracting feature vectors from each of the documents to be detected to obtain a first feature vector set for each of the documents to be detected; Using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set, a target document of the target business is selected from the multiple documents to be detected; The target document is divided into a plurality of associated segments, and relationship mapping is extracted for the associated segments according to each triple of the entity relationship set to obtain a plurality of segment data associated with each triple, thereby obtaining a segment data set of the target business.

2. The data extraction method according to claim 1, characterized in that: The step of using the first feature vector set to obtain the association relationship between each of the documents to be detected and the entity relationship set and selecting the target document of the target business from the multiple documents to be detected includes: Performing feature extraction on each triple of the entity relationship set to obtain a second feature vector set of the entity relationship set; Using the first feature vector set and the second feature vector set, obtaining the similarity between each document to be detected and the entity relationship set; The document to be detected whose similarity is higher than a set threshold is determined as the target document.

3. The data extraction method according to claim 2, characterized in that: The step of obtaining the similarity between each document to be detected and the entity relationship set by using the first feature vector set and the second feature vector set includes: Using the vector angle between the first feature vector set and the second feature vector set of the document to be detected; The similarity between the document to be detected and the entity relationship set is determined based on the vector angle; wherein the similarity is negatively correlated with the vector angle.

4. The data extraction method according to any one of claims 1 to 3, characterized in that: The step of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining a segment data set of the target business includes: Dividing the target document into a plurality of associated segments, and obtaining segment vectors of the associated segments; Relationship mapping is performed between the fragment vectors and the triples respectively to obtain a plurality of fragment data associated with the triples, thereby obtaining a fragment data set of the target service.

5. The data extraction method according to claim 4, characterized in that: The step of obtaining the fragment vector of the associated fragment comprises: Dividing the associated fragment into a plurality of sentences, and converting each sentence into a sentence vector respectively; Semantic recognition is performed on the sentence to obtain a semantic recognition result, and according to the semantic recognition result, the sentence vectors of the context with an associated relationship are spliced ​​into the segment vector.

6. The data extraction method according to claim 1, characterized in that: After the step of dividing the target document into a plurality of associated segments, performing relationship mapping extraction on the associated segments according to each triple of the entity relationship set, obtaining a plurality of segment data associated with each triple, and obtaining the segment data set of the target business, the step further includes: Obtaining a data extraction instruction for at least one set target of the target business; wherein the extraction instruction carries an extraction condition for the set target; The specific data set corresponding to the set target is extracted from the segment data set using the extraction condition.

7. The data extraction method according to claim 6, characterized in that: The extraction condition includes a keyword, and a frequency threshold of the number of occurrences of the keyword and / or a probability threshold of the probability of the keyword appearing in each of the segment data; The step of extracting a specific data set corresponding to a set target from the segment data set by using the extraction condition comprises: Select from the fragment data set the fragment data in which the number of occurrences of the keyword exceeds the number threshold or / and select the fragment data in which the probability of the keyword appearing in the fragment data set is greater than the set probability, and determine the combination of the fragment data as the specific data set.

8. The data extraction method according to claim 7, characterized in that: The step of obtaining the probability of the keyword appearing in the segment data set includes: Segmenting the fragment data to obtain word combinations of the fragment; The number of occurrences and / or the probability of occurrence of the keyword in the word combination are counted to obtain the number of occurrences of the keyword in the segment data set and / or the probability of occurrence of the keyword in the segment data.

9. A data extraction system, characterized in that: The data extraction system comprises: A document extraction module is used to obtain a plurality of documents to be detected and to construct an entity relationship set of a target business; wherein the entity relationship set includes a plurality of triples, each of which includes two entities related to the target business and a corresponding relationship between the two entities; feature vectors are extracted from each of the documents to be detected to obtain a first feature vector set of each of the documents to be detected; A relationship matching module, used to obtain the association relationship between each of the documents to be detected and the entity relationship set by using the first feature vector set, and select a target document of the target business from the multiple documents to be detected; The extraction and mapping module is used to divide the target document into multiple related fragments, perform relationship mapping extraction on the related fragments according to each triple of the entity relationship set, obtain multiple fragment data associated with each triple, and obtain the fragment data set of the target business.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the data extraction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the data extraction method according to any one of claims 1 to 8 is implemented.

Images