A cross-platform knowledge graph construction method and a construction system

Abstract

The application belongs to the technical field of data processing, and particularly relates to a cross-platform knowledge graph construction method and a construction system. The cross-platform knowledge graph construction method comprises the following steps: S1, after source data is obtained from a data source platform, the source data is subjected to data cleaning to obtain standard data; S2, the standard data is segmented into data segments; S3, standard entity pairs and standard entity relationship pairs in the data segments are obtained, and meanwhile, data verification and error tracing processing are performed according to the proportion of the standard entity relationship pairs relative to the standard entity pairs; and S4, the standard entity relationship pairs are sent into a graph database, a knowledge graph is constructed and stored. The application can efficiently and accurately and reliably process cross-platform data to construct a corresponding knowledge graph.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a cross-platform knowledge graph construction method and system. Background Technology

[0002] In today's digital age, knowledge graph construction technology plays an important role in many fields, but it also faces a series of limitations that urgently need to be addressed.

[0003] The information on the internet is complex and diverse, containing a large amount of redundant data, erroneous information, and incomplete data fragments. If this low-quality data is directly entered into the knowledge graph construction process, it will increase the difficulty of subsequent processing and reduce the accuracy and reliability of the constructed knowledge graph.

[0004] Furthermore, when errors are discovered during data processing, whether they occur during data acquisition or source data segmentation, we need to reprocess the data from scratch, which significantly reduces the efficiency of knowledge graph construction.

[0005] Therefore, there is an urgent need for a cross-platform knowledge graph construction method that can efficiently, accurately and reliably process cross-platform data and then construct the corresponding knowledge graph.

[0006] This application content

[0007] The purpose of this application is to overcome the shortcomings of the prior art and provide a cross-platform knowledge graph construction method that can efficiently and reliably process cross-platform data to construct the corresponding knowledge graph.

[0008] To achieve the above objectives, this application adopts the following technical solution:

[0009] A cross-platform knowledge graph construction method includes:

[0010] S1. After obtaining source data from the data source platform, the source data is cleaned to obtain standard data.

[0011] S2 divides the standard data into data segments;

[0012] S3, obtain standard entity pairs and standard entity relationship pairs in the data segment, and perform data verification and error tracing based on the proportion of standard entity relationship pairs relative to standard entity pairs in the data segment;

[0013] S4 inputs standard entity-relation pairs into the graph database, constructs a knowledge graph, and stores it.

[0014] Preferably, step S1 further includes the following sub-steps:

[0015] S11, retrieve source data from a specified data source platform via web crawler;

[0016] S12 involves performing data cleaning steps such as deduplication, outlier removal, format conversion, and encoding conversion on the newly acquired source data to obtain standard data.

[0017] Preferably, step S12 further includes the following sub-steps:

[0018] S121, compare the newly acquired source data with the existing source data in the first database. If the newly acquired source data already exists in the first database, discard the newly acquired source data; otherwise, store the newly acquired source data in the first database.

[0019] S1221, The range of normal values ​​is preset. After obtaining the newly added source data in the first database, the source data containing abnormal values ​​is removed to obtain normal source data. The normal source data is stored in the second database.

[0020] S1222, The standard format of the data is pre-defined, and the normal source data newly added in the second database is converted into the standard format and stored in the third database;

[0021] S1223, after converting the encoding of the newly added standard format data in the third database into the target format encoding through encoding conversion, it is stored in the fourth database.

[0022] Preferably, S2 also includes the following:

[0023] Determine the source platform of the newly added standard data in the fourth database, and perform corresponding segmentation processing based on the different source platforms of the newly added standard data;

[0024] If the new standard data comes from a social networking site, then a new standard data is divided into several data segments. Except for the last data segment, each data segment contains k1 characters. And except for the first data segment, the first k2 characters of each data segment overlap with the last k2 characters of the adjacent previous data segment; where k2 < k1.

[0025] If the new standard data originates from articles, a semantically aware sliding block technique based on the BERT-transformer model is used to segment each new standard data entry. Then, the quality of the segmented data segments is evaluated: the semantic similarity between any two sentences within a data segment is calculated. If the semantic similarity between all sentences within a data segment is greater than a set threshold k3, the current data segment is considered to have passed the quality evaluation. If all data segments in a new standard data entry pass the quality evaluation, the current new standard data segmentation is considered to be successful. If more than one data segment in a new standard data entry fails the quality evaluation, the current new standard data segmentation is considered unsuccessful, and the data segments of the current new standard data are re-segmented.

[0026] Preferably, step S3 further includes the following sub-steps:

[0027] S31, after obtaining all the standard entities in the data segment, construct the standard entity pair set of the corresponding data segment;

[0028] S32, The standard entity pairs and corresponding data segments are fed into the relation extraction model to obtain the corresponding entity relations and form standard entity relation pairs;

[0029] S33, if the proportion of standard entity relationship pairs in the current data segment is lower than the first proportion P1, then the thinking chain reasoning will be automatically triggered to perform data verification and error tracing on the entity relationship pairs obtained based on the current data segment.

[0030] Preferably, step S31 further includes the following sub-steps:

[0031] S131, after obtaining all entities in the data segment, replace the entity names with the corresponding standard names; record the entities corresponding to the standard entity names as standard entities;

[0032] S132, after removing duplicate standard entities from the current data segment, construct a set of standard entity pairs for the current data segment: any two different standard entities constitute a standard entity pair; each standard entity pair in the set of standard entity pairs for the current data segment is unique.

[0033] S32 also includes the following:

[0034] Let the standard entity pair be <first entity, second entity>. The relation extraction model predicts the entity relationship between the first entity and the second entity in the current standard entity pair based on the standard entity pair and the corresponding data segment text, and calculates the confidence of the predicted entity relationship. If the confidence is higher than the set threshold k4, the relation extraction model integrates the currently predicted entity relationship with the corresponding standard entity pair into a standard entity relationship pair in the form of a triple and outputs it. The standard entity relationship pair is in the form of a triple <first entity, entity relationship, second entity>.

[0035] Preferably, step S33 further includes the following sub-steps:

[0036] S331, The pre-set first prompt word and the text of the current data segment are fed into the LLM large language model. The LLM large language model outputs a set containing several triplet-form entity relation pairs.

[0037] S332, the parser extracts triplet-form entity relation pairs from the output of the LLM large language model. If the parser extracts more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, then proceed to the data verification stage of S333; if the parser does not extract more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, then the output of the LLM large language model, the prompt word, and the text of the current data segment are sent to the fifth database.

[0038] S333, for each triplet entity relation pair output by the LLM large language model with a confidence level higher than the set threshold k5, the relationship between entities in the triplet entity relation pair is verified: the two entities in the triplet entity relation pair and the pre-set second prompt word are sent to the data verification agent; the data verification agent will identify whether the data corresponding to the current entity can be regarded as an entity, and whether a relationship can be established between the data corresponding to the current entity.

[0039] If the recognition result of the data verification agent is that the data corresponding to the current entity cannot be regarded as an entity and / or a relationship cannot be established between the data corresponding to the current entity, the verification fails, and the error tracing process is performed on the triple form entity relationship that fails the verification.

[0040] If the recognition result of the data verification agent is that the data corresponding to the current entity can be regarded as an entity and a relationship can be established between the data corresponding to the current entity, then the verification is successful; the entity relationship pair in the form of a triplet that has been successfully verified is recorded as a standard entity relationship pair.

[0041] Preferably, the error tracing process includes S334 to S336:

[0042] S334: For the data segment corresponding to the triplet entity relation that failed the verification, execute the processing of S31 to S33. If the same data segment triggers data verification twice in a row and the verification fails, then execute S335.

[0043] S335: Re-execute the processing steps of S2 to S3 on the standard data to which the same data segment that has been triggered for data verification twice and failed the verification twice. If any data segment obtained after the same standard data has been re-segmented triggers data verification and fails the verification, then execute S336.

[0044] S336, the crawling address of the source data to which the standard data in S335 belongs is recorded as the original crawling address. Then, the corresponding source data is crawled again from the original crawling address and the operations of S1 to S3 are performed. If any data segment obtained from the source data crawled from the same original crawling address triggers data verification and the verification fails, the source data, standard data, segmented data segments and entity relationship pairs crawled during the verification failure process are packaged and sent to the fifth database, waiting for subsequent technical personnel to verify and analyze the fifth database.

[0045] This application also provides a cross-platform knowledge graph construction system, including: a cleaning module, a segmentation module, an entity pair and entity relationship pair acquisition module, a verification and error tracing module, and a graph database; the cleaning module cleans the source data obtained from the data source platform to obtain standard data, and then sends the standard data to the segmentation module; the segmentation module is used to segment the standard data into data segments; the entity pair and entity relationship pair acquisition module is used to obtain standard entity pairs and standard entity relationship pairs from the data segments, and then sends the standard entity pairs and standard entity relationship pairs to the verification and error tracing module, and sends the standard entity relationship pairs to the graph database; the verification and error tracing module is used to perform data verification and error tracing processing based on the proportion of standard entity relationship pairs relative to standard entity pairs in the data segments; the graph database is used to construct and store the knowledge graph based on the standard entity relationship pairs; each module / library is programmed or configured to perform the steps of the cross-platform knowledge graph construction method described above.

[0046] This application also provides a computer-readable storage medium, characterized in that: the computer-readable storage medium stores a computer program that is programmed or configured to perform a cross-platform knowledge graph construction method as described above.

[0047] The beneficial effects of this application are as follows:

[0048] (1) The cross-platform knowledge graph construction method of this application can efficiently and accurately and reliably process cross-platform data and then construct the corresponding knowledge graph.

[0049] (2) The error tracing process described in this application can locate the operation that caused the error in the shortest possible time, and correct errors that can self-correct. This not only improves the efficiency of locating erroneous operations, but also significantly reduces the amount of computation in the error correction process. Only errors that cannot be located through error tracing are sent to the fifth database for processing by technical personnel, thus greatly reducing the workload of manual error handling. The entire process forms a closed-loop feedback loop, with close collaboration and mutual feedback between each link, which can quickly locate and solve data processing problems in the process of building knowledge graphs, greatly improving the efficiency and quality of knowledge graph construction.

[0050] (3) Based on the ratio between standard entity relationship pairs and standard entity pairs, this application can detect whether errors occur in the process of obtaining standard entity pairs in a timely manner, and can efficiently locate erroneous operations and correct them by tracing the source of errors, thereby improving the overall construction efficiency of knowledge graph construction.

[0051] (4) Error tracing can ensure the data quality and the continuity and stability of the processing link in the construction of knowledge graph, salvage abnormal data to the greatest extent, and avoid blocking the overall process due to abnormal data in a single batch.

[0052] (5) This application reduces the problems of low efficiency and increased errors in knowledge graph construction caused by cross-platform data crawling by working together in crawling, cleaning, segmentation, data verification and error tracing. Attached Figure Description

[0053] Figure 1 This is a flowchart of a cross-platform knowledge graph construction method according to this application. Detailed Implementation

[0054] To make the technical solution of this application clearer and more explicit, the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the technical solution of this application without creative effort all fall within the protection scope of this application.

[0055] like Figure 1 The diagram shown is a flowchart of a cross-platform knowledge graph construction method according to this application, including the following steps:

[0056] S1. After obtaining source data from the data source platform, the source data is cleaned to obtain standard data.

[0057] S2 divides the standard data into data segments;

[0058] S3, obtain standard entity pairs and standard entity relationship pairs in the data segment, and perform data verification and error tracing based on the proportion of standard entity relationship pairs relative to standard entity pairs in the data segment;

[0059] S4 inputs standard entity-relation pairs into the graph database, constructs a knowledge graph, and stores it.

[0060] S1 also includes the following:

[0061] S11, retrieve source data from a specified data source platform via web crawler;

[0062] S12 involves performing data cleaning steps such as deduplication, outlier removal, format conversion, and encoding conversion on the newly acquired source data to obtain standard data.

[0063] In S11: Source data is obtained stably and efficiently from Wikipedia by using APIs combined with IP polling proxy pools; source data is obtained from Facebook and WeChat official accounts by using APIs combined with cookie pools and IP polling proxy pools; and source data is obtained from Twitter by using a combination of cookie pools, IP polling proxy pools, and account pools.

[0064] The above methods for obtaining source data from various designated data source platforms are existing technologies and will not be elaborated here.

[0065] S12 also includes the following:

[0066] S121, compare the newly acquired source data with the existing source data in the first database. If the newly acquired source data already exists in the first database, discard the newly acquired source data; otherwise, store the newly acquired source data in the first database.

[0067] S122, after obtaining the newly added source data in the first database, perform outlier removal, format conversion and encoding conversion to obtain the corresponding standard data.

[0068] S122 also includes the following:

[0069] S1221, Technicians pre-set the range of normal values, remove source data containing outliers, or correct outliers in the source data to the range of normal values, and record them as normal source data, and store the normal source data in the second database.

[0070] S1221 performs deduplication of source data to avoid repeated processing of the same data that has been repeatedly forwarded.

[0071] S1222, Technicians pre-set the standard format of the data, convert the newly added normal source data in the second database into the standard format, and then store it in the third database.

[0072] The third database contains all data in a standard format, which avoids incompatibility issues between the data and the processing system during subsequent data processing.

[0073] S1223, After converting the encoding of the newly added standard format data in the third database into the target format encoding through encoding conversion, it is stored in the fourth database; the fourth database stores the standard data.

[0074] Encoding conversion in S1223 can effectively solve the problem of garbled characters, ensure that characters are displayed correctly, and ensure that data can be read and processed correctly in the future.

[0075] The cleaning described in S12 and its sub-steps can effectively remove noisy and invalid data.

[0076] In S2: Determine the source platform of the newly added standard data in the fourth database, and perform corresponding segmentation processing based on the different source platforms of the newly added standard data;

[0077] If the new standard data originates from social networking sites (such as Facebook, Twitter, etc.), then a single piece of new standard data will be divided into several data segments. Except for the last data segment, each data segment contains k1 characters; and except for the first data segment, the first k2 characters of each data segment overlap with the last k2 characters of the adjacent preceding data segment; where k2 < k1. In other words, the last data segment obtained after dividing a piece of standard data may contain fewer than k1 characters.

[0078] If the new standard data originates from articles (such as Wikipedia, WeChat official accounts, etc.), then a semantically aware sliding block technique based on the BERT-transformer model is used to segment each new standard data entry. The quality of the segmented data segments is then evaluated: the semantic similarity between each pair of sentences within a data segment is calculated. If the semantic similarity between all sentences within a data segment is greater than a set threshold k3, then the current data segment is considered to have passed the quality evaluation. If all data segments in a new standard data entry pass the quality evaluation, then the segmentation of the current new standard data is deemed successful. If more than one data segment in a new standard data entry fails the quality evaluation, then the segmentation of the current new standard data is deemed unsuccessful, and the data segments of the current new standard data are re-segmented.

[0079] In this embodiment, k1=200, k2=20, and k3=0.8.

[0080] Optionally, if more than k4 consecutive instances of newly added standard data segmentation fail to meet the requirements, an alert will be sent to the technical staff. The technical staff will then retrain and iteratively optimize the BERT-transformer model to improve the segmentation performance.

[0081] When new standard data comes from social networking sites, its fragmented language style and concise content make it suitable for a recursive intelligent segmentation strategy. This strategy ensures a certain degree of overlap between adjacent data segments during the segmentation process. This not only ensures that clear local semantics (i.e., a specific data segment) are captured in subsequent processing, but also allows the data segments to be linked together through overlap, avoiding information fragmentation.

[0082] When new standard data originates from articles, its more complex text structure and tighter semantic connections require more refined processing. The semantically aware sliding segmentation technique based on the BERT-transformer model dynamically segments the text based on semantic understanding, leveraging the powerful representational capabilities of pre-trained language models to uncover deep semantics. Then, the quality of the segmented data is verified through quality assessment, ensuring that all sentences within each segment of the qualified standard data are semantically closely related, semantically coherent, and that the information within each segment is relatively complete.

[0083] In S2, based on the characteristics of data from different sources, different segmentation methods are precisely adapted to standard data: social posts pursue efficient segmentation and fast processing, while news or encyclopedia articles focus on semantic integrity and quality assessment.

[0084] Dividing standard data into several data segments can reduce the complexity of subsequent data processing (i.e., obtaining entity pairs and entity relationship pairs) and improve processing efficiency.

[0085] S3 also includes the following sub-steps:

[0086] S31, after obtaining all the standard entities in the data segment, construct the standard entity pair set of the corresponding data segment;

[0087] S32, the standard entity pairs and corresponding data segments are fed into the relation extraction model to obtain the corresponding entity relations and form standard entity relation pairs.

[0088] S31 also includes the following sub-steps:

[0089] S131, after obtaining all entities in the data segment, replace the entity names with the corresponding standard names; record the entities corresponding to the standard entity names as standard entities.

[0090] S132, after removing duplicate standard entities in the current data segment, construct a set of standard entity pairs for the current data segment: any two different standard entities constitute a standard entity pair; each standard entity pair in the set of standard entity pairs for the current data segment is unique.

[0091] Because an entity may have different entity names in the text, it is necessary to replace the entity name with the corresponding standard name in S131. In S131, an entity recognition model is used to obtain the entities in the data segment.

[0092] S32 also includes:

[0093] Let the standard entity pair be <first entity, second entity>. The relation extraction model predicts the entity relationship between the first entity and the second entity in the current standard entity pair based on the standard entity pair and the corresponding data segment text, and calculates the confidence level of the predicted entity relationship. If the confidence level is higher than the set threshold k4, the relation extraction model integrates the currently predicted entity relationship with the corresponding standard entity pair into a standard entity relationship pair in the form of a triple and outputs it. That is, the standard entity relationship pair is <first entity, entity relationship, second entity>.

[0094] Optionally, in S132, the two standard entities in the standard entity pair appear in the same sentence. Correspondingly, in S32, the relation extraction model predicts the entity relationship between the first and second entities in the current standard entity pair based on the standard entity pair and the corresponding sentence, and calculates the confidence level of the predicted entity relationship.

[0095] If the current entity pair does not have an entity relationship in the current data segment, then S32 will not output the corresponding standard entity relationship pair.

[0096] In this embodiment, k4=0.9.

[0097] Optionally, S33 may also be included after S32:

[0098] S33, if the proportion of standard entity relationship pairs in the current data segment is lower than the first proportion P1, then the thinking chain reasoning will be automatically triggered to perform data verification and error tracing on the entity relationship pairs obtained based on the current data segment.

[0099] S33 also includes the following:

[0100] S331, the pre-set first prompt word and the text of the current data segment are fed into the LLM large language model. The LLM large language model outputs a set containing several entity relation pairs in the form of triples.

[0101] S332: The parser extracts triplet-form entity relation pairs from the output of the LLM large language model. If the parser extracts more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, the process proceeds to the data verification stage in S333. If the parser does not extract more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, the prompt word and the text of the current data segment are sent to the fifth database for subsequent verification and analysis by technical personnel.

[0102] S333, for each triplet-form entity relation pair output by the LLM large language model with a confidence level higher than the set threshold k5, the relationships between entities are verified: the two entities in the triplet-form entity relation pair and the pre-set second prompt word are sent to the data verification agent; the data verification agent will identify whether the data corresponding to the current entity can be regarded as an entity, and whether a relationship can be established between the data corresponding to the current entity; if the identification result of the data verification agent is that the data corresponding to the current entity cannot be regarded as an entity and / or a relationship cannot be established between the data corresponding to the current entity, then the current entity relation pair verification fails; if the identification result of the data verification agent is that the data corresponding to the current entity can be regarded as an entity and a relationship can be established between the data corresponding to the current entity, then the current entity relation pair verification succeeds; only when all triplet-form entity relation pairs with a confidence level higher than the set threshold k5 corresponding to the current data segment are successfully verified, then the current data segment is successfully verified; error tracing is performed on the data segments that fail verification; the successfully verified entity relation pairs in the successfully verified data segments are recorded as standard entity relation pairs.

[0103] Data verification agents are composite AI applications with pre-trained LLM models as key components. As this is an existing technology, it will not be elaborated upon here.

[0104] The parser used in this embodiment is a parser composed of regular expressions; the first ratio P1 = 0.1.

[0105] Error tracing and handling include S334 to S336:

[0106] S334: For data segments that fail verification, execute the processing steps of S31 to S33. If the same data segment triggers data verification twice consecutively and the verification fails, then execute S335.

[0107] S335: Re-execute the processing steps of S2 to S3 on the standard data to which the same data segment that has been triggered for data verification twice and failed the verification twice. If any data segment obtained after the same standard data has been re-segmented triggers data verification and fails the verification, then execute S336.

[0108] S336, the crawling address of the source data to which the standard data in S335 belongs is recorded as the original crawling address. Then, the corresponding source data is crawled again from the original crawling address and the operations of S1 to S3 are performed. If any data segment obtained from the source data crawled from the same original crawling address triggers data verification and the verification fails, the source data, standard data, segmented data segments and entity relationship pairs crawled during the verification failure process are packaged and sent to the fifth database, waiting for subsequent technical personnel to verify and analyze the fifth database.

[0109] In S4: Constructing knowledge graphs based on standard entity relationships is an existing technique, which will not be elaborated here.

[0110] The graph database used in this embodiment is the Neo4j graph database.

[0111] This application presents a cross-platform knowledge graph construction method that can efficiently, accurately, and reliably process cross-platform data to construct the corresponding knowledge graph.

[0112] In the construction method of this application, if there are many standard entity pairs in a data segment, but few corresponding standard entity relationship pairs, it indicates that some standard entity relationship pairs may have been missed, or the standard entity pairs may have been obtained incorrectly (there are actually not that many). Errors may occur in the process of obtaining standard entity relationship pairs, and there are many steps that may cause errors. Therefore, this application needs to trigger the thinking chain reasoning to perform data verification and error tracing on the entity relationship pairs obtained based on the current data segment.

[0113] This application, based on the first prompt word and the LLM large language model, re-acquires all entity relation pairs in the current data segment (these newly acquired entity relation pairs may include standard entity relation pairs). If the confidence of these re-acquired entity relation pairs is lower than the set threshold k5, it indicates that the accuracy of these re-acquired entity relation pairs is very low. Therefore, the output of the LLM large language model, the prompt word, and the text of the current data segment need to be sent to the fifth database for subsequent verification and analysis by technical personnel. If more than one of these re-acquired entity relation pairs has a confidence higher than the set threshold k5, the entity relation pairs in the data segment with a confidence higher than the set threshold k5 are verified, and the successfully verified entity relation pairs in the data segment are recorded as standard entity relation pairs. If the verification of the data segment fails, the error source is traced by progressively expanding the scope of reprocessing.

[0114] During the error tracing process, step S334 shows that the initial tracing back leads to the standard entity extraction and standard entity pair acquisition operations of the data segment. This involves re-acquiring the standard entity extraction and standard entity pairs of a data segment, and then recalculating the ratio between standard entity relationship pairs and standard entity pairs. If the ratio exceeds the first proportion P1, it indicates that an error may have occurred when acquiring the standard entity pairs for this data segment, and this error has been corrected by re-execution. Therefore, the newly obtained standard entity relationship pairs are valid and can be directly retained for constructing subsequent knowledge graphs. If the ratio is still lower than the first proportion P1, the thought chain reasoning will still trigger data verification, and the successfully verified entity relationship pairs in the data segment will be recorded as standard entity relationship pairs. If the data segment fails verification again (i.e., the same data segment triggers data verification twice consecutively and fails), the problem may lie in the data segment itself, meaning the reason might be an unreasonable segmentation of the standard data. Therefore, we proceed to the previous level of operation for error tracing.

[0115] The previous operation involved splitting a standard dataset, resulting in data segments that failed validation. We expand the reprocessing scope to the previous operation, re-splitting the standard dataset (S335). If this re-splitting corrects errors caused by improper splitting and no subsequent data validation is triggered, or if all split data segments pass validation, we can directly retain the newly obtained standard entity relationship pairs for constructing the subsequent knowledge graph. If any data segment obtained after re-splitting the same standard dataset triggers validation and fails, the problem may lie in the source data, meaning the source data being crawled might be incorrect. Therefore, we continue tracing the error back to the previous operation.

[0116] Re-execute the previous step, i.e., re-crawl new source data from the original crawling address, which is equivalent to executing S336. If we correct the error by re-crawling the source data and no subsequent data validation is triggered, or if all data segments after segmentation are successfully validated, we can directly retain the newly obtained standard entity-relation pairs for constructing the subsequent knowledge graph. If any data segment obtained from the re-crawled source data triggers data validation and fails, the problem is beyond the scope of error tracing and self-correction. In this case, the source data, standard data, segmented data segments, and entity-relation pairs crawled during this validation failure process need to be packaged and sent to the fifth database for subsequent verification and analysis by technical personnel.

[0117] This error tracing process can pinpoint the operation causing the error in the shortest possible time and correct errors that can self-correct. This not only improves the efficiency of locating erroneous operations but also significantly reduces the computational load during error correction. Only errors that cannot be located through error tracing are sent to the fifth database for processing by technical personnel, thus greatly reducing the workload of manual error handling. The entire process forms a closed-loop feedback loop, with close collaboration and mutual feedback between each link, enabling rapid location and resolution of data processing problems in the knowledge graph construction process, greatly improving the efficiency and quality of knowledge graph construction.

[0118] This application can detect errors in the process of acquiring standard entity pairs in a timely manner based on the ratio between standard entity relationship pairs and standard entity pairs. Furthermore, it can efficiently locate erroneous operations and correct them automatically through error tracing, thereby improving the overall construction efficiency of knowledge graph construction.

[0119] Error tracing can ensure the data quality, continuity, and stability of the processing chain in building knowledge graphs, maximize the recovery of abnormal data, and prevent the overall process from being blocked due to abnormal data in a single batch.

[0120] This application reduces the problems of low efficiency and increased errors in knowledge graph construction that may be caused by cross-platform data crawling by working together in crawling, cleaning, segmentation, data verification, and error tracing.

[0121] This invention achieves a breakthrough technological leap in the field of knowledge graph construction, with key performance indicators surpassing existing technologies: cross-platform relationship extraction accuracy is significantly improved, substantially optimizing entity relationship recognition precision; data processing throughput reaches a new industry high of 950 records / second, improving efficiency by more than 5 times; the error localization mechanism achieves rapid response, reducing diagnostic time from minutes to within 5 seconds, optimizing efficiency by 80.7%; and the data deduplication rate is significantly improved to 99.3%, effectively solving the redundancy problem in large-scale data processing. These synergistic breakthroughs in core indicators enhance knowledge automation processing technology.

[0122] This application also provides a cross-platform knowledge graph construction system, including: a cleaning module, a segmentation module, an entity pair and entity relationship pair acquisition module, a verification and error tracing module, and a graph database;

[0123] The cleaning module cleans the source data obtained from the data source platform to obtain standard data, and then sends the standard data into the segmentation module.

[0124] The segmentation module is used to divide standard data into data segments;

[0125] The entity pair and entity relationship pair acquisition module is used to obtain standard entity pairs and standard entity relationship pairs from the data segment, and then send the standard entity pairs and standard entity relationship pairs to the verification and error tracing module, and send the standard entity relationship pairs to the graph database.

[0126] The verification and error tracing module is used to perform data verification and error tracing processing based on the proportion of the data relative to the standard entity pairs in the data segment according to the standard entity relationships.

[0127] Graph databases are used to construct and store knowledge graphs based on standard entity relationships.

[0128] Each module / library is programmed or configured to perform the steps of a cross-platform knowledge graph construction method as described above.

[0129] This application also provides a computer-readable storage medium storing a computer program that is programmed or configured to perform a cross-platform knowledge graph construction method as described above.

[0130] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the cross-platform knowledge graph construction method described above.

[0131] The technologies, shapes, and structures not described in detail in this application are all well-known technologies. It should also be noted that the above are merely preferred embodiments of this application and are not intended to limit the scope of this application. The components or steps in the embodiments of this application can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered as equivalent solutions of this application and should all fall within the protection scope of this application.

Claims

1. A cross-platform knowledge graph construction method, characterized in that, Includes the following steps: S1. After obtaining source data from the data source platform, the source data is cleaned to obtain standard data. S2 divides the standard data into data segments; S3, obtain standard entity pairs and standard entity relationship pairs in the data segment, and perform data verification and error tracing based on the proportion of standard entity relationship pairs relative to standard entity pairs in the data segment; S4. Input the standard entity relationship pairs into the graph database, construct a knowledge graph and store it; S3 also includes the following sub-steps: S31, after obtaining all the standard entities in the data segment, construct the standard entity pair set of the corresponding data segment; S32, The standard entity pairs and corresponding data segments are fed into the relation extraction model to obtain the corresponding entity relations and form standard entity relation pairs; S33, If the proportion of standard entity relationship pairs in the current data segment is lower than the first proportion P1, the thinking chain reasoning will be automatically triggered to perform data verification and error tracing on the entity relationship pairs obtained based on the current data segment. S31 also includes the following sub-steps: S131, after obtaining all entities in the data segment, replace the entity names with the corresponding standard names; record the entities corresponding to the standard entity names as standard entities; S132, after removing duplicate standard entities from the current data segment, construct a set of standard entity pairs for the current data segment: any two different standard entities constitute a standard entity pair; each standard entity pair in the set of standard entity pairs for the current data segment is unique. S32 also includes the following: Let the standard entity pair be <first entity, second entity>. The relation extraction model predicts the entity relationship between the first entity and the second entity in the current standard entity pair based on the standard entity pair and the corresponding data segment text, and calculates the confidence level of the predicted entity relationship. If the confidence level is higher than the set threshold k4, the relation extraction model integrates the currently predicted entity relationship with the corresponding standard entity pair into a standard entity relationship pair in the form of a triple and outputs it. The standard entity relationship pair is in the form of a triple <first entity, entity relationship, second entity>. S33 also includes the following sub-steps: S331, The pre-set first prompt word and the text of the current data segment are fed into the LLM large language model. The LLM large language model outputs a set containing several triplet-form entity relation pairs. S332, the parser extracts triplet-form entity relation pairs from the output of the LLM large language model. If the parser extracts more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, then proceed to the data verification stage of S333; if the parser does not extract more than one triplet-form entity relation pair with a confidence level higher than the set threshold k5 from the output of the LLM large language model, then the output of the LLM large language model, the prompt word, and the text of the current data segment are sent to the fifth database. S333, verify the relationship between entities in each triplet entity relationship pair output by the LLM large language model with a confidence level higher than the set threshold k5: send the two entities in the triplet entity relationship pair and the pre-set second prompt word into the data verification agent. The data verification agent will identify whether the data corresponding to the current entity can be considered an entity, and whether a relationship can be established between the data corresponding to the current entity. If the recognition result of the data verification agent is that the data corresponding to the current entity cannot be regarded as an entity and / or a relationship cannot be established between the data corresponding to the current entity, then the verification of the current entity relationship pair fails. If the recognition result of the data verification agent is that the data corresponding to the current entity can be regarded as an entity and a relationship can be established between the data corresponding to the current entity, then the current entity relationship pair is successfully verified; only if all triplet-form entity relationship pairs corresponding to the current data segment with a confidence level higher than the set threshold k5 are successfully verified, then the current data segment is successfully verified. Perform error tracing for data segments that fail verification; The entity relation pairs that are successfully verified in the data segment are recorded as standard entity relation pairs. Error tracing and handling include S334 to S336: S334: For data segments that fail verification, execute the processing of S31 to S33. If the same data segment triggers data verification twice consecutively and the verification fails, then execute S335. S335, re-execute the processing of the same data segment to which the data verification was triggered twice and failed twice. If any data segment obtained after the same standard data is re-segmented triggers data verification and fails, then execute S336. S336, the crawling address of the source data to which the standard data in S335 belongs is recorded as the original crawling address. Then, the corresponding source data is crawled again from the original crawling address and the operations of S1 to S3 are performed. If any data segment obtained from the source data crawled from the same original crawling address triggers data verification and the verification fails, the source data, standard data, segmented data segments and entity relationship pairs crawled during the verification failure process are packaged and sent to the fifth database, waiting for subsequent technical personnel to verify and analyze the fifth database. 2.The cross-platform knowledge graph construction method of claim 1, wherein, S1 also includes the following sub-steps: S11, retrieve source data from a specified data source platform via web crawler; S12 involves performing data cleaning steps such as deduplication, outlier removal, format conversion, and encoding conversion on the newly acquired source data to obtain standard data. 3.The cross-platform knowledge graph construction method of claim 2, wherein, S12 also includes the following sub-steps: S121, compare the newly acquired source data with the existing source data in the first database. If the newly acquired source data already exists in the first database, discard the newly acquired source data; otherwise, store the newly acquired source data in the first database. S1221, The range of normal values ​​is preset. After obtaining the newly added source data in the first database, the source data containing abnormal values ​​is removed to obtain normal source data. The normal source data is stored in the second database. S1222, The standard format of the data is pre-defined, and the normal source data newly added in the second database is converted into the standard format and stored in the third database; S1223, after converting the encoding of the newly added standard format data in the third database into the target format encoding through encoding conversion, it is stored in the fourth database. 4.The cross-platform knowledge graph construction method of claim 3, wherein, S2 also includes the following: Determine the source platform of the newly added standard data in the fourth database, and perform corresponding segmentation processing based on the different source platforms of the newly added standard data; If the new standard data comes from social networking sites, then a new standard data is divided into several data segments. Except for the last data segment, each data segment contains k1 characters; and except for the first data segment, the first k2 characters of each data segment overlap with the last k2 characters of the adjacent previous data segment. Where k2 < k1; If the new standard data originates from articles, a semantically aware sliding block technique based on the BERT-transformer model is used to segment each new standard data entry. Then, the quality of the segmented data segments is evaluated: the semantic similarity between any two sentences within a data segment is calculated. If the semantic similarity between all sentences within a data segment is greater than a set threshold k3, the current data segment is considered to have passed the quality evaluation. If all data segments in a new standard data entry pass the quality evaluation, the current new standard data segmentation is considered to be successful. If more than one data segment in a new standard data entry fails the quality evaluation, the current new standard data segmentation is considered unsuccessful, and the data segments of the current new standard data are re-segmented. 5.A cross-platform knowledge graph construction system, characterized in that, include: The system includes a data cleaning module, a segmentation module, an entity pair and entity relationship pair acquisition module, a verification and error tracing module, and a graph database. The data cleaning module cleans the source data obtained from the data source platform to obtain standard data, and then sends the standard data into the segmentation module. The segmentation module is used to divide standard data into data segments; The entity pair and entity relationship pair acquisition module is used to obtain standard entity pairs and standard entity relationship pairs from the data segment, and then send the standard entity pairs and standard entity relationship pairs to the verification and error tracing module, and send the standard entity relationship pairs to the graph database; the verification and error tracing module is used to perform data verification and error tracing processing based on the proportion of standard entity relationship pairs relative to standard entity pairs in the data segment. Graph databases are used to construct and store knowledge graphs based on standard entity relationships; Each module / library is programmed or configured to perform the steps of a cross-platform knowledge graph construction method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform a cross-platform knowledge graph construction method as described in any one of claims 1-4.

Images