Supply chain financial heterogeneous data cleaning and fusion processing system

Abstract

The invention discloses a supply chain finance heterogeneous data cleaning and fusion processing system, and relates to the technical field of supply chain finance data management, and the system comprises a heterogeneous data processing platform which is in communication connection with the following modules: a data obtaining module which is used for obtaining heterogeneous data from each participant of supply chain finance, comprising structured data, semi-structured data and unstructured data, and preprocessing the obtained multi-source heterogeneous data; and the knowledge graph construction module is used for performing semantic annotation on entities and relationships in the heterogeneous data and constructing a supply chain finance knowledge graph model. According to the method, modeling is carried out through semantic annotation and relation extraction, the incidence relation between heterogeneous data is defined, then semantic alignment and fusion processing are carried out, data from different sources are mapped into a unified knowledge graph framework, the accuracy and reliability of data fusion are improved, and deeper data insight is provided for supply chain financial services.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of supply chain financial data management, and particularly relates to a supply chain financial heterogeneous data cleaning and fusion processing system. BACKGROUND

[0002] Supply chain finance involves multiple participants, including suppliers, core enterprises, financial institutions, logistics enterprises, etc. The systems and data formats used by these subjects are different, and the business scenarios of supply chain finance are complex, covering multiple links such as procurement, production, sales, and logistics. Each link has its specific data requirements and analysis goals. The data quality of different sources differs greatly, and there are problems such as duplication, errors, and missing data. Data cleaning is a key step to solve these problems. By removing duplicate or incorrect data and standardizing the format, the quality of the data can be improved.

[0003] In the prior art, due to the different parties involved in supply chain finance, the technology stacks, data formats, and database systems used are different. When integrating data, only the surface features and basic attributes of the data are often focused on, making it difficult to deeply mine the potential association relationships between data. Moreover, different sources of data have semantic differences, making it difficult to effectively handle semantic differences, resulting in a decrease in the accuracy of data fusion. Therefore, how to use a knowledge graph to semantically annotate entities and relationships, clarify the meaning of each entity and relationship, show the association relationships between data, and solve the problem of semantic differences in data, thereby improving the accuracy of data fusion, is a problem to be solved by the present application. For this reason, a supply chain financial heterogeneous data cleaning and fusion processing system is proposed. SUMMARY

[0004] The present application aims to provide a supply chain financial heterogeneous data cleaning and fusion processing system to solve the problems raised in the background art.

[0005] To solve the above technical problems, the technical solution adopted by the present application is:

[0006] A supply chain financial heterogeneous data cleaning and fusion processing system, comprising a heterogeneous data processing platform, characterized in that the heterogeneous data processing platform is communicatively connected with the following modules, wherein:

[0007] A data acquisition module for acquiring heterogeneous data from each participant in supply chain finance, including structured data, semi-structured data, and unstructured data, and pre-processing the acquired multi-source heterogeneous data to ensure that all types of data in supply chain finance business can be fully acquired;

[0008] A knowledge graph construction module for semantically annotating entities and relationships in heterogeneous data and constructing a knowledge graph model of supply chain finance to clarify the association relationships between heterogeneous data.

[0009] The data fusion processing module is configured to perform semantic alignment on the heterogeneous data according to the knowledge graph model and the semantic annotation information, and perform fusion processing and data correlation analysis, and mine potential correlation between the data.

[0010] The data application management module is configured to apply the fused data to specific business scenarios of the supply chain finance.

[0011] The further improvement of the technical scheme of the application is that the data acquisition module specifically comprises:

[0012] The data sources of each participant in the supply chain finance are identified to ensure that all relevant data types are covered, and the data sources are divided into structured data, semi-structured data and unstructured data, and then according to the type of the data source, a corresponding access mode is selected to establish a connection with each participant in the supply chain finance;

[0013] Data is collected from each data source at a predetermined frequency and in a predetermined manner, wherein for real-time data, real-time pulling is performed through an API or a message queue, and for static data, file transfer or database query is performed periodically to obtain the data, and then the collected data is preliminarily processed, including removing duplicate data, correcting obvious errors, filling in missing values, and for unstructured data, format conversion is performed, and at the same time, the collected data is preliminarily annotated to record metadata information of the source, type and collection time of the data;

[0014] The collected multi-source heterogeneous data is preprocessed, different formats of data are converted into a unified intermediate format, and the logical consistency of the data is checked, for inconsistent data, correction or marking is performed, and then data from different participants and of different types are integrated to construct a complete supply chain finance data view.

[0015] The further improvement of the technical scheme of the application is that the knowledge graph construction module comprises a semantic annotation unit and a relationship extraction modeling unit.

[0016] The semantic annotation unit is configured to identify entities in the heterogeneous data by using natural language processing technology, and perform semantic annotation on the entities to clearly define the meaning and attributes of each entity.

[0017] The relationship extraction modeling unit constructs a knowledge graph model of the supply chain finance based on the results of the semantic annotation, defines a hierarchical structure and rules of entities, relationships and attributes, and displays the correlation between the data to help users better understand the complex relationships in the supply chain finance business.

[0018] The further improvement of the technical scheme of the application is that the semantic annotation unit specifically comprises:

[0019] The text information in the heterogeneous data in the supply chain finance is preprocessed, irrelevant characters, special symbols and redundant spaces are removed through text cleaning, the neatness of the text is ensured, the text format is unified, the text of different sources is converted into a unified coding format, and according to the language characteristics of Chinese or English, a corresponding word segmentation tool is used for word segmentation processing, for Chinese text, the word segmentation tool is used for word segmentation, for English text, the NLTK word segmentation tool is used, and then the text is split into a word sequence, at the same time, part-of-speech tagging is performed, and each word is tagged with its part-of-speech;

[0020] Based on the preprocessed text information, entity recognition is performed using natural language processing technology, a BiLSTM-CRF model based on deep learning is used to identify key entities in the text, including enterprise name, product name, amount, date and location, and in the identification process, the context information, part-of-speech and grammar structure of the word are analyzed, the entity boundary is judged, after the entity is identified, preliminary semantic tagging is performed, and each entity is assigned a general category label to clearly define the approximate meaning and category of the entity;

[0021] On the basis of preliminary tagging, the attribute information of the entity is further mined, and deep semantic tagging is performed, the semantic relationship of the text where the entity is located is analyzed, the domain knowledge base is combined, the association between entities and the attributes of the entity itself are determined.

[0022] The further improvement of the technical scheme of the application is that the relationship extraction modeling unit specifically comprises:

[0023] According to the semantic tagging result, the core entities of the supply chain finance knowledge graph are determined, including core enterprises, suppliers, financial institutions, logistics enterprises, products and transaction orders, the basic attributes of each core entity are defined, and the relationship types between the core entities are sorted out, and then the concept model of the knowledge graph is constructed;

[0024] The data after semantic tagging is mapped and converted according to the concept model of the knowledge graph, and the extracted entities and relationships are integrated to form the knowledge graph model of the supply chain finance;

[0025] The constructed knowledge graph is visualized by using the graphical interface of Neo4j, in the visualization interface, the entities are presented in the form of nodes, the relationships are presented in the form of edges, the entity types and relationship types are distinguished by different colors, shapes and labels, the association between data is displayed, at the same time, the knowledge graph is optimized, whether there are duplicate entities and incorrect relationships is checked, through data cleaning technology, the duplicate nodes and edges are removed, and through a rule correction method, the incorrect relationship attributes are corrected.

[0026] The further improvement of the technical scheme of the application lies in that the construction process of the knowledge graph model of the supply chain finance is:

[0027] For the structured data after semantic annotation, entity and relationship information is directly extracted according to the pre-constructed supply chain finance knowledge graph concept model, the node position of each entity in the knowledge graph is determined, and the connection mode of the relationship between the nodes is determined, and the extracted entity and relationship information is accurately filled into the corresponding position of the knowledge graph;

[0028] For semi-structured data and unstructured data, natural language processing technology is used to further analyze the entity and relationship information therein, wherein, for semi-structured data, a file parsing tool is used to extract key information, and for unstructured data, OCR technology and NLP technology are combined to extract entities and relationships, the parsed entities and relationships are compared and integrated with the concept model, for the part matching the existing entities and relationships in the concept model, corresponding filling is directly performed, and for new entities and relationships, adding is performed according to the expansion rules of the concept model, so that the semi-structured data and unstructured data are integrated into the knowledge graph;

[0029] The entities and relationships extracted from the structured data, semi-structured data and unstructured data are integrated, different sources of data are uniformly mapped into the model based on the concept model of the knowledge graph, the consistency of the entities and relationships is ensured, and meanwhile, the edges of the knowledge graph are constructed according to the relationship types between the entities, so as to form a complete supply chain finance knowledge graph model.

[0030] The further improvement of the technical scheme of the application lies in that the data fusion processing module comprises a semantic alignment unit and a data fusion unit;

[0031] The semantic alignment unit matches the entities in different source data based on the entity and relationship annotation in the knowledge graph model, calculates the entity similarity index, compares the entity similarity index with a preset similarity threshold, identifies the same or similar entities, and performs semantic alignment on the data of different sources;

[0032] The data fusion unit is used for fusing the data after semantic alignment, integrating the scattered data into a unified data structure according to the association relationship in the knowledge graph, and performing data correlation analysis in combination with the fused data and the knowledge graph model.

[0033] The further improvement of the technical scheme of the application lies in that the semantic alignment unit specifically comprises:

[0034] According to the constructed supply chain financial knowledge graph model, based on the entity annotation, the entities in the data of different sources are matched, the entity identifiers are extracted from each data source, and the text similarity calculation method is used to calculate the entity similarity index, compared with the preset similarity threshold, the same or similar entities are identified, and then the same or similar entities are mapped to the corresponding node position in the unified knowledge graph framework according to the semantic annotation in the knowledge graph model;

[0035] On the basis of completing entity alignment, the relationships in the data of different sources are aligned, the relationship between entities in each data source is analyzed according to the relationship type and rule defined in the knowledge graph model, and the relationship is matched with the relationship in the knowledge graph, and the semantic consistency of the relationship is checked, whether the attributes of the relationship are consistent is checked, if there is inconsistency, the reason is further analyzed and modified or marked, the semantic consistency of the relationship is ensured, and the reliability of data fusion is improved;

[0036] The data after entity matching and relationship alignment are fused, the heterogeneous data of different sources are integrated into the unified knowledge graph framework, a complete data structure is formed, the repeated data of the same entity or relationship is averaged or merged, data redundancy is avoided, and the semantic information of the data is further integrated according to the semantic annotation and association rule in the knowledge graph model, and the semantic difference between the data sources is eliminated.

[0037] The further improvement of the technical scheme of the application is that the data fusion unit specifically comprises:

[0038] The data after semantic alignment is integrated according to the association relationship in the knowledge graph model, and the data in the scattered data sources is mapped into a unified data structure according to the entities and relationships defined in the knowledge graph;

[0039] The integrated data is fused to eliminate repeated data, wherein, for repeated data of the same entity or relationship, the data quality is evaluated, the optimal data is selected for retention by factors such as data source reliability, update frequency and accuracy, and for repeated relationship data, a unique identifier is used to identify whether it is a repeated record, and a de-duplication process is performed to ensure the uniqueness of the data and avoid data redundancy, thereby improving the usability of the data;

[0040] The data after fusion and the knowledge graph model are combined to perform data association analysis, the association relationship in the knowledge graph is used to mine the potential relationship between the data, and the integrated data is optimized to check the integrity and accuracy of the data and correct the existing errors or missing information.

[0041] The further improvement of the technical scheme of the application is that the data application management module specifically comprises:

[0042] The fused data is docked to specific business scenarios of supply chain finance, data requirements of different business scenarios are analyzed, and corresponding data sets are extracted from the fused data according to the data requirements, and preprocessed to ensure that the data can be directly applied to the business system;

[0043] A data dynamic updating mechanism is established, an updating plan is formulated according to the updating frequency of the data source and the business requirement, and the updated data is checked and verified to ensure the integrity and consistency of the data;

[0044] The data is classified and managed, different access permissions are set according to the sensitivity of the data and the business requirement, so that only authorized users can access the corresponding data, and a data audit mechanism is established to record the access and operation logs of the data.

[0045] Due to the adoption of the above technical solutions, the technical progress achieved by the present application relative to the prior art is:

[0046] 1. The present application provides a supply chain finance heterogeneous data cleaning and fusion processing system, which clearly defines the association between heterogeneous data through semantic annotation and relationship extraction modeling, solves the problem of semantic difference between data, and then performs semantic alignment and fusion processing, maps data from different sources to a unified knowledge graph framework, eliminates semantic ambiguity, improves the accuracy and reliability of data fusion, and provides deeper data insight for supply chain finance business.

[0047] 2. The present application provides a supply chain finance heterogeneous data cleaning and fusion processing system, which effectively removes duplicate and erroneous data, fills in missing values, and converts data of different formats to a unified intermediate format through comprehensive collection and preprocessing of multi-source heterogeneous data, ensures data quality and consistency, avoids analysis result deviation caused by data problems, and improves the accuracy and reliability of supply chain finance business. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0049] Figure 1 The workflow diagram of the present application;

[0050] Figure 2 The workflow diagram of the semantic alignment unit of the present application. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments but not all of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] As shown in Embodiment 1, Figure 1 , Figure 2 The present application provides a supply chain finance heterogeneous data cleaning and fusion processing system, which comprises a heterogeneous data processing platform, and the heterogeneous data processing platform is communicatively connected with the following modules, wherein:

[0053] The heterogeneous data processing platform is the center of the whole system, responsible for coordinating the operation of each module, realizing the flow, processing and management of data, connected with each module through a communication interface, ensuring efficient transmission and interaction of data between modules;

[0054] The data acquisition module is used for acquiring heterogeneous data from each participant of the supply chain finance, including structured data, semi-structured data and unstructured data, and pre-processing the acquired multi-source heterogeneous data, ensuring that all types of data in the supply chain finance business can be fully acquired, improving the quality and consistency of the data, avoiding inaccurate analysis results due to data missing, identifying the data sources of each participant of the supply chain finance, ensuring that all related data types are covered, and dividing the data sources into structured data, semi-structured data and unstructured data, and then selecting the corresponding access mode to establish a connection with each participant of the supply chain finance according to the type of the data source, wherein for structured data, including the financial statements and transaction records of enterprises, the structured data is directly extracted through a database interface, for semi-structured data, including logistics documents and invoices, the semi-structured data is captured by using a file parsing tool, and for unstructured data, including contract texts and picture reports, the unstructured data is identified and collected by means of OCR and NLP technologies, and the data is collected from each data source at a predetermined frequency and in a predetermined manner, wherein for real-time data, the real-time data is pulled in real time through an API or a message queue, and for static data, the static data is acquired periodically through file transmission or database query, and then the collected data is preliminarily processed, including removing duplicate data, correcting obvious errors and filling missing values, for unstructured data, format conversion is performed, and at the same time, the collected data is preliminarily labeled to record the metadata information of the source, type and collection time of the data, the collected multi-source heterogeneous data is pre-processed to convert data in different formats into a unified intermediate format, and the logical consistency of the data is checked, for inconsistent data, the inconsistent data is corrected or marked, and then data from different participants and of different types is integrated to build a complete supply chain finance data view;

[0055] a knowledge graph construction module, configured to perform semantic annotation on entities and relationships in the heterogeneous data, and construct a knowledge graph model of the supply chain finance, and clarify the association between the heterogeneous data, the knowledge graph construction module comprising a semantic annotation unit and a relationship extraction modeling unit;

[0056] The semantic annotation unit is configured to identify entities in the heterogeneous data by using natural language processing technology, and perform semantic annotation thereon, clarify the meaning and attributes of each entity, and preprocess text information in the heterogeneous data in the supply chain finance, remove irrelevant characters, special symbols and redundant spaces and the like by text cleaning, ensure the neatness of the text, unify the text format, convert text from different sources into a unified encoding format, and use a corresponding word segmentation tool for word segmentation processing according to the language characteristics of Chinese or English, for Chinese text, use the Jieba word segmentation tool for word segmentation, and for English text, use the NLTK word segmentation tool, and then split the text into a sequence of words, at the same time, perform part-of-speech tagging to tag the part-of-speech of each word, based on the preprocessed text information, perform entity recognition by using natural language processing technology, identify key entities in the text from the text by using a BiLSTM-CRF model based on deep learning, including enterprise name, product name, amount, date and location and the like, and in the identification process, analyze the context information, part-of-speech and grammatical structure of the words, judge the entity boundary, after identifying the entity, perform preliminary semantic annotation to assign a general category label to each entity, clarify the approximate meaning and category of the entity, on the basis of the preliminary annotation, further mine the attribute information of the entity, perform deep semantic annotation, determine the association between entities and the attributes of the entity itself by analyzing the semantic relationship of the text in which the entity is located and combining a domain knowledge base, construct an association network between entities by analyzing the semantic relationship between entities, make the logical relationship between data more clear, at the same time, make the meaning and attributes of each entity more clear and complete through attribute association and deep annotation;

[0057] The relationship extraction modeling unit constructs a knowledge graph model of supply chain finance based on the results of semantic annotation, defines the hierarchical structure and rules of entities, relationships and their attributes, and displays the association between data to help users better understand the complex relationships in supply chain finance business. According to the results of semantic annotation, the core entities of the supply chain finance knowledge graph are determined, including core enterprises, suppliers, financial institutions, logistics enterprises, products and transaction orders. Among them, the core enterprise is the leader of the supply chain, and its decision and operation have a significant impact on the entire supply chain. Suppliers provide raw materials or products for core enterprises and are an important link in the supply chain. Financial institutions provide financial support for enterprises in the supply chain to promote business operations. Logistics enterprises are responsible for the transportation and storage of goods to ensure smooth logistics in the supply chain. Products are the subject of supply chain transactions, and transaction orders record business transactions between enterprises. For each core entity, its basic attributes are defined. At the same time, the relationship types between core entities are sorted out, and a conceptual model of the knowledge graph is constructed. The data after semantic annotation is mapped and converted according to the conceptual model of the knowledge graph, and the extracted entities and relationships are integrated to form a knowledge graph model of supply chain finance. The constructed knowledge graph is visualized using the graphical interface of Neo4j. In the visualization interface, entities are presented in the form of nodes, and relationships are represented by edges. Different colors, shapes and labels are used to distinguish entity types and relationship types to display the association between data. At the same time, the knowledge graph is optimized to check for problems such as duplicate entities and incorrect relationships. Through data cleaning techniques, duplicate nodes and edges are removed, and rule correction methods are used to correct incorrect relationship attributes.

[0058] In addition, the construction process of the knowledge graph model of supply chain finance is as follows:

[0059] For the structured data after semantic annotation, according to the pre-constructed supply chain finance knowledge graph concept model, entity and relationship information is directly extracted, the node position of each entity in the knowledge graph is determined, and the connection mode of the relationship between the nodes is determined. The extracted entity and relationship information is accurately filled into the corresponding position of the knowledge graph, and the filled data is preliminarily checked to ensure the integrity and accuracy of the data and avoid data loss or errors. For semi-structured data and unstructured data, natural language processing technology is used to further analyze the entity and relationship information in the data. For semi-structured data, a file parsing tool is used to extract key information, and for unstructured data, OCR technology (for picture reports) and NLP technology (for contract text) are used to extract entities and relationships. The parsed entities and relationships are compared and integrated with the concept model. For the part that matches the existing entities and relationships in the concept model, corresponding filling is directly performed. For new entities and relationships, they are added according to the expansion rules of the concept model, so that semi-structured data and unstructured data are integrated into the knowledge graph. The entities and relationships extracted from structured data, semi-structured data and unstructured data are integrated, and based on the concept model of the knowledge graph, the data from different sources are uniformly mapped into the model to ensure the consistency of entities and relationships. At the same time, according to the relationship type between entities, the edges of the knowledge graph are constructed to form a complete supply chain finance knowledge graph model. Through data cleaning and checking, errors and repeated information in the data are corrected to ensure the accuracy and integrity of the knowledge graph.

[0060] The data fusion processing module is used for semantic alignment of heterogeneous data according to the knowledge graph model and semantic annotation information, and performs fusion processing and data correlation analysis to mine potential correlation between data.

[0061] The data application management module is used for applying the fused data to specific business scenarios of supply chain finance, and is responsible for storage, updating, maintenance and security management of the fused data, supports dynamic updating of the data, and ensures the timeliness and accuracy of the data.

[0062] In the embodiment 2 as shown in the Figure 1 、 Figure 2 The data fusion processing module includes a semantic alignment unit and a data fusion unit.

[0063] The semantic alignment unit matches entities in different source data based on entity and relationship annotations in the knowledge graph model, calculates an entity similarity index, compares it with a preset similarity threshold, identifies the same or similar entities, and performs semantic alignment on different source data. The heterogeneous data from different sources can be mapped into a unified knowledge graph framework, eliminating semantic ambiguity and improving the accuracy and reliability of data fusion. Based on the constructed supply chain finance knowledge graph model, the entities in different source data are matched based on the entity annotations therein, the entity identifiers are extracted from each data source, and the text similarity calculation method is used to calculate the entity similarity index, which is compared with the preset similarity threshold to identify the same or similar entities. Then, according to the semantic annotations in the knowledge graph model, the same or similar entities are mapped to the corresponding node positions in the unified knowledge graph framework. Based on the entity alignment, the relationships in different source data are aligned, the relationships between entities in each data source are analyzed according to the relationship types and rules defined in the knowledge graph model, and they are matched with the relationships in the knowledge graph. At the same time, the semantic consistency of the relationships is checked to check whether the attributes of the relationships are consistent. If there is inconsistency, the reason is further analyzed and corrected or marked to ensure the semantic consistency of the relationships and improve the reliability of data fusion. The data after entity matching and relationship alignment are fused to integrate the heterogeneous data from different sources into a unified knowledge graph framework to form a complete data structure. For repeated data of the same entity or relationship, the average value or merging operation is performed to avoid data redundancy. At the same time, according to the semantic annotations and association rules in the knowledge graph model, the semantic information of the data is further integrated to eliminate the semantic differences between the data sources and realize the deep fusion of heterogeneous data.

[0064] In addition, the calculation process of the entity similarity index is as follows:

[0065] According to the entity types defined in the knowledge graph model, the attributes of the entities to be compared are determined, and the attribute values of the two entities e1 and e2 are extracted from different data sources. For each attribute A i , the attribute values A i1 and A i2 of entities e1 and e2 are extracted, respectively.

[0066] For each attribute A i , the attribute values A i1 and A i2 are converted into vector representation. If the attribute value is text, it can be converted into a vector using Word2Vec, and the cosine similarity between the two vectors is calculated. Then, for each attribute A i , the weighted similarity is calculated according to its weight w i and cosine similarity cos(θ i ).

[0067] The weighted similarities of all attributes are summed up, and the sum is normalized to obtain the final similarity index S(e1, e2). The similarity index S(e1, e2) is compared with a set similarity threshold. If S(e1, e2) ≥ τ, it is considered that the entities e1 and e2 are the same or similar. If S(e1, e2) < τ, it is considered that the entities e1 and e2 are not similar.

[0068] The calculation expression of the entity similarity index is as follows:

[0069]

[0070] In the formula, S(e1, e2) is the entity similarity index between the entities e1 and e2, N is the number of attributes, w i is the weight of the i-th attribute, reflecting the importance of the attribute in the similarity calculation, θ i is the angle between the i-th attribute values, obtained by cosine similarity calculation, A i1 and A i2 are the vector representations of the i-th attribute values of the entities e1 and e2 respectively, |A i | is the vector length of the i-th attribute value, used to adjust the similarity contribution of the attribute value, log(1+|A i |) is used to adjust the length of the attribute value, avoiding the deviation caused by the different lengths of the attribute values. The value range of the similarity index is (0, 1], when the two entities are completely the same, the similarity index is 1, and when the two entities are completely different, the similarity index tends to 0.

[0071] The data fusion unit is used to fuse the semantically aligned data, integrate the scattered data into a unified data structure according to the association relationship in the knowledge graph, and combine the fused data with the knowledge graph model to perform data association analysis, integrate the semantically aligned data according to the association relationship in the knowledge graph model, and map the data in the scattered data sources to a unified data structure according to the entities and relationships defined in the knowledge graph. At the same time, the consistency check is performed on the integrated data to check whether there is any conflict in the information in different data sources, avoid information conflicts caused by different data sources, fuse the integrated data, and eliminate duplicate data. Duplicate data is evaluated based on data quality. The optimal data is selected for retention based on factors such as data source reliability, update frequency, and accuracy. For duplicate relational data, unique identifiers are used to identify whether it is a duplicate record and perform deduplication processing to ensure data uniqueness, avoid data redundancy, and improve data availability. Data association analysis is performed by combining the fused data with the knowledge graph model. The association relationships in the knowledge graph are used to explore potential connections between data. At the same time, the fused data is optimized, the data integrity and accuracy are checked, and any errors or missing information are corrected. Through data association analysis, deeper insights and decision support are provided for supply chain finance business.

[0072] The data application management module specifically includes:

[0073] Connect the fused data to the specific business scenarios of supply chain finance, analyze the data requirements of different business scenarios, and extract the corresponding data sets from the fused data based on the data requirements and perform preprocessing to ensure that the data can be directly applied to the business system. At the same time, develop data interfaces to achieve seamless connection between fused data and business systems, so that business personnel can easily obtain and use data, improve the efficiency and accuracy of business decision-making, establish a dynamic data update mechanism, formulate update plans based on the update frequency of data sources and business needs, and adopt a combination of incremental updates and full updates during the data update process to reduce the impact of data updates on system performance. At the same time, check and verify the updated data to ensure data integrity and consistency, classify and manage the data, set different access rights based on the sensitivity of the data and business needs, and ensure that only authorized users can access the corresponding data. At the same time, establish a data audit mechanism to record data access and operation logs to facilitate tracking and monitoring of data usage. Through data security and access management, ensure the security of data during storage, update and use.

[0074] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A supply chain finance heterogeneous data cleaning and fusion processing system, including a heterogeneous data processing platform, characterized in that: The heterogeneous data processing platform is communicatively connected to the following modules, wherein: The data acquisition module is used to obtain heterogeneous data from various participants in supply chain finance, including structured data, semi-structured data, and unstructured data, and pre-process the obtained multi-source heterogeneous data; The knowledge graph construction module is used to semantically annotate entities and relationships in heterogeneous data, build a knowledge graph model for supply chain finance, and clarify the association relationships between heterogeneous data; The data fusion processing module is used to semantically align heterogeneous data based on the knowledge graph model and semantic annotation information, and perform fusion processing and data association analysis to explore potential associations between data; The data application management module is used to apply the integrated data to specific business scenarios of supply chain finance.

2. A supply chain finance heterogeneous data cleaning and fusion processing system according to claim 1, characterized in that: The data acquisition module specifically includes: Identify the data sources of each supply chain finance participant and classify them into structured data, semi-structured data, and unstructured data. Then, based on the type of data source, select the appropriate access method to establish connections with each supply chain finance participant. Collect data from various data sources according to a predetermined frequency and method. For real-time data, pull it in real time through APIs or message queues. For static data, obtain it regularly through file transfers or database queries. Then perform preliminary processing on the collected data. At the same time, preliminarily label the collected data and record metadata information such as the source, type, and collection time of the data. Pre-process the collected multi-source heterogeneous data, convert data in different formats into a unified intermediate format, check the logical consistency of the data, correct or mark inconsistent data, and then integrate data from different participants and different types to build a complete supply chain finance data view.

3. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 1 is characterized by: The knowledge graph construction module includes a semantic annotation unit and a relationship extraction modeling unit; The semantic annotation unit is used to identify entities in heterogeneous data using natural language processing technology and perform semantic annotation on them to clarify the meaning and attributes of each entity; The relationship extraction modeling unit constructs a knowledge graph model of supply chain finance based on the results of semantic annotation, defines the hierarchical structure and rules of entities, relationships and their attributes, and displays the association relationship between data.

4. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 3 is characterized by: The semantic annotation unit specifically includes: Preprocess the text information in heterogeneous data in supply chain finance by cleaning the text to remove irrelevant characters, special symbols, and extra spaces, unify the text format, convert text from different sources into a unified encoding format, and use corresponding word segmentation tools based on the language characteristics of Chinese or English. For Chinese text, use the Jieba word segmentation tool for word segmentation, and for English text, use the NLTK word segmentation tool to split the text into word sequences. At the same time, perform part-of-speech tagging to mark each word with its part of speech. Based on the preprocessed text information, natural language processing technology is used for entity recognition. The deep learning-based BiLSTM-CRF model is used to identify key entities in the text. During the recognition process, the context, part of speech, and grammatical structure of the words are analyzed to determine the entity boundaries. After the entities are identified, preliminary semantic annotation is performed, and a common category label is assigned to each entity. On the basis of preliminary annotation, we further explore the attribute information of entities and perform deep semantic annotation. By analyzing the semantic relationship of the text in which the entities are located and combining with the domain knowledge base, we can determine the association between entities and the attributes of the entities themselves.

5. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 3 is characterized by: The relationship extraction modeling unit specifically includes: Based on the semantic annotation results, the core entities of the supply chain finance knowledge graph are identified, including core enterprises, suppliers, financial institutions, logistics companies, products, and transaction orders. For each core entity, its basic attributes are defined. At the same time, the relationship types between core entities are sorted out, and then a conceptual model of the knowledge graph is constructed. Map and transform the semantically annotated data according to the conceptual model of the knowledge graph, and integrate the extracted entities and relationships to form a knowledge graph model for supply chain finance; The constructed knowledge graph is visualized using Neo4j's graphical interface. In the visualization interface, entities are presented as nodes and relationships are represented as edges. Different colors, shapes, and labels are used to distinguish entity types and relationship types, and to display the association between data. At the same time, the knowledge graph is optimized to check whether there are duplicate entities and incorrect relationships. Duplicate nodes and edges are removed through data cleaning technology, and incorrect relationship attributes are corrected using rule correction methods.

6. A supply chain finance heterogeneous data cleaning and fusion processing system according to claim 5, characterized in that: The construction process of the knowledge graph model of supply chain finance is as follows: For the semantically annotated structured data, based on the pre-built supply chain finance knowledge graph conceptual model, we directly extract entity and relationship information, clarify the node position of each entity in the knowledge graph, and the connection method between the nodes of the relationship, and fill the extracted entity and relationship information into the corresponding position of the knowledge graph; For semi-structured and unstructured data, natural language processing technology is used to further analyze the entity and relationship information therein. For semi-structured data, file parsing tools are used to extract key information. For unstructured data, OCR technology and NLP technology are combined to extract entities and relationships. The parsed entities and relationships are compared and integrated with the conceptual model. Parts that match existing entities and relationships in the conceptual model are directly filled in accordingly. For new entities and relationships, they are added according to the expansion rules of the conceptual model, so that semi-structured and unstructured data can be integrated into the knowledge graph. Integrate the entities and relationships extracted from structured data, semi-structured data and unstructured data, and uniformly map data from different sources into the model based on the conceptual model of the knowledge graph. At the same time, construct the edges of the knowledge graph according to the relationship types between entities to form a complete supply chain finance knowledge graph model.

7. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 3 is characterized by: The data fusion processing module includes a semantic alignment unit and a data fusion unit; The semantic alignment unit matches entities in data from different sources based on entity and relationship annotations in the knowledge graph model, calculates entity similarity indexes, compares them with a preset similarity threshold, identifies identical or similar entities, and semantically aligns data from different sources. The data fusion unit is used to fuse the semantically aligned data, integrate the scattered data into a unified data structure according to the association relationship in the knowledge graph, and perform data association analysis by combining the fused data and the knowledge graph model.

8. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 7 is characterized by: The semantic alignment unit specifically includes: Based on the constructed supply chain finance knowledge graph model and the entity annotations therein, entities in different source data are matched, entity identifiers are extracted from each data source, and the entity similarity index is calculated using the text similarity calculation method. This is compared with the preset similarity threshold to identify the same or similar entities. Then, based on the semantic annotations in the knowledge graph model, the same or similar entities are mapped to the corresponding node positions in the unified knowledge graph framework; After completing entity alignment, align the relationships in data from different sources. According to the relationship types and rules defined in the knowledge graph model, analyze the relationships between entities in each data source and match them with the relationships in the knowledge graph. At the same time, verify the semantic consistency of the relationships and check whether the attributes of the relationships are consistent. If there are inconsistencies, further analyze the reasons and make corrections or mark them. The data after entity matching and relationship alignment are fused, and heterogeneous data from different sources are integrated into a unified knowledge graph framework to form a complete data structure. For repeated data of the same entity or relationship, the average or merger operation is performed. At the same time, according to the semantic annotations and association rules in the knowledge graph model, the semantic information of the data is further integrated to eliminate the semantic differences between data sources.

9. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 8 is characterized by: The data fusion unit specifically includes: Integrate semantically aligned data according to the association relationships in the knowledge graph model, and map data from scattered data sources into a unified data structure based on the entities and relationships defined in the knowledge graph; The integrated data is fused and duplicate data is eliminated. For duplicate data of the same entity or relationship, the quality of the data is evaluated and the best data is selected for retention. For duplicate relationship data, unique identifiers are used to identify whether it is a duplicate record and then perform deduplication. Combine the fused data with the knowledge graph model to conduct data association analysis, use the association relationships in the knowledge graph to explore the potential connections between the data, and at the same time, optimize the fused data, check the completeness and accuracy of the data, and correct any errors or missing information.

10. The supply chain finance heterogeneous data cleaning and fusion processing system according to claim 1 is characterized by: The data application management module specifically includes: Connect the fused data to specific business scenarios of supply chain finance, analyze the data requirements of different business scenarios, and extract the corresponding data sets from the fused data for preprocessing based on the data requirements; Establish a dynamic data update mechanism, formulate an update plan based on the update frequency of the data source and business needs, and verify and validate the updated data; Data is classified and graded, and different access permissions are set according to the sensitivity of the data and business needs to ensure that only authorized users can access the corresponding data. At the same time, a data audit mechanism is established to record data access and operation logs.

Images