Attribution analysis method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an attribution analysis method and device, equipment and a storage medium, and the method comprises the steps: classifying data, and obtaining structured data and unstructured data; generating a first sub-data consanguinity according to a mapping relation between fields in the structured data; according to an association relationship between entity nodes identified from the unstructured data, a second sub-data blood relationship is generated, and the entity nodes correspond to time sequence nodes in the unstructured data; generating a data consanguinity based on the first sub-data consanguinity and the second sub-data consanguinity; according to the type of the application layer link event, attribution analysis is carried out on the application layer link event based on the data consanguinity, an attribution analysis result is obtained, attribution analysis can be carried out on unstructured data while attribution analysis is carried out on structured data, and attribution analysis of various types of data is achieved.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to an attribution analysis method, device, equipment and storage medium. Background Art

[0002] Attribution Analysis is a method to evaluate the impact of different factors on results. Its purpose is to identify and evaluate the various factors that affect the results in order to better understand the role of these factors and make decisions accordingly.

[0003] Currently, attribution analysis can be performed based on the association between data, but this method is more suitable for structured data. When the data set contains a large amount of unstructured data, it is difficult to perform attribution analysis. Therefore, there is an urgent need for a solution that can perform attribution analysis on various types of data. Summary of the invention

[0004] Based on the above problems, the present application provides an attribution analysis method, device, equipment and storage medium.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In the first aspect, the present application provides an attribution analysis method, including:

[0007] Classify data to obtain structured data and unstructured data;

[0008] Generate a first sub-data lineage according to a mapping relationship between fields in the structured data; generate a second sub-data lineage according to an association relationship between entity nodes identified from the unstructured data, wherein the entity nodes correspond to time series nodes in the unstructured data;

[0009] Generate data lineage based on the first sub-data lineage and the second sub-data lineage;

[0010] According to the type of the application layer link event, an attribution analysis is performed on the application layer link event based on the data lineage to obtain an attribution analysis result.

[0011] In a possible implementation, generating the second sub-data lineage according to the association relationship between the entity nodes identified from the unstructured data includes:

[0012] Identify time series nodes from unstructured data;

[0013] Based on the time series node, determining an entity node corresponding to the time series node from the unstructured data;

[0014] A second sub-data lineage is generated according to the association relationship between the plurality of entity nodes.

[0015] In a possible implementation, determining, based on the time series node, from the unstructured data, an entity node corresponding to the time series node includes:

[0016] Parsing unstructured text information corresponding to the time series node in the unstructured data;

[0017] Determine the type and name of the entity node and the starting position and ending position of the entity node in the unstructured data from the unstructured text information;

[0018] Based on the type and name of the entity node and the start position and end position of the entity node in the unstructured data, an entity node corresponding to the time series node is determined.

[0019] In a possible implementation, generating data lineage based on the first sub-data lineage and the second sub-data lineage includes:

[0020] Determine an original field from the first sub-data lineage and / or the second sub-data lineage;

[0021] The original field is derived according to the data lineage generation rule of the derived field to generate data lineage.

[0022] In a possible implementation, the deriving process of the original field according to the data lineage generation rule of the derived field to generate data lineage includes:

[0023] When the original field is a time interval, the difference between the start time and the end time is calculated;

[0024] When the difference is greater than or equal to 1 year, a time interval is generated in the data lineage and displayed in years; when the difference is less than 1 year and greater than or equal to 1 month, a time interval is generated in the data lineage and displayed in months; when the difference is less than 1 month and greater than or equal to 1 day, a time interval is generated in the data lineage and displayed in days; when the difference is less than 1 day, a time interval is generated in the data lineage and displayed in hours.

[0025] In a possible implementation, the deriving process of the original field according to the data lineage generation rule of the derived field to generate data lineage includes:

[0026] When the original field is the name of the entity node, data cleaning is performed on the original field based on the knowledge base to obtain a derived field;

[0027] The original fields in the first sub-data lineage and / or the second sub-data lineage are displayed as the derived fields to generate data lineage.

[0028] In a possible implementation, performing attribution analysis on the application layer link event based on the data lineage according to the type of the application layer link event to obtain the attribution analysis result includes:

[0029] Determining the priority of the application layer link event according to the occurrence frequency identifier and the importance level identifier of each field in the data lineage;

[0030] According to the priority of the application layer link event, according to the type of the application layer link event, and based on the data lineage, attribution analysis is performed on the application layer link event to obtain an attribution analysis result.

[0031] A second aspect: The present application embodiment provides an attribution analysis device, including:

[0032] A classification unit, a first generation unit, a second generation unit, and an attribution analysis unit;

[0033] The classification unit is used to classify the data to obtain structured data and unstructured data;

[0034] The first generating unit is used to generate a first sub-data lineage according to a mapping relationship between fields in the structured data; and to generate a second sub-data lineage according to an association relationship between entity nodes identified from the unstructured data, wherein the entity nodes correspond to time series nodes in the unstructured data;

[0035] The second generating unit is used to generate data lineage based on the first sub-data lineage and the second sub-data lineage;

[0036] The attribution analysis unit is used to perform attribution analysis on the application layer link event based on the data lineage according to the type of the application layer link event to obtain an attribution analysis result.

[0037] A third aspect: An embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory;

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is used to execute the steps of any attribution analysis method in the first aspect according to the instructions in the program code.

[0040] Fourth aspect: An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any attribution analysis method in the above-mentioned first aspect are implemented.

[0041] Compared with the prior art, this application has the following beneficial effects:

[0042] The embodiments of the present application provide an attribution analysis method, apparatus, device and storage medium, which obtain structured data and unstructured data by classifying data; generate a first sub-data lineage according to the mapping relationship between fields in the structured data; generate a second sub-data lineage according to the association relationship between entity nodes identified from the unstructured data, and the entity nodes correspond to the timing nodes in the unstructured data; according to the type of application layer link event, perform attribution analysis on the application layer link event based on the data lineage generated by the first sub-data lineage and the second sub-data lineage, and obtain an attribution analysis result. The method provided in the embodiments of the present application takes into account both structured data and unstructured data in the process of generating data lineage, and can perform attribution analysis on unstructured data while being able to perform attribution analysis on structured data, thereby realizing attribution analysis on multiple types of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A flowchart of an attribution method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of formatting processing provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a corresponding relationship provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a first sub-data lineage provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a second sub-data lineage provided in an embodiment of the present application;

[0049] Figure 6 A visualized second sub-data lineage diagram provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of generating a derived field provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of an attribution analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] To facilitate understanding, the technical concepts involved in the embodiments of the present application are introduced below.

[0054] Data lineage refers to the natural relationship between data, similar to human bloodline, that is formed throughout the data's life cycle, from its generation, processing, fusion, and circulation to its final extinction.

[0055] Attribution Analysis is a method to evaluate the impact of different factors on results. Its purpose is to identify and evaluate the various factors that affect the results in order to better understand the role of these factors and make decisions accordingly.

[0056] As described above, attribution analysis can be performed based on the correlation between data, but this method is more suitable for structured data. When the data set contains a large amount of unstructured data, it is difficult to perform attribution analysis.

[0057] Taking the medical industry as an example, data in the medical industry can come from many different business systems, such as Hospital Information System (HIS), Laboratory Information System (LIS), and Picture Archiving and Communication System (PACS).

[0058] There are differences in the data types of different business systems, including structured data and unstructured data. At present, attribution analysis can only be performed on structured data, and there is a lack of attribution analysis methods for unstructured data. As a result, the process of attribution analysis of medical industry data is limited by the data type and has low applicability.

[0059] Therefore, there is an urgent need for a solution that can perform attribution analysis on various types of data.

[0060] Based on this, an embodiment of the present application provides an attribution analysis method, which generates a first sub-data lineage based on the mapping relationship between fields in structured data; and generates a second sub-data lineage based on the association relationship between entity nodes identified in unstructured data. Data lineage is generated based on the first sub-data lineage and the second sub-data lineage, and on this basis, according to the type of application layer link event, attribution analysis is performed on the application layer link event based on the data lineage. Since the embodiment of the present application takes both structured data and unstructured data into consideration in the process of generating data lineage, attribution analysis of multiple types of data can be achieved based on the data lineage.

[0061] The following is an introduction to an attribution method provided by an embodiment of the present application in conjunction with the accompanying drawings. Figure 1 As shown, the figure is a flowchart of an attribution method provided in an embodiment of the present application, including S101-S104.

[0062] S101. Classify data to obtain structured data and unstructured data.

[0063] Among them, structured data refers to data with clear, fixed formats and definitions. Structured data is usually stored in relational databases such as MySQL, PostgreSQL, etc.

[0064] Unstructured data refers to data without a fixed format or model, such as text information, images, etc. Unstructured data is usually stored in non-relational databases such as MongoDB, Cassandra, etc., or stored in the form of files in file systems or cloud storage.

[0065] The data that needs to be analyzed for attribution may include both structured data and unstructured data. In the embodiment of the present application, the data can be classified based on the characteristics of the structured data and the unstructured data to obtain the structured data and the unstructured data in the data.

[0066] S102: Generate a first sub-data lineage according to a mapping relationship between fields in the structured data; and generate a second sub-data lineage according to an association relationship between entity nodes identified from the unstructured data.

[0067] In the embodiment of the present application, data lineage can be represented by data nodes. Data nodes are business carriers with independent functions. The forms of carrying business include but are not limited to databases, data tables, data fields, etc.

[0068] Data nodes are mainly divided into three types: outflow nodes, inflow nodes and intermediate nodes. Among them, outflow nodes are used to provide the most basic data source; inflow nodes are terminal nodes in the data lineage relationship, which receive data from intermediate nodes or other data sources and are the end point of the entire data flow path; intermediate nodes are the largest in number and are used for data mapping, data cleaning, data processing and data warning, etc.

[0069] In a possible implementation, an embodiment of the present application can map fields in structured data through SQL to obtain a mapping relationship between the fields, and then generate a first sub-data lineage, that is, a data lineage of the structured data, and the steps include steps 21 to 23.

[0070] Step 21: Format the SQL statement to obtain a formatted SQL statement.

[0071] In the embodiment of the present application, by formatting the SQL statements, SQL statements with non-uniform formats can be processed into formatted SQL statements that comply with certain specifications.

[0072] like Figure 2 As shown, this figure is a schematic diagram of formatting processing provided by an embodiment of the present application. The format of the SQL statement before formatting is relatively messy. In order to improve the readability and maintainability of the SQL statement, the SQL statement can be formatted based on unified indentation rules, line break rules, and letter case rules, etc., thereby obtaining a formatted SQL statement that meets certain specifications.

[0073] Step 22: Parse the formatted SQL statement to obtain the first sub-data lineage analysis result.

[0074] like Figure 3 As shown, this figure is a schematic diagram of a corresponding relationship provided by an embodiment of the present application. In the embodiment of the present application, by parsing statements related to the source table, the table name and field name of the source table can be obtained; by parsing statements related to the target table, the table name and field name of the target table can be obtained; by parsing the mapping relationship between the fields, the mapping relationship between the source table and the target table fields, as well as the function information of the source field when mapping, etc. can be obtained.

[0075] In the embodiment of the present application, by parsing the formatted SQL statement, the mapping relationship between the fields in the entire link can be obtained. Exemplarily, the mapping relationship between the fields may include a one-to-one mapping and a one-to-many mapping. For example, in the case where a mapping relationship exists between field A and field B, the mapping relationship between field A and field B belongs to a one-to-one mapping; in the case where field C, field D, field E all have a mapping relationship with field F, the mapping relationship between field C, field D, field E and field F belongs to a many-to-one or one-to-many mapping.

[0076] After determining the mapping relationship between fields and mapping these fields, a data lineage relationship is established between the table to which the field belongs and the database where the table is located. In this data lineage relationship, each data node and the data governance tasks related to it can be connected in series into a data governance link as data lineage is formed.

[0077] Step 23: Visualize the results of the first sub-data lineage analysis to obtain the first sub-data lineage.

[0078] like Figure 4 As shown, the figure is a schematic diagram of a first sub-data lineage provided by an embodiment of the present application. As can be seen from the figure, by performing lake entry processing on the source table fields, the lake entry table fields can be obtained, and by performing data governance on the lake entry table fields, the standard table fields can be obtained.

[0079] In an embodiment of the present application, by visualizing the results of the blood relationship analysis of the first sub-data, the flow and conversion relationship between the data can be displayed in a graphical manner, making complex data relationships clear and easy to understand, helping users to quickly understand the source, processing process and destination of the data, thereby improving the efficiency and accuracy of data management.

[0080] In a possible implementation, the embodiment of the present application can generate a second sub-data lineage, that is, data lineage of the unstructured data, based on the association relationship between entity nodes identified from the unstructured data.

[0081] Among them, unstructured data refers to text data with irregular or incomplete data structure, no predefined data model, and data that cannot be represented by a two-dimensional logical table in a database, such as the chief complaint, current medical history, and past history in case documents.

[0082] The process of generating the second sub-data lineage according to the association relationship between the entity nodes identified from the unstructured data may include first identifying a time series node from the unstructured data; based on the time series node, determining the entity node corresponding to the time series node from the unstructured data.

[0083] In the embodiment of the present application, the time series nodes in the unstructured data correspond to entity nodes.

[0084] In a possible implementation, the process of determining the entity node corresponding to the time series node from the unstructured data based on the time series node may include: parsing the unstructured text information corresponding to the time series node in the unstructured data; determining the type, name, and starting and ending positions of the entity node in the unstructured data from the unstructured text information; and determining the entity node corresponding to the time series node based on the type, name, and starting and ending positions of the entity node in the unstructured data.

[0085] like Figure 5 As shown, in the embodiment of the present application, when processing unstructured data, structured semantic parsing of the unstructured data can be performed through natural language processing technology. When structured semantic parsing of the unstructured data is performed, unstructured text information corresponding to the time series node in the unstructured data can be obtained.

[0086] After obtaining the unstructured text information corresponding to the timing node, various types of nodes in the unstructured text information can be marked by index pre-identification, such as marking the timing node and the type, name, and start and end positions of the entity node corresponding to the timing node in the unstructured data.

[0087] After the entity node corresponding to the time series node is determined, the second sub-data lineage may be generated according to the association relationship between the plurality of entity nodes.

[0088] In the embodiment of the present application, by identifying the association relationship between each entity node, the second sub-data lineage can be constructed to generate the second sub-data lineage.

[0089] For ease of understanding, the second sub-data bloodline generation method is introduced below using medical structure data as an example, including steps 31 to 34.

[0090] Step 31: Identify entities representing time series in unstructured data and construct time series nodes.

[0091] For example, for a piece of unstructured data: 7 years ago, the patient felt dizzy and blackout during activities, and lost consciousness, then recovered on his own, the duration is unknown, denied chest pain, palpitations, sweating, headaches, nausea, limb movement disorders, speech difficulties, incontinence, and visited our hospital. He reported a heart rate of about 40bpm, and the ECG showed "third-degree atrioventricular block" (no report). A permanent pacemaker (DDDR) was implanted. After the operation, there was no dizziness or syncope, and the pacemaker was programmed regularly and worked normally. 4 years ago, there was no precordial discomfort without any inducement, denied chest pain, palpitations, and dizziness, and visited an external hospital.

[0092] Among them, ECG is electrocardiogram.

[0093] By recognizing the above unstructured text, it is possible to identify entities representing time series in the unstructured data, including "7 years ago" and "4 years ago", and use them as time series nodes.

[0094] Step 32: Construct entity nodes based on the unstructured text information corresponding to the time series nodes.

[0095] In an embodiment of the present application, after determining the time series node, the unstructured text information corresponding to the time series node can be parsed from the unstructured data.

[0096] Exemplarily, the unstructured text information corresponding to "7 years ago" includes: 7 years ago, the patient felt dizzy and blackout during activities and lost consciousness, but recovered on his own and the duration was unknown. He denied chest pain, palpitations, sweating, headache, nausea, limb movement disorders, speech difficulties, and incontinence. He visited our hospital and reported a heart rate of about 40bpm. The electrocardiogram showed "third degree atrioventricular block" (not reported). A permanent pacemaker (DDDR) was implanted. After the operation, he did not experience dizziness or syncope again. The pacemaker is regularly programmed and works normally.

[0097] The unstructured text information corresponding to “4 years ago” includes: 4 years ago, precordial discomfort occurred without any inducement, the patient denied chest pain, palpitations, and dizziness, and visited another hospital for treatment.

[0098] In an embodiment of the present application, after determining the unstructured text information corresponding to the timing node, an entity node can be constructed under each timing node. The information of the entity node includes a start position identifier, an end position identifier, an entity type, and an entity name, etc.

[0099] Taking the above-mentioned time series node "4 years ago" as an example, after semantic analysis of the unstructured text information corresponding to the time series node "4 years ago", the identified entity nodes include "4 years ago", "no cause", "chest pain", "palpitations", and "dizziness".

[0100] Therefore, the recognition result of the entity node corresponding to the time series node "4 years ago" can be expressed as follows:

[0101] The recognition result of the entity node "4 years ago" is: starting position identifier: 152; ending position identifier: 154; entity type: symptom occurrence time; entity name: 4 years ago.

[0102] The recognition result of the entity node "no inducement" is: starting position identifier: 155; ending position identifier: 157; entity type: inducement; entity name: no inducement.

[0103] The recognition result of the entity main point "precordial area" is: starting position identifier: 160; ending position identifier: 162; entity type: part; entity name: precordial area.

[0104] The recognition result of the entity node "chest pain" is: start position identifier: 168; end position identifier: 169; entity type: no symptoms; entity name: chest pain.

[0105] The recognition result of the entity node "palpitations" is: start position identifier: 171; end position identifier: 172; entity type: no symptoms; entity name: palpitations.

[0106] The recognition result of the entity node "dizziness" is: start position identifier: 174; end position identifier: 175; entity type: no symptoms; entity name: dizziness.

[0107] The starting position identifier is used to indicate the starting position of the entity node in the corresponding unstructured text information; and the ending position identifier is used to indicate the ending position of the entity node in the corresponding unstructured text information.

[0108] Exemplarily, the start position is marked as P, then the first character of the entity node is the Pth character in the unstructured text information; the end position is marked as Q, and the last character of the entity node is the Qth character in the unstructured text information, where P and Q are both integers greater than 0.

[0109] Taking the entity node "no incentive" as an example, its starting position is identified as 155 and its ending position is identified as 157, indicating that the "no" in the entity node "no incentive" is the 155th character in the corresponding unstructured text information, and the "cause" in the entity node "no incentive" is the 157th character in the corresponding unstructured text information.

[0110] Step 33: Construct association relationships between entity nodes.

[0111] In the embodiment of the present application, after obtaining the recognition result of the entity node, the relationship of each entity node can be identified, the association relationship between the entity nodes is established, and the entity relationship is marked.

[0112] For example, taking the above time series node "4 years ago" as an example, the association relationship between the entity nodes obtained by performing relationship identification on the entity nodes corresponding to the time series node is as follows:

[0113] Entity node "chest pain", entity node "no cause"; association relationship: symptoms and symptom causes.

[0114] Entity node “Palpitations”, entity node “No Cause”; association relationship: symptoms and symptom causes.

[0115] Entity node “dizziness”, entity node “no cause”; association relationship: symptoms and symptom causes.

[0116] Entity node “chest pain”, entity node “precordial area”; associated relationship: symptoms and symptom location.

[0117] Entity node “palpitations”, entity node “precordial area”; associated relationship: symptoms and symptom locations.

[0118] Entity node “dizziness”, entity node “precordial area”; associated relationship: symptoms and symptom locations.

[0119] Step 34: Visualize the results of the second sub-data lineage analysis to obtain the second sub-data lineage.

[0120] In order to facilitate users to quickly understand the source, processing process and destination of data, make complex data relationships clear and easy to understand, and improve the efficiency and accuracy of data management, in the embodiment of the present application, the second sub-data lineage analysis results are visualized to obtain the second sub-data lineage. Figure 6 As shown, this figure is a visualized second sub-data bloodline diagram provided in an embodiment of the present application.

[0121] The unstructured data shown in the figure are: 10 years before the patient was admitted to the hospital, he had dull pain in the lower part of the sternum without obvious cause, a sense of impending death, the pain radiated to the sides, and he was forced to stop moving. The pain was continuous and lasted for 10 minutes. It was relieved after resting for 30 minutes, accompanied by sweating and dizziness, but no palpitations, dyspnea, blackouts, syncope, upper abdominal discomfort, nausea, or vomiting. He had never sought medical treatment.

[0122] By processing the unstructured data based on the above steps 31 to 34, a visualized second sub-data lineage can be obtained, which reflects the time series nodes, entity nodes, the association relationship between entity nodes and various identification information in the unstructured data.

[0123] S103: Generate data lineage based on the first sub-data lineage and the second sub-data lineage.

[0124] In the embodiment of the present application, the data lineage of structured data and the data lineage of unstructured data are considered simultaneously in the process of generating data lineage, thereby facilitating subsequent attribution analysis of various types of data.

[0125] In one possible implementation, in order to facilitate users to better understand and utilize data, in the process of generating data lineage based on the first sub-data lineage and the second sub-data lineage, the original fields in at least one of the first sub-data lineage and the second sub-data lineage can be derived to generate corresponding derived fields and obtain the data lineage.

[0126] Specifically, the original field is determined from the first sub-data lineage and / or the second sub-data lineage; and the original field is derived according to a data lineage generation rule of a derived field to generate the data lineage.

[0127] Among them, derived fields are new fields generated based on the original data fields through some generation rules or calculations. For example, based on one or more of the rules of operation, combination, conversion, cleaning and processing, one or more original data fields are processed to obtain derived fields. Compared with the original data fields, derived fields can provide more information, enhance the data analysis capabilities and model performance, and facilitate users to better understand and use data.

[0128] Exemplarily, the generation rules of the derived fields may include, but are not limited to, generation rules of time intervals, generation rules of payment methods, and derivation rules of names.

[0129] like Figure 7 As shown, this figure is a schematic diagram of derived field generation provided in an embodiment of the present application. For the first field and the second field in the data lake, the first field and the second field are operated, combined, cleaned, processed, etc. based on the generation rule of the derived field to obtain the corresponding derived field information. The derived field information includes original field nodes, rule nodes, and derived field nodes. Among them, the original field node indicates that the derived field is derived from it; the rule node indicates that the derived field is generated based on the corresponding derived field generation rule; the derived field node represents the derived field obtained.

[0130] In a possible implementation manner, when the original field is a time interval, the time interval is processed based on a function corresponding to the time interval to obtain a derived field corresponding to the time interval.

[0131] Exemplarily, the original field may include but is not limited to date, admission time, etc.

[0132] The logic of the function corresponding to the time interval is to calculate the difference between the start time and the end time; when the difference is greater than or equal to 1 year, the time interval is displayed in years in the generated data lineage; when the difference is less than 1 year and greater than or equal to 1 month, the time interval is displayed in months in the generated data lineage; when the difference is less than 1 month and greater than or equal to 1 day, the time interval is displayed in days in the generated data lineage; when the difference is less than 1 day, the time interval is displayed in hours in the generated data lineage.

[0133] In a possible implementation, when the original field is a payment method, the original field may be derived based on a generation rule of the payment method to obtain a derived field corresponding to the original field.

[0134] For example, when the payment method is "self-payment", the derived field is self-payment; when the payment method is "public expense", the derived field is public expense; when the payment method is "New Rural Cooperative Medical Scheme | New Rural Cooperative Medical Scheme in this city", the derived field is New Rural Cooperative Medical Scheme; when the payment method is "Basic Medical Insurance for Urban Employees | Urban Employees", the derived field is Urban Employees; when the payment method is "Basic Medical Insurance for Urban and Rural Residents in this city", the derived field is Urban Residents.

[0135] In one possible implementation, when the original field is the name of an entity node, the original field can be cleaned based on the knowledge base to obtain a derived field corresponding to the original field; the original field in the first sub-data lineage and / or the second sub-data lineage can be displayed as a derived field to generate the data lineage.

[0136] In the embodiment of the present application, the generation rule based on the derived field can be used to identify the original field in the first sub-data lineage and / or the second sub-data lineage in the process of generating the derived field, and derive the original field to obtain the derived field corresponding to the original field. By deriving the time interval in the first sub-data lineage and / or the second sub-data lineage, the time interval can be normalized, and by deriving the entity node, the entity node can be normalized, so that users can better understand and use the data.

[0137] For example, after performing derivative processing on the unstructured text information corresponding to the above “7 years ago”, the following results can be obtained:

[0138] When the time series node is 7 years ago and the original field node is 2017-05-01, the generated derived field is the time series is 7 years ago and the consultation time is 2024-05-01.

[0139] When the entity node is "Headache", a derived field "Headache" may be generated.

[0140] When the entity node is "ECG", a derived field "electrocardiogram" may be generated.

[0141] In the embodiment of the present application, S104 may be executed after data lineage is generated based on the above method.

[0142] S104: According to the type of the application layer link event, perform attribution analysis on the application layer link event based on the data lineage to obtain an attribution analysis result.

[0143] In one possible implementation, the priority of the application layer link event can be determined based on the frequency of occurrence and importance level of each field in the data lineage; according to the priority of the application layer link event and according to the type of the application layer link event, the application layer link event is attribution analyzed based on the data lineage to obtain an attribution analysis result.

[0144] The application layer is a key layer in the computer network architecture. Its main responsibility is to interact directly with users and provide users with various network services or applications.

[0145] For some modules of the application layer, different fields have different importance. For example, some fields that can directly affect the use of the application can be marked as required fields. For fields marked as non-required fields, their importance can be determined according to their frequency of occurrence in the application layer, that is, the higher the frequency of occurrence, the higher the importance.

[0146] Based on this, the occurrence frequency and importance level of the fields can be marked at the application layer based on data lineage to obtain the occurrence frequency identifier and importance level identifier of each field.

[0147] Exemplarily, when the importance level of a field is marked as a required field, it means that the field is a required field, has the highest importance, and corresponds to the highest priority; when the importance level of a field is marked as a non-required field, and the occurrence frequency is marked as 0, it means that the field is a non-concern field among non-required fields, and changes in its lineage will not affect its use at the application layer, and it has the lowest importance and corresponds to the lowest priority; when the importance level of a field is marked as a non-required field, and the occurrence frequency is marked as a value greater than 0, it means that the field is a concern field among non-required fields, and its importance and priority can be determined according to the ranking of the occurrence frequency of the field in the occurrence frequency of all concerned fields.

[0148] In one possible implementation, changes in the data type of nodes on the data lineage link may affect the post-query data processing logic of the business system. For example, converting numeric type data into string type data, or converting string type data into date type data, etc., may affect the post-query data processing logic of the business system and cause data conversion anomalies.

[0149] Therefore, when the type of application layer link event is a change in the data type of a node on a data lineage link, an attribution analysis is performed on the application layer link event based on data lineage, and the attribution analysis result is a conversion of the data type at the inflow node of the data lineage to keep the existing data type in the production environment unchanged and to keep the application layer unaffected.

[0150] In one possible implementation, if the field name of a node on a data lineage link is changed, it may result in the inability to obtain data when the field is used at the application layer. For example, if the admission time is changed to the consultation time, the corresponding data cannot be obtained by querying "consultation time".

[0151] Therefore, when the type of application layer link event is the change of the field name of a node on the data lineage link, an attribution analysis is performed on the application layer link event based on the data lineage, and the attribution analysis result can be obtained that when the field of the application layer is used, an adaptation conversion is performed based on the historical record of field name changes of the data lineage, so that the application layer is not affected.

[0152] In a possible implementation, when the generation rule of the derived field on the data lineage link changes, the value range of the derived field changes. If the value range of the derived field changes, the original processing logic of the application layer may no longer be applicable to process the changed value range. In this case, the original processing logic of the application layer needs to be adjusted accordingly to process the changed value range based on the adjusted processing logic.

[0153] Based on this, when the type of the application layer link event is a change in the generation rule of the derived field on the data lineage link, the application layer link event is attributed and analyzed based on the data lineage, and the attribution analysis result can be obtained as the generation rule of the derived field and the related functions are synchronized to the application layer in real time. Therefore, when the generation rule of the derived field changes, the application layer can perceive it in time, and based on the change of the generation rule of the derived field, the original processing logic of the application layer is adjusted accordingly, so that the application layer can process the value domain of the changed derived field accordingly based on the adjusted processing logic.

[0154] In a possible implementation, when the original field list node is changed, the meaning of the derived field may change. If the meaning of the derived field changes, the original traceability logic of the application layer may no longer be applicable to trace the derived field after the meaning has changed. In this case, the original traceability logic of the application layer needs to be adjusted accordingly to trace the derived field after the meaning has changed based on the adjusted traceability logic.

[0155] Based on this, when the type of the application layer link event is a change in the original field list node, the application layer link event is attributed and analyzed based on data lineage, and the attribution analysis result can be obtained by synchronizing the original field list of the derived field lineage to the application layer in real time. Therefore, when the meaning of the derived field changes, the application layer can perceive it in time, and based on the change in the meaning of the derived field, the original traceability logic of the application layer is adjusted accordingly, so that the application layer can trace the derived field after the change in meaning based on the adjusted traceability logic.

[0156] In a possible implementation, the type of the application layer link event is the loss of a lineage link of a field. Based on data lineage, the application layer link event is attributed and analyzed, and the attribution analysis result can be obtained as an additional verification of the data lineage, and when it is determined that the lost field is a field of high importance, additional lineage processing is performed.

[0157] In the embodiment of the present application, when the lineage link of a field is lost, by additionally checking the lineage of the data, it can be determined whether the lost field is a field of high importance. For example, a field of high importance may include but is not limited to a required field of the highest importance, a field of interest whose frequency of occurrence is higher than a frequency threshold, etc. If it is determined that the lost field is a field of high importance, additional lineage processing is performed, that is, the source and flow of the lost data are added to complete the lost data; if it is determined that the lost field is not a field of high importance, no processing is required.

[0158] It should be noted that in the embodiments of the present application, there is no specific limitation on the numerical value of the frequency threshold, which can be set according to requirements.

[0159] In a possible implementation, when the field of the standard table field node is a required field, its integrity will affect the function of the application layer. On this basis, when the type of the application layer link event is that the field integrity of the standard table field node in the data lineage of structured data is low, the application layer link event is attributed based on the data lineage. The attribution analysis result can be obtained as follows: when establishing the data lineage of structured data, the field integrity of the standard table field node is sampled and detected.

[0160] When the completeness is lower than the completeness threshold, trace back from the standard table field node to determine the standard table field node corresponding to the first field whose completeness is lower than the completeness threshold, and repair each field whose completeness is lower than the completeness threshold.

[0161] In a possible implementation, after the semantic analysis process is optimized, the accuracy of the semantic analysis is improved, which may cause the data lineage of the unstructured data to change, but this change does not affect the use of the application layer. Therefore, when the type of the application layer link event is that the data lineage of the unstructured data has changed, the application layer link event is attributed based on the data lineage. The attribution analysis result can be obtained as no processing is required.

[0162] In summary, the embodiment of the present application provides an attribution analysis method, which generates a first sub-data lineage according to the mapping relationship between fields in structured data; and generates a second sub-data lineage from the association relationship between entity nodes identified in unstructured data. Based on the first sub-data lineage and the second sub-data lineage, data lineage can be generated. Since both structured data and unstructured data are considered in the process of generating the data lineage, according to the type of application layer link event, the application layer link event is attributed and analyzed based on the data lineage, which can be applicable to various types of data and realize attribution analysis of various types of data.

[0163] At the same time, in the embodiment of the present application, the original fields in the first sub-data lineage and the second sub-data lineage are derived to facilitate users to better understand and use the data.

[0164] This application provides an attribution analysis device, see Figure 8 , this figure is a structural schematic diagram of an attribution analysis device provided in an embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above method, and some contents will not be repeated here.

[0165] The embodiment of the present application provides an attribution analysis device 1100, including:

[0166] A classification unit 1101, a first generation unit 1102, a second generation unit 1103 and an attribution analysis unit 1104;

[0167] The classification unit 1101 is used to classify data to obtain structured data and unstructured data;

[0168] The first generating unit 1102 is used to generate a first sub-data lineage according to a mapping relationship between fields in the structured data; and to generate a second sub-data lineage according to an association relationship between entity nodes identified from the unstructured data, wherein the entity nodes correspond to time series nodes in the unstructured data;

[0169] The second generating unit 1103 is used to generate data lineage based on the first sub-data lineage and the second sub-data lineage;

[0170] The attribution analysis unit 1104 is used to perform attribution analysis on the application layer link event based on the data lineage according to the type of the application layer link event to obtain an attribution analysis result.

[0171] In a possible implementation manner, the first generating unit is specifically configured to:

[0172] Identify time series nodes from unstructured data;

[0173] Based on the time series node, determining an entity node corresponding to the time series node from the unstructured data;

[0174] A second sub-data lineage is generated according to the association relationship between the plurality of entity nodes.

[0175] In a possible implementation manner, the first generating unit is specifically configured to:

[0176] Parsing unstructured text information corresponding to the time series node in the unstructured data;

[0177] Determine the type and name of the entity node and the starting position and ending position of the entity node in the unstructured data from the unstructured text information;

[0178] Based on the type and name of the entity node and the start position and end position of the entity node in the unstructured data, an entity node corresponding to the time series node is determined.

[0179] In a possible implementation manner, the second generating unit is specifically configured to:

[0180] Determine an original field from the first sub-data lineage and / or the second sub-data lineage;

[0181] The original field is derived according to the data lineage generation rule of the derived field to generate data lineage.

[0182] In a possible implementation manner, the second generating unit is specifically configured to:

[0183] When the original field is a time interval, the difference between the start time and the end time is calculated;

[0184] When the difference is greater than or equal to 1 year, the time interval is displayed in years in the generated data lineage; when the difference is less than 1 year and greater than or equal to 1 month, the time interval is displayed in months in the generated data lineage; when the difference is less than 1 month and greater than or equal to 1 day, the time interval is displayed in days in the generated data lineage; when the difference is less than 1 day, the time interval is displayed in hours in the generated data lineage.

[0185] In a possible implementation manner, the second generating unit is specifically configured to:

[0186] When the original field is the name of the entity node, data cleaning is performed on the original field based on the knowledge base to obtain a derived field;

[0187] The original fields in the first sub-data lineage and / or the second sub-data lineage are displayed as the derived fields to generate data lineage.

[0188] In a possible implementation, the attribution analysis unit is specifically configured to:

[0189] Determine the priority of the application layer link event based on the occurrence frequency and importance level of each field in the data lineage;

[0190] According to the priority of the application layer link event, according to the type of the application layer link event, based on data lineage, attribution analysis is performed on the application layer link event to obtain an attribution analysis result.

[0191] In summary, an embodiment of the present application provides an attribution analysis device, which generates a first sub-data lineage based on a mapping relationship between fields in structured data, and generates a second sub-data lineage based on an association relationship between entity nodes identified in unstructured data. Based on the first sub-data lineage and the second sub-data lineage, data lineage can be generated. Since both structured data and unstructured data are considered in the process of generating the data lineage, according to the type of application layer link event, the application layer link event is attributed and analyzed based on the data lineage, which can be applicable to various types of data and realize attribution analysis of various types of data.

[0192] An embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory;

[0193] The memory is used to store program code and transmit the program code to the processor;

[0194] The processor is used to execute the steps of the attribution analysis method described above according to the instructions in the program code.

[0195] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of an attribution analysis method as described above are implemented.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An attribution analysis method, characterized in that: include: Classify data to obtain structured data and unstructured data; Generate first sub-data lineage according to the mapping relationship between fields in the structured data; generating second sub-data lineage according to association relationships between entity nodes identified from the unstructured data, wherein the entity nodes correspond to time series nodes in the unstructured data; Generate data lineage based on the first sub-data lineage and the second sub-data lineage; According to the type of the application layer link event, an attribution analysis is performed on the application layer link event based on the data lineage to obtain an attribution analysis result.

2. The method according to claim 1, characterized in that The generating of the second sub-data lineage according to the association relationship between the entity nodes identified from the unstructured data includes: Identify time series nodes from unstructured data; Based on the time series node, determining an entity node corresponding to the time series node from the unstructured data; A second sub-data lineage is generated according to the association relationship between the plurality of entity nodes.

3. The method according to claim 2, characterized in that The determining, based on the time series node, from the unstructured data, an entity node corresponding to the time series node includes: Parsing unstructured text information corresponding to the time series node in the unstructured data; Determine the type and name of the entity node and the starting position and ending position of the entity node in the unstructured data from the unstructured text information; Based on the type and name of the entity node and the start position and end position of the entity node in the unstructured data, an entity node corresponding to the time series node is determined.

4. The method according to claim 1, characterized in that: The generating data lineage based on the first sub-data lineage and the second sub-data lineage includes: determining an original field from the first sub-data lineage and / or the second sub-data lineage; The original field is derived according to the data lineage generation rule of the derived field to generate data lineage.

5. The method according to claim 4, characterized in that The step of performing a derivation process on the original field according to the data lineage generation rule of the derived field to generate data lineage includes: When the original field is a time interval, the difference between the start time and the end time is calculated; When the difference is greater than or equal to 1 year, a time interval is generated in the data lineage and displayed in years; when the difference is less than 1 year and greater than or equal to 1 month, a time interval is generated in the data lineage and displayed in months; when the difference is less than 1 month and greater than or equal to 1 day, a time interval is generated in the data lineage and displayed in days; when the difference is less than 1 day, a time interval is generated in the data lineage and displayed in hours.

6. The method according to claim 4, characterized in that The step of performing a derivation process on the original field according to the data lineage generation rule of the derived field to generate data lineage includes: When the original field is the name of the entity node, data cleaning is performed on the original field based on the knowledge base to obtain a derived field; The original fields in the first sub-data lineage and / or the second sub-data lineage are displayed as the derived fields to generate data lineage.

7. The method according to claim 1, characterized in that The performing attribution analysis on the application layer link event based on the data lineage according to the type of the application layer link event to obtain the attribution analysis result includes: Determining the priority of the application layer link event according to the occurrence frequency identifier and the importance level identifier of each field in the data lineage; According to the priority of the application layer link event, according to the type of the application layer link event, and based on the data lineage, attribution analysis is performed on the application layer link event to obtain an attribution analysis result.

8. An attribution analysis device, characterized in that: include: A classification unit, a first generation unit, a second generation unit, and an attribution analysis unit; The classification unit is used to classify the data to obtain structured data and unstructured data; The first generating unit is used to generate the first sub-data lineage according to the mapping relationship between the fields in the structured data; generating second sub-data lineage according to association relationships between entity nodes identified from the unstructured data, wherein the entity nodes correspond to time series nodes in the unstructured data; The second generating unit is used to generate data lineage based on the first sub-data lineage and the second sub-data lineage; The attribution analysis unit is used to perform attribution analysis on the application layer link event based on the data lineage according to the type of the application layer link event to obtain an attribution analysis result.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of an attribution analysis method as described in any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of an attribution analysis method according to any one of claims 1 to 7 are implemented.

Images