Data cleaning method and system of AI intelligent database

Abstract

The application relates to the technical field of data processing, in particular to an AI intelligent database data cleaning method and system. The method comprises the following steps: acquiring external event data and internal data of an AI intelligent database; acquiring a driving effect coefficient of the external event data on any breakpoint data of the internal data; acquiring an event driving score of the external event data on any breakpoint data according to the driving effect coefficient; and performing adaptive data cleaning on the internal data of the AI intelligent database according to the event driving score. The application confirms the rationality of internal data changes of the database by analyzing the external event driving influence of data breakpoints, thereby dynamically identifying breakpoint data and performing adaptive cleaning on the breakpoint data, can effectively retain structural data changes with business significance, avoids information loss caused by false cleaning, and significantly improves the data quality management capability and decision support level of the intelligent database in a high dynamic environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a data cleaning method and system of an AI intelligent database. BACKGROUND

[0002] The AI intelligent database is a database system combined with artificial intelligence technology, which can automatically manage, analyze and optimize data, automatically identify abnormal values, missing values and noise data in the data by using machine learning and deep learning algorithms, and clean and repair the data, and is suitable for enterprise application scenarios requiring rapid decision-making. Data cleaning is an important part of data preprocessing, and its goal is to improve the accuracy, completeness and consistency of data, including identifying noise data such as missing values, duplicate values, abnormal values, etc., and taking appropriate processing measures. The AI intelligent database can automatically complete data preprocessing and optimization by integrating data cleaning functions, thereby providing a high-quality data basis for data analysis and machine learning model training.

[0003] In the prior art, the data cleaning method based on the AI intelligent database usually relies on predefined rules or expert knowledge, and uses clustering, anomaly detection, regression prediction and other algorithms to identify outliers or complete missing values based on global statistical characteristics of the data. However, there is a lack of identification and modeling of external complex features, and when facing sudden data segments caused by external factors, it is often difficult to accurately capture and identify data changes, and directly applying predefined cleaning strategies may misclassify jump data reflecting real market signals and business changes as abnormal, thereby causing valuable information loss; and if not cleaned, data quality problems will be introduced, reducing the accuracy and reliability of subsequent analysis and modeling. SUMMARY

[0004] To solve the above problems, the application embodiment provides a data cleaning method and system of an AI intelligent database. By analyzing the external event-driven influence of data breakpoints, the rationality of internal data changes of the database is confirmed, so as to dynamically identify breakpoint data and adaptively clean the breakpoint data, so as to effectively retain structural data changes with business significance, avoid information loss caused by false cleaning, and significantly improve the data quality management capability and decision support level of the intelligent database in a high dynamic environment.

[0005] According to a first aspect of the application embodiment, a data cleaning method of an AI intelligent database is provided, and the method comprises:

[0006] obtaining external event data and internal data of the AI intelligent database;

[0007] obtaining a driving effect coefficient of the external event data on any breakpoint data of the internal data;

[0008] According to the driving effect coefficient, an event driving score of the external event data on the any breakpoint data is obtained;

[0009] According to the event driving score, internal data of the AI intelligent database is adaptively cleaned.

[0010] In an embodiment, the driving effect coefficient of the external event data on the any breakpoint data of the internal data comprises:

[0011] A logical jump degree of the neighborhood segment data of the any breakpoint data is obtained;

[0012] A linkage coefficient of the external event data and the neighborhood segment data of the any breakpoint data is obtained;

[0013] A triggering effect coefficient of the external event data on the internal data is obtained;

[0014] According to the logical jump degree, the linkage coefficient, and the triggering effect coefficient, the driving effect coefficient of the external event data on the any breakpoint data of the internal data is obtained.

[0015] In an embodiment, the logical jump degree of the neighborhood segment data of the any breakpoint data comprises:

[0016] A deviation degree of the any breakpoint data of any type data of the internal data is obtained;

[0017] A data type quantity of the internal data is obtained;

[0018] A correlation coefficient of any two types of data in the internal data in the neighborhood segment data of the any breakpoint data is obtained;

[0019] A mean value of the correlation coefficient of any two types of data in the internal data in the neighborhood segment data of all breakpoint data is obtained;

[0020] According to the deviation degree, the data type quantity, the correlation coefficient, and the mean value, the logical jump degree of the neighborhood segment data of the any breakpoint data is obtained.

[0021] In an embodiment, the linkage coefficient of the external event data and the neighborhood segment data of the any breakpoint data comprises:

[0022] A heat index change sequence of the external event data is obtained;

[0023] A data change sequence of the neighborhood segment data of the any breakpoint data of any type data of the internal data is obtained;

[0024] Obtaining the Pearson correlation coefficient between the heat index change sequence and the data change sequence;

[0025] According to the Pearson correlation coefficient, a linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data is obtained.

[0026] In one embodiment, obtaining the trigger effect coefficient of the external event data on the internal data includes:

[0027] Obtaining the event coreness of any keyword in the external event data;

[0028] According to the event coreness, obtaining the semantic matching degree between the external event data and the AI ​​intelligent database;

[0029] According to the semantic matching degree, a triggering effect coefficient of the external event data on the internal data is obtained.

[0030] In one embodiment, obtaining the event coreness of any keyword in the external event data includes:

[0031] Clustering all keywords in the external event data based on point mutual information values ​​between the keywords;

[0032] Obtain the number of keywords in the cluster where any keyword in the external event data is located;

[0033] Obtaining the total number of keywords in the external event data;

[0034] Obtaining a point mutual information value between any two keywords in the external event data;

[0035] The event coreness of any keyword in the external event data is obtained according to the number of keywords, the total number, and the point mutual information value.

[0036] In one embodiment, obtaining the semantic matching degree between the external event data and the AI ​​intelligent database according to the event coreness includes:

[0037] Obtaining the number of intersections between the external event data and the keywords of the internal data;

[0038] Obtaining a union number of keywords of the external event data and the internal data;

[0039] Obtaining a first event coreness of any keyword in the external event data;

[0040] Obtaining a second event coreness of any keyword in the internal data;

[0041] According to the intersection quantity, the union quantity, the first event core degree, and the second event core degree, a semantic matching degree between the external event data and the AI intelligent database is obtained.

[0042] In an implementation manner, the obtaining, according to the semantic matching degree, of a trigger effect coefficient of the external event data on the internal data comprises:

[0043] A type semantic matching degree of the external event data and any type data of the internal data is obtained.

[0044] According to the semantic matching degree between the external event data and the AI intelligent database and the type semantic matching degree, a trigger effect coefficient of the external event data on the internal data is obtained.

[0045] In an implementation manner, the obtaining, according to the driving effect coefficient, of an event driving score of the external event data on the any breakpoint data comprises:

[0046] A set data quantity of neighborhood segment data of the any breakpoint data is obtained.

[0047] A logic jump degree of any two adjacent neighborhood segment data in the set data quantity of neighborhood segment data is obtained.

[0048] According to the set data quantity and the logic jump degree of the any two adjacent neighborhood segment data, a logic recovery degree of the any breakpoint data of any type data in the internal data is obtained.

[0049] According to the driving effect coefficient and the logic recovery degree, an event driving score of the external event data on the any breakpoint data is obtained.

[0050] According to a second aspect of the embodiments of the present application, an AI intelligent database data cleaning system is provided, and the system comprises a database platform, and the database platform comprises:

[0051] a memory having a computer program stored thereon;

[0052] a processor configured to execute the computer program in the memory to implement the steps of the method according to any one of the first aspect.

[0053] In summary, the embodiment of the present application provides a data cleaning method of an AI intelligent database, which comprises: acquiring external event data and internal data of the AI intelligent database; acquiring a driving effect coefficient of the external event data on any breakpoint data of the internal data; acquiring an event driving score of the external event data on the any breakpoint data according to the driving effect coefficient; and performing adaptive data cleaning on the internal data of the AI intelligent database according to the event driving score. The embodiment of the present application confirms the rationality of the change of the internal data of the database by analyzing the external event driving influence of the data breakpoint, thereby dynamically identifying the breakpoint data and performing adaptive cleaning on the breakpoint data, which can effectively retain the structural data variation with business significance, avoid information loss caused by false cleaning, and significantly improve the data quality management capability and decision support level of the intelligent database in a high dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced below, and it should be understood that the drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of the drawings.

[0055] Figure 1 is a flowchart of a data cleaning method of an AI intelligent database according to an exemplary embodiment.

[0056] Figure 2 is a flowchart of a method for acquiring a driving effect coefficient of external event data on any breakpoint data of internal data according to an exemplary embodiment.

[0057] Figure 3 is a flowchart of a method for acquiring a logic jump degree of neighborhood segment data of any breakpoint data according to an exemplary embodiment.

[0058] Figure 4 is a schematic diagram of neighborhood segment data of breakpoint data according to an exemplary embodiment.

[0059] Figure 5 is a flowchart of a method for acquiring a linkage coefficient of external event data and neighborhood segment data of any breakpoint data according to an exemplary embodiment.

[0060] Figure 6 is a flowchart of a method for acquiring a triggering effect coefficient of external event data on internal data according to an exemplary embodiment.

[0061] Figure 7is a flow chart of a method of obtaining an event core degree of any keyword in external event data according to an exemplary embodiment.

[0062] Figure 8 is a flow chart of a method of obtaining a semantic matching degree between external event data and an AI intelligent database according to an event core degree according to an exemplary embodiment.

[0063] Figure 9 is a flow chart of a method of obtaining a trigger effect coefficient of external event data on internal data according to a semantic matching degree according to an exemplary embodiment.

[0064] Figure 10 is a flow chart of a method of obtaining an event driving score of external event data on any breakpoint data according to a driving effect coefficient according to an exemplary embodiment.

[0065] Figure 11 is a block diagram of a data cleaning system of an AI intelligent database according to an exemplary embodiment.

[0066] Figure 12 is a block diagram of a database platform according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] In order to clearly illustrate the technical features of the scheme, specific implementation manners will be described below in conjunction with the drawings.

[0068] Embodiments of the present application will be described below in greater detail with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, but rather these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are for exemplary purposes only, and are not intended to limit the scope of protection of the present application.

[0069] It should be understood that each step described in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0070] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related definitions are given below in the description of the application.

[0071] It should be noted that the terms "first", "second", and the like in the present application are only used to distinguish different devices, modules or units, and are not intended to limit the functions performed by these devices, modules or units or the sequence or interdependence of these functions.

[0072] It should be noted that the terms "one", "multiple" in the present application are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as "one or more". In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two, and other quantifiers are similar; "at least one", "one or more" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one a can represent any number of a; for example, one or more of a, b and c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple; "and / or" is a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean: A alone, A and B together, B alone, of which A and B can be singular or plural.

[0073] In the embodiments of the present application, although the operations or steps are described in a specific order in the accompanying drawings, it should not be understood as requiring the operations or steps to be performed in the specific order or serial order shown, or requiring all the operations or steps to be performed to obtain the desired results. In the embodiments of the present application, the operations or steps can be performed in series; they can also be performed in parallel; or a part of them can be performed.

[0074] At the same time, it can be understood that the data involved in the technical solution (including but not limited to data itself, data acquisition or use) should comply with the requirements of relevant laws, regulations and relevant provisions.

[0075] First, the application scenario of the present application is described. In the prior art, through predefined rules and the logic between data, the process of cleaning AI intelligent database data is realized. If the data jumps due to external event impact, it violates the logic and rules between data, and is easy to be cleaned as abnormal data, resulting in loss of valuable information.

[0076] To solve the problems in the prior art, the application provides a data cleaning method and system of an AI intelligent database. By analyzing the correlation and driving influence of external events on current data changes before and after data breakpoint occurrence, the rationality of data jump is judged, thereby realizing adaptive cleaning of breakpoint jump data of the intelligent database, effectively retaining structural data changes with business significance, avoiding information loss caused by false cleaning, and significantly improving the data quality management capability and decision support level of the intelligent database in a high dynamic environment.

[0077] By constructing logical constraint relationships between data and combining statistical distribution of historical intervals, the application embodiment can automatically identify mutant data points that do not conform to semantics or historical trends, can capture potential logical contradictions between multiple fields, and effectively improves the accuracy of data anomaly identification. By realizing semantic association modeling between database labels and external event texts, whether the current jump is possibly driven by related events can be quickly judged, real data changes affected by macro events can be avoided from being misjudged as data anomalies, and more context-aware data cleaning is realized. By analyzing the time synchronicity and lag responsiveness of external event heat rise and data jump, a causality relationship of event-driven data is constructed, whether the database data is changed due to event driving is judged from the time sequence, the scientificity of jump point discrimination is significantly improved, and the dynamic understanding capability for complex background data is improved. An event-driven scoring model is constructed to quantitatively evaluate each jump breakpoint data, can intelligently distinguish "data to be cleaned" (abnormal data) and "data to be retained" (real signal change data), and automatically decide the corresponding processing mode (such as correction, marking, segmentation, etc.), realizes personalized, adaptive and traceable processing of data cleaning, and significantly improves the data governance capability of the intelligent database in a complex environment. The application will be described below in combination with specific embodiments.

[0078] Figure 1 is a flowchart of a data cleaning method of an AI intelligent database according to an example embodiment. As shown in Figure 1 the application embodiment provides a data cleaning method of an AI intelligent database, which can include the following steps:

[0079] In step S10, external event data and internal data of the AI intelligent database are acquired.

[0080] In this step, external event data and internal data of AI intelligent database are acquired. Exemplarily, data extraction and connection can be performed first, and ETL (Extract-Transform-Load) technology is adopted to extract data from various distributed databases and real-time data streams through a database connector (JDBC) or an API interface; for real-time data, a stream processing platform (Flink) is used to realize low-latency collection. By analyzing the database mode and data dictionary, the definition of each data field, table structure information and related business rules are collected. Then the data sources are integrated, and the external event data related to the business are collected through web crawlers and API interfaces (such as news and social media monitoring). Then the event information is parsed, and the natural language processing technology is used to identify entities and extract keywords from external text data, to generate structured event records with time labels and event categories. Then the heat indicators in the external data, such as news dissemination volume, search index and social media discussion frequency, are integrated as quantitative factors of external event influence. Finally, the data is preprocessed, and since the data formats of different systems may be different, a preprocessing module is used to convert the original data into a unified structured format, and record the timestamps, source identifiers and other key information of the data, and time-align the external event data and the internal data of the AI intelligent database.

[0081] In step S20, a driving effect coefficient of the external event data on any breakpoint data of the internal data is acquired.

[0082] In this step, the driving effect coefficient of the external event data on any breakpoint data of the internal data is acquired. Exemplarily, the logical jump degree of the neighborhood segment data of any breakpoint data can be acquired first, then the linkage coefficient of the external event data and the neighborhood segment data of any breakpoint data is acquired, then the triggering effect coefficient of the external event data on the internal data is acquired, and finally the driving effect coefficient of the external event data on any breakpoint data of the internal data is acquired according to the logical jump degree, the linkage coefficient, and the triggering effect coefficient.

[0083] In step S30, an event driving score of the external event data on the any breakpoint data is acquired according to the driving effect coefficient.

[0084] In this step, according to the driving effect coefficient, the event driving score of the external event data to any breakpoint data is obtained. For example, a set number of adjacent segments of any breakpoint data is obtained first, then the logical jump degree of any two adjacent adjacent segment data in the set number of adjacent segments is obtained, then the logical recovery degree of any breakpoint data of any type data in the internal data is obtained according to the set number and the logical jump degree of any two adjacent adjacent segment data, and finally the event driving score of the external event data to any breakpoint data is obtained according to the driving effect coefficient and the logical recovery degree.

[0085] In step S40, the internal data of the AI intelligent database is adaptively cleaned according to the event driving score.

[0086] In this step, the internal data of the AI intelligent database is adaptively cleaned according to the event driving score. For example, there can be multiple external events, and the maximum event driving score of the data jump point is selected as the judgment basis of the data jump point. If the event driving score is high, it indicates that the data jump is driven by a real event, and it is recommended to retain or partially repair, while retaining the breakpoint mark for subsequent analysis; retaining the jump data of the real event driven can ensure that the data contains important signals reflecting the changes in the external environment, and avoid erasing key information due to uniform cleaning; at the same time, partial repair processing ensures that the data maintains real changes and improves data quality and stability of subsequent modeling.

[0087] Specifically, when the event driving score is greater than a first score threshold (for example, 0.7), the jump data is directly retained, which can reflect real market changes or business adjustments. If the jump data has small noise or local abnormality, the local abnormal value can be corrected (for example, using local mean smoothing or slight correction based on a local statistical model) while retaining the main change. And record the data breakpoint information in the cleaning result, so as to carry out deeper business analysis and risk assessment in the future.

[0088] When the event driving score is less than or equal to the first score threshold (for example, 0.7) and greater than a second score threshold (for example, 0.3), it indicates that the data has certain external event driving signal and also has certain uncertainty or noise, then local repair and slight adjustment can be performed, the system automatically applies local smoothing, local mean correction or local regression model to slightly adjust the data to correct the information with obvious abnormality but without destroying the overall trend, retain the signal brought by real events, and reduce the deviation caused by data noise. At the same time, warning marks and manual review are carried out, the system marks the data jump as "to be reviewed" or "uncertain", records detailed adjustment log and event driving score, and provides basis for subsequent manual review.

[0089] When the event-driven score is less than or equal to a second score threshold (for example, 0.3), it is considered that the low-score jump data is data jump, which is data collection or processing error, and strong cleaning (abnormal data rejection or correction) is performed. The strong cleaning of low-score jump data can remove collection errors and invalid noise, and avoid misleading model training and analysis. Specifically, the strong cleaning can include a forced cleaning strategy for the jump data segment, directly rejecting or correcting abnormal data by a high-intensity data correction method (such as using a global statistical model to re-fill). Abnormal data deletion can also be performed, and if the data is completely inconsistent with the historical benchmark, the data is rejected to ensure the overall data quality, and the cleaning process, data changes before and after cleaning, and decision basis are recorded in detail for subsequent audit and model feedback optimization.

[0090] In summary, the embodiment of the present application provides a data cleaning method of an AI intelligent database, which comprises: acquiring external event data and internal data of the AI intelligent database; acquiring a driving effect coefficient of the external event data on any breakpoint data of the internal data; acquiring an event-driven score of the external event data on the any breakpoint data according to the driving effect coefficient; and performing adaptive data cleaning on the internal data of the AI intelligent database according to the event-driven score. The embodiment of the present application confirms the rationality of the internal data change of the database by analyzing the external event-driven influence of the data breakpoint, thereby dynamically identifying the breakpoint data and performing adaptive cleaning on the breakpoint data, which can effectively retain the structural data variation with business significance, avoid information loss caused by false cleaning, and significantly improve the data quality management capability and decision support level of the intelligent database in a high dynamic environment.

[0091] Figure 2 is a flow chart of a method for acquiring a driving effect coefficient of external event data on internal data according to an exemplary embodiment. As shown in Figure 2 The acquisition of the driving effect coefficient of the external event data on the internal data can include the following steps:

[0092] In step S201, the logical jump degree of the neighborhood segment data of the any breakpoint data is acquired.

[0093] In this step, the logical jump degree of the neighborhood segment data of any breakpoint data is obtained. For example, the deviation degree of any breakpoint data of any type of data in the internal data of the database can be first obtained, then the number of data types in the internal data can be obtained, and then the correlation coefficient of the neighborhood segment data of any two types of data in the internal data can be obtained. Then, the average of the correlation coefficients of the neighborhood segment data of any two types of data in the internal data for all breakpoint data can be obtained. Finally, based on the deviation degree, the number of data types, the correlation coefficient, and the average, the logical jump degree of the neighborhood segment data of any breakpoint data can be obtained.

[0094] In step S202, a linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data is obtained.

[0095] In this step, the linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data is obtained. For example, the heat index change sequence of the external event data can be obtained first, and then the data change sequence of the neighborhood segment data of any breakpoint data of any type of internal data can be obtained. Then, the Pearson correlation coefficient between the heat index change sequence and the data change sequence of the neighborhood segment data is obtained. Finally, based on the Pearson correlation coefficient, the linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data can be obtained.

[0096] In step S203, a trigger effect coefficient of the external event data on the internal data is obtained.

[0097] In this step, the trigger effect coefficient of the external event data on the internal data is obtained. For example, the event coreness of any keyword in the external event data can be obtained first. Then, based on the event coreness, the semantic matching degree between the external event data and the AI ​​intelligent database is obtained. Finally, based on the semantic matching degree, the trigger effect coefficient of the external event data on the internal data is obtained.

[0098] In step S204 , a driving effect coefficient of the external event data on any breakpoint data of the internal data is obtained according to the logic jump degree, the linkage coefficient, and the trigger effect coefficient.

[0099] In this step, according to the logic transition , linkage coefficient , and the trigger effect coefficient , get external event data For the database The driving effect coefficient of the neighborhood data of any breakpoint i of the class data For example, the external event data a is The driving effect coefficient of the neighborhood data of any breakpoint i of the class data It can be obtained by the following formula:

[0100] Formula 1

[0101] in, Indicates the The external event data and the Class data The linkage coefficient of the neighborhood segment data of the breakpoint, Indicates the The external event data is the first The trigger effect coefficient of class data, Indicates the first Class data The logical jump degree of the neighborhood segment data of a breakpoint, Represents an exponential function with the natural number e as its base.

[0102] Indicates the The external event data is the first Class data The linkage (regulation) of the neighborhood segment data of a breakpoint and the difference between the triggering effect coefficient of the external event data on this type of data. The smaller the difference, the more likely it is that the impact of the external event on this type of data matches the linkage change of the data, and the more likely it is that the data jump is driven by the external event.

[0103] To further confirm whether a data jump is caused by an external event, the external event's influence on the breakpoint data type should be synchronized with the actual data linkage changes. The greater the influence on the breakpoint data type, the stronger the linkage changes in the actual breakpoint data. Based on the external event's trigger effect coefficient on the breakpoint data item and the linkage coefficient on the breakpoint data change, the driving effect coefficient of the external event on the breakpoint data is obtained.

[0104] Figure 3 FIG. 1 is a flow chart showing a method for obtaining the logical jump degree of the neighborhood segment data of any breakpoint data according to an exemplary embodiment. Figure 3 As shown, obtaining the logical jump degree of the neighborhood segment data of any breakpoint data may include the following steps:

[0105] In step S2011, the deviation of any breakpoint data of any type of data of the internal data is obtained.

[0106] CUSUM (Cumulative Sum) algorithm calculates the difference between each data point and a preset target value (e.g. historical mean or expected value), and accumulates these differences in chronological order. When the accumulated deviation continuously deviates from the target value, the cumulative sum will gradually increase or decrease, and when it exceeds a preset threshold (e.g. 0.7), an alarm is triggered, identifying a breakpoint, i.e. a jump point. Using the CUSUM breakpoint detection algorithm, the jump points of each data sequence in the database can be obtained in real time, and the deviation of each jump point can also be obtained. Since the process of using the CUSUM breakpoint detection algorithm to detect each jump point and the deviation of the jump point belongs to the prior art, it will not be described here.

[0107] When the data jumps at a certain time, the internal logical consistency of the database is destroyed, prompting that the data may be abnormal or caused by external events. By comparing the current data with the historical benchmark, using logical constraint rules, the data jump can be accurately captured, and the logical jump degree of each type of data breakpoint can be obtained.

[0108] Based on the least square principle, a fitting curve of each type of data is drawn, and the data between the minimum point before the breakpoint and the breakpoint forms the neighborhood segment data of the breakpoint.

[0109] Figure 4 is a schematic diagram of neighborhood segment data of a breakpoint according to an exemplary embodiment. As shown in Figure 4 , the data segment between the breakpoint data B and the minimum value A is the neighborhood segment data of the breakpoint data B.

[0110] In this step, the deviation of any breakpoint data i of any type of data o of the internal data of the database is obtained .

[0111] In step S2012, the number of data types of the internal data is obtained.

[0112] In this step, the number of data types of the internal data of the database is obtained .

[0113] In step S2013, the correlation coefficient of any two types of data in the internal data in the neighborhood segment data of the any breakpoint data is obtained.

[0114] In this step, the correlation coefficient between the corresponding data of any two types of data o and j in the internal data of the database on the time sequence of the neighborhood segment data of any breakpoint data i is obtained . Exemplarily, the correlation coefficient The Pearson correlation coefficient can be the corresponding data in the time series of the neighborhood segments of any two data types o and j in the database internal data at any breakpoint data i. It should be understood that if the lengths of the neighborhood segments of any two data types o and j at any breakpoint data i are inconsistent, the smaller neighborhood segment data can be interpolated using data interpolation to ensure that the lengths of the two neighborhood segments are consistent.

[0115] In step S2014, the mean value of the correlation coefficient of any two types of data in the internal data in the neighborhood segment data of all breakpoint data is obtained.

[0116] In this step, the mean of the correlation coefficients of any two types of data o and j in the neighborhood of all breakpoint data is obtained. .

[0117] In step S2015, the logical jump degree of the neighborhood segment data of any breakpoint data is obtained according to the deviation, the number of data types, the correlation coefficient, and the mean.

[0118] In this step, according to the deviation , number of data types , correlation coefficient , and the mean , obtain the logical jump degree of the neighborhood segment data of any breakpoint data i of any type of data o in the internal data of the database For example, the logical jump degree of the neighborhood segment data of any breakpoint data i of any type of data o in the internal data of the database is It can be obtained by the following formula:

[0119] Formula 2

[0120] Wherein, j represents any type of data in the database internal data, and N is not zero.

[0121] Indicates the first The mean of the difference in the correlation between the class data and each other type of data reflects the logical relationship between the internal data of the database. The larger the value, the more likely the logical relationship of the internal data of the database is to be destroyed.

[0122] Figure 5 This is a flow chart showing a method for obtaining a linkage coefficient between external event data and neighboring segment data of any breakpoint data according to an exemplary embodiment. Figure 5 As shown, obtaining the linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data may include the following steps:

[0123] In step S2021, a change sequence of the heat index of the external event data is obtained.

[0124] In this step, a change sequence of the heat index of the external event data a is obtained . For example, the heat index of the external event data a can be integrated by the news spread volume, the search index, the social media discussion frequency, etc. as the heat index of the external event data a, and then the time parameter of the occurrence of the event is added to obtain the change sequence of the heat index of the external event data a .

[0125] In step S2022, a data change sequence of the neighborhood segment data of any breakpoint data of any type of data of the internal data is obtained.

[0126] In this step, a data change sequence of the neighborhood segment data of any breakpoint data i of any type of data o of the internal data of the database is obtained . For example, the data change sequence of the neighborhood segment data of any breakpoint data i of any type of data o of the internal data of the database can be obtained according to the values of the data points of the neighborhood segment data of any breakpoint data i of any type of data o of the internal data of the database and the acquisition time corresponding to the data points .

[0127] In step S2023, a Pearson correlation coefficient of the change sequence of the heat index and the data change sequence is obtained.

[0128] In this step, the Pearson correlation coefficient of the change sequence of the heat index and the data change sequence is obtained. It should be understood that if the lengths of the change sequence of the heat index and the data change sequence are inconsistent, the data interpolation method can be used to interpolate the change sequence with a smaller length to ensure that the lengths of the two change sequences are consistent.

[0129] In this step, the linkage coefficient of the external event data a and the neighborhood segment data of any breakpoint data i of any type of data o of the internal data of the database is obtained according to the Pearson correlation coefficient . For example, the linkage coefficient of the external event data a and the neighborhood segment data of any breakpoint data i of any type of data o of the internal data of the database can be obtained by the following formula:

[0130] ​ Formula 3

[0131] wherein, is normalized.

[0132] Quantify the degree of association between external events and changes in internal data of the database through time series analysis, and obtain the driving quantification of external events on the current data jump.

[0133] Figure 6 is a flow chart of a method for obtaining a triggering effect coefficient of external event data on internal data according to an exemplary embodiment. As shown in Figure 6 obtaining the triggering effect coefficient of the external event data on the internal data can include the following steps:

[0134] In step S2031, the event core degree of any keyword in the external event data is obtained.

[0135] In this step, the event core degree of any keyword in the external event data is obtained. Exemplarily, the point mutual information value between keywords in the external event data can be clustered first, then the number of keywords in the cluster of the external event data is obtained, then the total number of keywords in the external event data is obtained, then the point mutual information value between any two keywords in the external event data is obtained, and finally the event core degree of any keyword in the external event data is obtained according to the keyword number, the total number, and the point mutual information value. In the same way, the event core degree of any keyword in the internal data of the database can also be obtained.

[0136] In step S2032, the semantic matching degree between the external event data and the AI intelligent database is obtained according to the event core degree.

[0137] In this step, the semantic matching degree between the external event data and the AI intelligent database is obtained according to the event core degree. Exemplarily, the number of intersection of keywords of the external event data and the internal data of the database can be obtained first, then the number of union of keywords of the external event data and the internal data of the database is obtained, then the first event core degree of any keyword in the external event data is obtained, then the second event core degree of the corresponding any keyword in the internal data of the database is obtained, and finally the semantic matching degree between the external event data and the AI intelligent database is obtained according to the intersection number, the union number, the first event core degree, and the second event core degree.

[0138] In step S2033, the triggering effect coefficient of the external event data on the internal data is obtained according to the semantic matching degree.

[0139] In this step, the triggering effect coefficient of the external event data on the internal data is obtained according to the semantic matching degree. For example, the type semantic matching degree between the external event data and any type of data in the database internal data is obtained first, and then the triggering effect coefficient of the external event data on the database internal data is obtained according to the semantic matching degree between the external event data and the AI intelligent database and the type semantic matching degree.

[0140] Figure 7 is a flow chart of a method for obtaining the event core degree of any keyword in external event data according to an exemplary embodiment. As shown in Figure 7 , the method for obtaining the event core degree of any keyword in the external event data can include the following steps:

[0141] In step S20311, all keywords in the external event data are clustered by the point mutual information value between the keywords.

[0142] In this step, all keywords in the external event data are clustered by the point mutual information value between the keywords. For example, the keywords can be extracted from external data such as news and announcements by using NLP (Natural Language Processing) technology. At the same time, all keywords of each external event are clustered by the point mutual information value (PMI (Pointwise mutual information)) between the keywords by DBSCAN clustering.

[0143] To analyze the influence of external events on data jump, it is necessary to confirm the association between the external events and the AI intelligent database. If the external events coincide with the research content direction of the database, the external events will have a certain influence on the data change of the database, which is obtained by the matching degree between the external events and the keywords in the database.

[0144] In step S20312, the number of keywords in the cluster where any keyword in the external event data is located is obtained.

[0145] In this step, the number of keywords in the cluster where any keyword l in the external event data a is located is obtained .

[0146] In step S20313, the total number of keywords in the external event data is obtained.

[0147] In this step, the total number of keywords in the external event data a is obtained .

[0148] In step S20314, the point mutual information value between any two keywords in the external event data is obtained.

[0149] In this step, the point mutual information value between any keyword l and other keywords k in the cluster where any keyword l in the external event data a is located is obtained. For example, the point mutual information value between any keyword l and other keywords k in the cluster where any keyword l in the external event data a is located is It can represent the association relationship between two keywords l and k.

[0150] In step S20315, the event coreness of any keyword in the external event data is obtained according to the number of keywords, the total number, and the point mutual information value.

[0151] In this step, based on the number of keywords , total quantity , and the point mutual information value , obtain the event coreness of any keyword l in the external event data a For example, the event coreness of any keyword l in the external event data a is It can be obtained by the following formula:

[0152] Formula 4

[0153] in, is not zero, Not zero.

[0154] Indicates the The first external event The ratio of the number of keywords in the cluster where the keyword is located to the total number of keywords represents the importance of the cluster where the keyword is located in the external event; Indicates the The first external event The mean of the correlation between a keyword and other keywords in its cluster. The larger the value of this formula is, the closer the relationship between the keyword and all keywords is, and the higher its coreness is.

[0155] Figure 8 This is a flow chart showing a method for obtaining semantic matching between external event data and an AI intelligent database based on event coreness according to an exemplary embodiment. Figure 8 As shown, obtaining the semantic matching degree between the external event data and the AI ​​intelligent database according to the event coreness may include the following steps:

[0156] In step S20321, the number of intersection of keywords of the external event data and the internal data is obtained.

[0157] In this step, the number of intersection of keywords of the external event data a and the internal data of the database is obtained .

[0158] In step S20322, the number of union of keywords of the external event data and the internal data is obtained.

[0159] In this step, the number of union of keywords of the external event data a and the internal data of the database is obtained .

[0160] In step S20323, the first event core degree of any keyword in the external event data is obtained.

[0161] In this step, the first event core degree of any keyword m in the external event data a is obtained .

[0162] In step S20324, the second event core degree of the any keyword in the internal data is obtained.

[0163] In this step, the second event core degree of any keyword m in the internal data of the database is obtained .

[0164] In step S20325, the semantic matching degree between the external event data and the AI intelligent database is obtained according to the number of intersection, the number of union, the first event core degree, and the second event core degree.

[0165] In this step, the semantic matching degree between the external event data a and the AI intelligent database is obtained according to the number of intersection , the number of union , the first event core degree , and the second event core degree . For example, the semantic matching degree between the external event data a and the AI intelligent database can be obtained by the following formula:

[0166] Formula 5

[0167] wherein, is not zero, is not zero, represents the exponential function with natural number e as the base.

[0168] represents the first The ratio of the repeated keywords and the total keywords between the external event and the AI intelligent database represents the matching degree of the external event a and the database; The average value representing the difference of the core degree of the external event a and the corresponding all keywords in the database represents the influence of the occurrence of the external event a on the internal data change of the database. The greater the matching degree of the external event a and the database, the greater the influence of the occurrence of the external event a on the internal data change of the database, and the greater the semantic matching degree between the external event a and the AI intelligent database.

[0169] Through the matching degree between the external event and the keywords in the database, the influence of the external event on the internal data change of the database can be obtained. The closer the core degree of the keywords in the external event and the database, the stronger the corresponding relationship at the semantic level.

[0170] Figure 9 is a flow chart of a method for obtaining a triggering effect coefficient of external event data on internal data according to a semantic matching degree according to an exemplary embodiment. As shown in Figure 9 , the method for obtaining the triggering effect coefficient of the external event data on the internal data according to the semantic matching degree can include the following steps:

[0171] In step S20331, the type semantic matching degree of the external event data and any type data of the internal data is obtained.

[0172] In this step, the type semantic matching degree of the external event data a and any type data o of the internal data of the database is obtained in the same way as obtaining the semantic matching degree between the external event data and the AI intelligent database. .

[0173] In step S20332, the triggering effect coefficient of the external event data on the internal data is obtained according to the semantic matching degree between the external event data and the AI intelligent database and the type semantic matching degree.

[0174] In this step, the triggering effect coefficient of the external event data a on the o-type data of the internal data of the database is obtained according to the semantic matching degree between the external event data and the AI intelligent database , and the type semantic matching degree . Exemplarily, the triggering effect coefficient of the external event data a on the o-type data of the internal data of the database can be obtained by the following formula:

[0175] Formula 6 ​​​

[0176] wherein, represents the number of types of data in the AI intelligent database, is not zero; represents the type semantic matching degree between the i-th external event and the o-th type of data in the AI intelligent database, represents the normalization processing, is not zero.

[0177] represents the difference between the semantic matching degree of the i-th external event to the o-th type of data in the database and the semantic matching degree of the i-th external event to the entire database, representing the influence of the external event a on the change of the type of data o. By performing semantic matching between the data item causing the data jump and the external event, it is determined whether the jump caused by the event in the type of data has a driving effect. At the same time, the system compares the semantic matching degrees of other types of data and the same external event, and if the semantic matching of the type of data is significantly higher than other types of data, it indicates that the external event has a strong triggering effect on this type of data.

[0178] is a flow chart of a method for obtaining an event driving score of external event data to any breakpoint data according to a driving effect coefficient according to an exemplary embodiment. As shown in , the method for obtaining an event driving score of external event data to any breakpoint data according to the driving effect coefficient can include the following steps:

[0179] Figure 10 In step S301, a set data quantity of neighborhood segment data of the any breakpoint data is obtained. Figure 10 In this step, a set data quantity of neighborhood segment data of the any breakpoint data i of the o-th type of data inside the database is obtained

[0180] . Exemplarily, the set data quantity may be 5.

[0181] In step S302, the logical jump degree of any two adjacent neighborhood segment data in the set data quantity of neighborhood segment data is obtained. In this step, the logical jump degrees of any two adjacent neighborhood segment data q+1 and q in the set data quantity of neighborhood segment data are obtained

[0182] and .

[0183]

[0184] ​​​​​In step S303, according to the set data quantity and the logic jump degree of any two adjacent neighborhood segment data, the logic recovery degree of any breakpoint data of any type data in the internal data is obtained.

[0185] In this step, according to the set data quantity and the logic jump degree of any two adjacent neighborhood segment data q+1 and q and , the logic recovery degree of any breakpoint data i of any type data o in the internal data of the database is obtained . Exemplarily, the logic recovery degree of any breakpoint data i of any type data o in the internal data of the database can be obtained by the following formula:

[0186] Formula 7

[0187] wherein, -1 is not zero, is a normalization processing.

[0188] represents the average value of the difference of the logic jump degree of the data points in the neighborhood of the i-th breakpoint data of the o-th type data in the internal data of the database, and the greater the value is, the greater the logic jump recovery degree of the breakpoint data i is. The abnormal noise is a short-term deviation, and the logic consistency of other data in the neighborhood is high, so there is no data logic recovery process. If it is noise, the value of the formula is smaller; if it is driven by an external event, there will be a gradual recovery process of data logic, and the value of the formula will be larger.

[0189] When the data jump of the internal data of the database is driven by an external event, it usually means that the market or business environment has changed substantially, thereby changing the data generation mechanism and the internal relationship between data items. With the gradual conduction of external influence, the internal data logic will also be adjusted and tend to a new stable state. If the data jump is caused by random abnormal noise, the internal logic relationship of the data does not change in essence, and the jump is only a short-term deviation. When the data returns, it still maintains the original consistency and does not need to go through an obvious logic adjustment or recovery process. The internal logic recovery degree of the breakpoint data can be obtained through the logic jump of the breakpoint neighborhood segment data.

[0190] When the data jump of the internal data of the database is driven by an external event, it usually means that the market or business environment has changed substantially, thereby changing the data generation mechanism and the internal relationship between data items. With the gradual conduction of external influence, the internal data logic will also be adjusted and tend to a new stable state. If the data jump is caused by random abnormal noise, the internal logic relationship of the data does not change in essence, and the jump is only a short-term deviation. When the data returns, it still maintains the original consistency and does not need to go through an obvious logic adjustment or recovery process. The internal logic recovery degree of the breakpoint data can be obtained through the logic jump of the breakpoint neighborhood segment data.

[0191] In step S304, according to the driving effect coefficient and the logic recovery degree, the event driving score of the external event data to the any breakpoint data is obtained.

[0192] ​In this step, according to the driving effect coefficient and the logical recovery degree , the event-driven score of the external event data a to any breakpoint data i of any type data o in the internal data of the database is obtained . For example, the event-driven score of the external event data a to any breakpoint data i of any type data o in the internal data of the database may be obtained by the following formula:

[0193] Formula 8

[0194] In summary, the embodiment of the present application provides a data cleaning method of an AI intelligent database, which comprises: obtaining external event data and internal data of the AI intelligent database; obtaining a driving effect coefficient of the external event data to any breakpoint data of the internal data; obtaining an event-driven score of the external event data to the any breakpoint data according to the driving effect coefficient; and performing adaptive data cleaning on the internal data of the AI intelligent database according to the event-driven score. Through analyzing the external event-driven influence of the data breakpoint, the embodiment of the present application confirms the rationality of the change of the internal data of the database, thereby dynamically identifying the breakpoint data and performing adaptive cleaning on the breakpoint data, which can effectively retain the structural data variation with business significance, avoid information loss caused by false cleaning, and significantly improve the data quality management capability and decision support level of the intelligent database in a high dynamic environment.

[0195] The present application also provides a computer readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the data cleaning method of the AI intelligent database provided by the present application.

[0196] Figure 11 is a block diagram of a data cleaning system of an AI intelligent database according to an exemplary embodiment. As shown in Figure 11 , the embodiment of the present application provides a data cleaning system 1100 of an AI intelligent database, which comprises a database platform 1200.

[0197] Figure 12 is a block diagram of a database platform according to an exemplary embodiment. For example, the database platform 1200 can be provided as a server. Referring to Figure 12The database platform 1200 includes a processing component 1222, which is further composed of one or more processors, and a memory resource represented by a memory 1232 for storing instructions executable by the processing component 1222, such as an application program. The application program stored in the memory 1232 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1222 is configured to execute the instructions to perform the data cleaning method of the AI intelligent database described above.

[0198] The database platform 1200 can also include a power supply component 1226 configured to perform power management of the database platform 1200, a communication component 1250 configured to connect the database platform 1200 to a network, and an input / output interface 1258. The database platform 1200 can operate based on an operating system stored in the memory 1232.

[0199] In another exemplary embodiment, a computer program product is also provided, which contains a computer program executable by a programmable electronic device, the computer program having code portions for performing the data cleaning method of the AI intelligent database described above when executed by the programmable electronic device.

[0200] The above-described embodiments are merely representative of several embodiments of the present application, which are described in a more specific and detailed manner, but should not be construed as limiting the scope of the present application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A data cleaning method for an AI intelligent database, characterized in that: The method comprises: Acquiring external event data and internal data of the AI ​​intelligent database; Obtaining a driving effect coefficient of the external event data on any breakpoint data of the internal data; Obtaining, according to the driving effect coefficient, an event-driven score of the external event data on any breakpoint data; Performing adaptive data cleaning on the internal data of the AI ​​intelligent database based on the event-driven scoring; The obtaining of the driving effect coefficient of the external event data on any breakpoint data of the internal data includes: Obtaining the logical jump degree of the neighborhood segment data of any breakpoint data; Obtaining a linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data; Obtaining a trigger effect coefficient of the external event data on the internal data; Obtaining a driving effect coefficient of the external event data on any breakpoint data of the internal data according to the logic jump degree, the linkage coefficient, and the trigger effect coefficient; The obtaining of the logical jump degree of the neighborhood segment data of any breakpoint data includes: Obtaining the deviation of any breakpoint data of any type of data of the internal data; Obtain the data type and quantity of the internal data; Obtaining the correlation coefficient between any two types of data in the internal data and the neighborhood segment data of any breakpoint data; Obtain the mean value of the correlation coefficient of any two types of data in the internal data in the neighborhood segment data of all breakpoint data; Obtaining a logical jump degree of neighborhood segment data of any breakpoint data according to the deviation, the number of data types, the correlation coefficient, and the mean; The obtaining of the trigger effect coefficient of the external event data on the internal data includes: Obtaining the event coreness of any keyword in the external event data; According to the event coreness, obtaining the semantic matching degree between the external event data and the AI ​​intelligent database; According to the semantic matching degree, a triggering effect coefficient of the external event data on the internal data is obtained.

2. The data cleaning method of the AI ​​intelligent database according to claim 1, characterized in that: The obtaining of the linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data includes: Obtaining a heat index change sequence of the external event data; Obtaining a data change sequence of neighborhood segment data of any breakpoint data of any type of data of the internal data; Obtaining the Pearson correlation coefficient between the heat index change sequence and the data change sequence; According to the Pearson correlation coefficient, a linkage coefficient between the external event data and the neighborhood segment data of any breakpoint data is obtained.

3. The data cleaning method of the AI ​​intelligent database according to claim 1, characterized in that: The obtaining of the event coreness of any keyword in the external event data includes: Clustering all keywords in the external event data based on point mutual information values ​​between the keywords; Obtain the number of keywords in the cluster where any keyword in the external event data is located; Obtaining the total number of keywords in the external event data; Obtaining a point mutual information value between any two keywords in the external event data; The event coreness of any keyword in the external event data is obtained according to the number of keywords, the total number, and the point mutual information value.

4. The data cleaning method of the AI ​​intelligent database according to claim 1, characterized in that: The obtaining of the semantic matching degree between the external event data and the AI ​​intelligent database according to the event coreness includes: Obtaining the number of intersections between the external event data and the keywords of the internal data; Obtaining a union number of keywords of the external event data and the internal data; Obtaining a first event coreness of any keyword in the external event data; Obtaining a second event coreness of any keyword in the internal data; According to the number of intersections, the number of unions, the first event coreness, and the second event coreness, a semantic matching degree between the external event data and the AI ​​intelligent database is obtained.

5. The data cleaning method of the AI ​​intelligent database according to claim 1, characterized in that: The obtaining, based on the semantic matching degree, a triggering effect coefficient of the external event data on the internal data includes: Obtaining a type semantic matching degree between the external event data and any type of data of the internal data; According to the semantic matching degree between the external event data and the AI ​​intelligent database, and the type semantic matching degree, a triggering effect coefficient of the external event data on the internal data is obtained.

6. The data cleaning method of the AI ​​intelligent database according to claim 1, characterized in that: The step of obtaining, based on the driving effect coefficient, an event-driven score of the external event data on any breakpoint data includes: Obtaining a set number of data of the neighborhood segment data of any breakpoint data; Obtaining the logic jump degree of any two adjacent neighborhood segment data in the set number of neighborhood segment data; Obtaining the logic recovery degree of any breakpoint data of any type of data in the internal data according to the set data quantity and the logic jump degree of any two adjacent neighborhood segment data; An event driving score of the external event data for any breakpoint data is obtained according to the driving effect coefficient and the logic recovery degree.

7. A data cleaning system for an AI intelligent database, characterized in that: The system includes a database platform, and the database platform includes: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.

Images