Data change detection method, device, medium, and program product

Abstract

This application provides a data change detection method, device, medium, and program product, relating to the field of data detection technology. The method includes: sampling data from a target database to obtain a current data sample of the target database; extracting data features from the current data sample; inputting the data features into a pre-trained change detection model to obtain a data change detection result output by the change detection model; wherein the data change detection result is used to characterize whether a data change event has occurred in the target database. This application improves the efficiency of data change detection and enhances the versatility of data change detection applications by training a model to learn the relationship between database data features and data changes, thereby utilizing the trained model to detect whether a data change has occurred based on the current data features of the database.

Description

Technical Field

[0001] This application relates to the field of data detection technology, and more specifically, to a data change detection method, device, medium, and program product. Background Technology

[0002] With the rapid development of information technology, the scale of data held by enterprises and organizations is constantly expanding, making it increasingly common to store this data in multiple different types of databases. In practical applications, it is often necessary to obtain information on changes to data in different databases in a timely manner in order to perform operations such as data synchronization, data analysis, and business process integration.

[0003] Traditional data change detection methods typically employ polling, triggers, or log analysis. However, polling consumes significant system resources and often fails to detect data changes promptly. Trigger-based methods require extensive coding for each database table, increasing development and maintenance costs and making them difficult to apply to heterogeneous databases. Log analysis, on the other hand, is heavily reliant on the database log format, also resulting in high development and maintenance costs, and the significant differences in log formats across different databases limit its versatility.

[0004] In summary, there is an urgent need for a solution that can efficiently and universally detect data changes. Summary of the Invention

[0005] The purpose of this application is to provide a data change detection method, device, medium, and program product to improve the efficiency and versatility of data change detection.

[0006] In a first aspect, embodiments of this application provide a data change detection method, including:

[0007] Data is sampled from the target database to obtain the current data sample of the target database;

[0008] Extract the data features of the current data sample;

[0009] The data features are input into a pre-trained change detection model to obtain the data change detection result output by the change detection model; wherein, the data change detection result is used to characterize whether a data change event has occurred in the target database.

[0010] In this embodiment, the relationship between data features and data changes in the database is learned by training a model. The trained model is then used to detect whether data changes have occurred based on the current data features of the database, which effectively improves the efficiency of data change detection and enhances the versatility of the data change detection application.

[0011] In some embodiments, sampling the target database to obtain a current data sample of the target database includes:

[0012] Determine the database type of the target database; wherein, the database type includes relational databases and non-relational databases;

[0013] Data is sampled from the target database based on the target sampling rules corresponding to the database type to obtain the current data sample of the target database.

[0014] In this embodiment, by selecting the appropriate sampling rules based on the database type for data sampling, data change detection can be performed on different databases, further improving the accuracy of data change detection.

[0015] In some embodiments, extracting the data features of the current data sample includes:

[0016] Determine the data type of the current data sample; wherein, the data type includes structured data and unstructured data;

[0017] Based on the feature extraction rules corresponding to the data type, feature extraction is performed on the current data sample to obtain the data features of the current data sample.

[0018] In this embodiment of the application, by selecting the corresponding feature extraction rules according to the data type of the data sample, the accuracy of data feature extraction is effectively improved, and the accuracy and efficiency of data change detection are further improved.

[0019] In some embodiments, the data change detection method further includes:

[0020] If the data change detection result indicates that a data change event has occurred in the target database, the change information of the data change event is obtained; wherein, the change information includes the change type and the change content;

[0021] The change information will be pushed to the data change business response system.

[0022] In this embodiment of the application, by obtaining the current data change type and content and pushing it to the data change business response system when a data change is determined to have occurred, the flexibility of data change management is effectively improved.

[0023] In some embodiments, obtaining the change information of the data change event includes:

[0024] If the target database is determined to be a relational database, obtain the first data sample from the previous sampling of the target database;

[0025] Identify the field differences between the first data sample and the current data sample;

[0026] The change type and content of the data change event are determined based on the field difference information.

[0027] In this embodiment of the application, for relational databases, the change type and content of data change events are determined by identifying the differences in field information before and after the change, which further improves the efficiency of data change management.

[0028] In some embodiments, obtaining the change information of the data change event includes:

[0029] If the target database is determined to be a non-relational database, obtain the first data sample from the previous sampling of the target database;

[0030] Based on the first data sample and the current data sample, a comprehensive semantic change result of the data change event is determined, and a version difference comparison result of the data change event is determined based on the first data sample and the current data sample.

[0031] The change information of the data change event is determined based on the comprehensive semantic change results and the version difference comparison results.

[0032] In this embodiment of the application, for non-relational databases, the change type and content of data change events are determined by detecting the comprehensive semantic change results before and after the change and the version difference comparison results, which further improves the accuracy of data change management.

[0033] In some embodiments, determining the comprehensive semantic change result of the data change event based on the first data sample and the current data sample includes:

[0034] Text content is extracted from the first data sample and the current data sample respectively to obtain the corresponding first text content data and second text content data;

[0035] Determine the semantic similarity between the first text content data and the second text content data;

[0036] Extract and statistically analyze the first keyword weight information of the first text content data, and extract and statistically analyze the second keyword weight information of the second text content data;

[0037] Based on the first keyword weight information and the second keyword weight information, the keyword change information between the first text content data and the second text content data is determined.

[0038] In this embodiment of the application, the accuracy of data change type detection is further improved by using the semantic similarity and keyword change information before and after data change as a comprehensive semantic change result.

[0039] In some embodiments, determining the version difference comparison result of the data change event based on the first data sample and the current data sample includes:

[0040] Extract the first structural data of the first data sample and the second structural data of the current data sample respectively, and determine the comprehensive structural change result based on the first structural data and the second structural data;

[0041] The first time-series data of the first data sample and the second time-series data of the current data sample are extracted respectively, and the comprehensive numerical change result is determined based on the first time-series data and the second time-series data;

[0042] Obtain the first field version vector corresponding to the first data sample and the second field version vector of the current data sample respectively, and determine the comprehensive version change result based on the first field version vector and the second field version vector.

[0043] In this embodiment of the application, by obtaining various information such as structural changes, time-series numerical changes and version change information before and after data change as version difference comparison results, the flexibility of data change type detection is further improved.

[0044] In some embodiments, determining the change information of the data change event based on the comprehensive semantic change result and the version difference comparison result includes:

[0045] If the semantic similarity is determined to be lower than a preset first similarity threshold, the keyword change information is determined to be a keyword change, and the comprehensive version change result is determined to be a field modification, then the change type of the data change event is determined to be content update.

[0046] If, based on the comprehensive numerical change results, it is determined that the magnitude of the data distribution change is greater than a preset magnitude threshold, and the semantic similarity is lower than a preset second similarity threshold, then the change type of the data change event is determined to be content update.

[0047] If, based on the comprehensive numerical change results, it is determined that the magnitude of the data distribution change is not greater than the magnitude threshold, and the semantic similarity is not lower than the second similarity threshold, then the change type of the data change event is determined to be data correction.

[0048] If, based on the comprehensive structural change results, it is determined that a field has been added or deleted, or that the structure of an array or object has changed, the change type of the data change event is determined to be a structural adjustment.

[0049] If, based on the comprehensive structural change results, it is determined that a non-critical field addition event has occurred, and it is determined that the semantic similarity is higher than the preset third similarity threshold, the change type of the data change event is determined to be data supplementation;

[0050] If a field modification event is determined to have occurred based on the comprehensive structural change results, and the semantic similarity is determined to be higher than the preset fourth similarity threshold, then the change type of the data change event is determined to be data correction.

[0051] In this embodiment of the application, the corresponding data change type is determined by combining the comprehensive semantic change results and the version difference comparison results to represent different situations, which further improves the flexibility of data change type detection.

[0052] Secondly, embodiments of this application provide a data change detection device, including:

[0053] The data sampling module is used to sample data from the target database to obtain the current data sample of the target database;

[0054] The feature extraction module is used to extract the data features of the current data sample;

[0055] The change detection module is used to input the data features into a pre-trained change detection model to obtain the data change detection result output by the change detection model; wherein, the data change detection result is used to characterize whether a data change event has occurred in the target database.

[0056] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any embodiment of the first aspect.

[0057] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0058] Fifthly, embodiments of this application provide a computer program product, which includes a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating a data change detection method provided in an embodiment of this application;

[0061] Figure 2 This is a schematic diagram of the overall process for data change detection provided in the embodiments of this application;

[0062] Figure 3 This is a schematic diagram of the data change type determination process provided in the embodiments of this application;

[0063] Figure 4 This is a schematic diagram of the structure of a data change detection device provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0066] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0067] It should be noted that, in view of the problems existing in the prior art, the embodiments of this application provide a data change detection method, which performs data change detection through artificial intelligence semantic perception across heterogeneous databases. This method can efficiently and accurately obtain data change information across databases, reduce resource consumption and development and maintenance costs, and effectively improve the universality and timeliness of data change acquisition.

[0068] like Figure 1 As shown in the figure, this application provides a data change detection method, which may include the following steps:

[0069] S1. Sample data from the target database to obtain the current data sample of the target database.

[0070] It should be noted that the data change detection method can be executed by a data change detection system. First, a connection is established between the data change detection system and multiple target databases to obtain basic information about each database, including database type (e.g., relational database, non-relational database), table structure, field information, etc., and a unique identifier is assigned to each database. Simultaneously, relevant thresholds and detection cycles for data change detection can be set.

[0071] For example, a lightweight data monitoring program can be deployed in each target database to sample the data in the database at a set period to obtain the corresponding current data sample. Different detection periods can be used for data sampling for different target databases.

[0072] In some embodiments, step S1 may include:

[0073] Determine the database type of the target database; the database type includes relational databases and non-relational databases.

[0074] Data is sampled from the target database based on the target sampling rules corresponding to the database type to obtain the current data sample of the target database.

[0075] It should be noted that different sampling rules can be used for different database types.

[0076] For example, for relational databases, data in a table can be retrieved using SQL queries; for non-relational databases, data can be retrieved using the corresponding API interface.

[0077] Understandably, each sample of data is treated as a data sample, and the sampling time and the database identifier of the sample are recorded.

[0078] S2. Extract the data features of the current data sample.

[0079] Specifically, for each data sample, feature extraction algorithms from artificial intelligence can be used to extract data features.

[0080] In some embodiments, step S2 may include:

[0081] Determine the data type of the current data sample; where data type includes structured data and unstructured data;

[0082] Based on the feature extraction rules corresponding to the data type, feature extraction is performed on the current data sample to obtain the data features of the current data sample.

[0083] It should be noted that different feature extraction rules (algorithms) can be used to extract features from different data types.

[0084] For example, for structured data (such as data in a relational database), features such as field values, data types, and data length can be extracted; for unstructured data (such as data in a document database), natural language processing techniques can be used to extract features such as keywords and semantic vectors.

[0085] Assuming the data sample is D and the extracted feature vector is F, the feature extraction process can be represented as:

[0086]

[0087] in, This represents the feature extraction function.

[0088] S3. Input the data features into the pre-trained change detection model to obtain the data change detection results output by the change detection model; the data change detection results are used to characterize whether a data change event has occurred in the target database.

[0089] It should be noted that the data features of the current data sample acquired and extracted in real time can be input into a pre-trained change detection model to obtain the data change detection results output by the model. These results represent either a data change event occurring in the target database between the current and previous detections, or no data change event occurring in the target database between the current and previous detections.

[0090] Understandably, change detection models can be based on open-source artificial intelligence models and trained by collecting a large number of data samples.

[0091] For example, first, the relevant parameters of the artificial intelligence model are initialized; then, following the process of step S1 above, a certain number of historical data samples and their corresponding change labels (with or without changes) are collected to construct a training dataset. The initialized artificial intelligence model is trained using the training dataset, employing supervised learning algorithms such as support vector machines (SVM) and neural networks to optimize the model parameters, enabling the model to accurately identify data changes.

[0092] For example, let the training dataset be T, containing n samples, each sample From the eigenvector and corresponding change tags Composition, that is The training objective of the model is to minimize the loss function L. Taking a neural network as an example, the loss function can be the cross-entropy loss function, whose formula can be expressed as:

[0093]

[0094] in, It is the model on the sample The probability of a change is predicted.

[0095] Based on this, by training a model to learn the relationship between the data characteristics of the database and data changes, the trained model can be used to detect whether data changes have occurred based on the current data characteristics of the database, which effectively improves the efficiency of data change detection and enhances the versatility of data change detection applications.

[0096] In some embodiments, the data change detection method further includes:

[0097] If the data change detection result indicates that a data change event has occurred in the target database, the change information of the data change event is obtained; the change information includes the change type and the change content.

[0098] The change information will be pushed to the data change business response system.

[0099] It should be noted that when a data change is detected in the database, the type of change (such as insertion, deletion, update, etc.) and the specific content of the change (which may include the specific data before and after the change, the location of the change, the time of the change, etc.) should be further analyzed.

[0100] Then, the identified change information (including change type and change content) is pushed to the data change business response system, so that the data change business response system can perform corresponding business according to the data situation of this change, including data synchronization, data analysis, business process integration, etc.

[0101] like Figure 2 As shown, for example, when a data change is detected, the data change information can be encapsulated in a preset format, including the changed database identifier, table name, field information, change type, change content, and change time, and pushed to the relevant application or system through message queues, API interfaces, etc., for subsequent business processing.

[0102] Based on this, by obtaining the current data change type and content when a data change is detected and pushing it to the data change business response system, the automation and flexibility of data change management are effectively improved.

[0103] In some embodiments, obtaining change information of a data change event includes:

[0104] If the target database is determined to be a relational database, obtain the first data sample from the previous sampling of the target database;

[0105] Identify the field differences between the first data sample and the current data sample;

[0106] The type and content of data change events are determined based on field difference information.

[0107] It should be noted that for relational databases, the type and content of the change can be directly determined by comparing the differences in field values ​​between the data samples before and after the change. The first data sample from the previous sampling is the data sample before the change (old data), and the current data sample collected this time is the data sample after the change (new data).

[0108] For example, the type and content of a change are determined by comparing the field values ​​of two data samples (old and new). Change types typically include: add, delete, and update. Specifically, adding a record refers to a record that exists in the new data but not in the old data (based on the primary key); deleting a record refers to a record that exists in the old data but not in the new data (based on the primary key); and updating a record refers to a record that exists in both the old and new data, but some field values ​​have changed. For updated records, it can be recorded which fields changed, and the values ​​before and after the change.

[0109] For example, suppose the old and new data before and after the change are as follows:

[0110] old_data = [

[0111] {'id': 1, 'name': 'Alice', 'age': 25},

[0112] {'id': 2, 'name': 'Bob', 'age': 30},

[0113] {'id': 3, 'name': 'Charlie', 'age': 35} ]

[0115] new_data = [

[0116] {'id': 1, 'name': 'Alice', 'age': 26}, # Age update

[0117] {'id': 2, 'name': 'Bob', 'age': 30}, # No changes

[0118] {'id': 4, 'name': 'David', 'age': 40} # Add ]

[0120] The change information (change type and change content) obtained through field comparison analysis is as follows:

[0121] Delete record: id=3

[0122] New record: id=4

[0123] Update record: id=1, field 'age' changed from 25 to 26.

[0124] Based on this, for relational databases, by identifying the differences in fields before and after the change, the type and content of the data change event can be determined, further improving the efficiency of data change management.

[0125] In some embodiments, obtaining change information of a data change event includes:

[0126] If the target database is determined to be a non-relational database, obtain the first data sample from the previous sampling of the target database;

[0127] The comprehensive semantic change result of the data change event determined based on the first data sample and the current data sample, and the version difference comparison result of the data change event determined based on the first data sample and the current data sample;

[0128] The change information of data change events is determined based on the comprehensive semantic change results and the version difference comparison results.

[0129] like Figure 3 As shown, it should be noted that for non-relational databases, the specific change information can be determined by combining the semantic information of the data with the version differences before and after the change.

[0130] Based on this, the type and content of data change events are determined by detecting the comprehensive semantic change results before and after the change and comparing the version differences, which further improves the accuracy of data change management.

[0131] In some embodiments, determining the comprehensive semantic change result of a data change event based on a first data sample and a current data sample includes:

[0132] Text content is extracted from the first data sample and the current data sample respectively to obtain the corresponding first text content data and second text content data;

[0133] Determine the semantic similarity between the first text content data and the second text content data;

[0134] Extract and statistically analyze the first keyword weight information of the first text content data, and extract and statistically analyze the second keyword weight information of the second text content data;

[0135] The keyword change information between the first and second text content data is determined based on the weight information of the first and second keywords.

[0136] Specifically, comprehensive semantic change analysis can mainly include two aspects: semantic similarity and keyword weight changes.

[0137] For example, for semantic similarity analysis, the corresponding text content data (such as comments and articles in a document database) is first extracted for the first data sample and the current data sample respectively.

[0138] For the extracted text content data, a pre-trained language model (such as BERT, GPT, etc.) can be used to convert the text into semantic vectors. Let the text before the change (the first text content data) be... The modified text (second text content data) is The corresponding semantic vectors are respectively and The vectorization process can be represented as:

[0139] ,

[0140] in, This represents a pre-trained language model.

[0141] Then, the degree of semantic change in text content can be assessed by calculating the cosine similarity between two semantic vectors. Cosine similarity is a metric used to measure the degree of similarity between two vectors, with values ​​ranging from -1 to 1. A value closer to 1 indicates greater similarity, a value closer to -1 indicates less similarity, and 0 indicates that the two vectors are orthogonal (independent). The calculation of cosine similarity can be expressed as:

[0142]

[0143] For example, when the similarity sim is lower than a set threshold (e.g., 0.8), it can be determined that the text content before and after the change has undergone a significant semantic change; otherwise, it is determined that no significant semantic change has occurred.

[0144] For keyword change detection, keyword sets K1 and K2 can be extracted from the text before and after the change, and the frequency of occurrence (word frequency information) or importance of each keyword in K1 and K2 in the corresponding text can be counted to identify the addition, deletion and modification of keywords.

[0145] The importance of each keyword can be determined using either TF-IDF or TextRank algorithms to obtain the weight of each keyword relative to the keyword set. TF-IDF, or Term Frequency-Inverse Document Frequency, is a commonly used weighting technique in information retrieval and text mining. TF represents term frequency, reflecting how frequently a word appears in a document; IDF represents inverse document frequency, measuring the importance of a word within the entire document set. A higher TF-IDF value indicates that the word is more important in the current document and appears less frequently in other documents. TextRank, on the other hand, is a graph-based text ranking algorithm. It treats sentences or words in the text as nodes in a graph, constructs edges based on the similarity between nodes, and then uses the idea of ​​the PageRank algorithm to calculate the importance score of each node, thereby extracting keywords or key sentences from the text.

[0146] Assume the weight of keyword k1 in K1 is The weight of k1 in K2 is The degree of keyword variation can then be expressed as: ,when When the threshold is exceeded, keyword k1 will be marked as a change keyword.

[0147] Based on this, the accuracy of data change type detection is further improved by using the semantic similarity and keyword change information before and after the data change as a comprehensive semantic change result, which serves as the basis for judging the change type and content.

[0148] In some embodiments, determining the version difference comparison result of a data change event based on a first data sample and a current data sample includes:

[0149] Extract the first structural data of the first data sample and the second structural data of the current data sample respectively, and determine the comprehensive structural change result based on the first structural data and the second structural data;

[0150] The first time series data of the first data sample and the second time series data of the current data sample are extracted respectively, and the comprehensive numerical change result is determined based on the first time series data and the second time series data;

[0151] Obtain the first field version vector corresponding to the first data sample and the second field version vector of the current data sample respectively, and determine the comprehensive version change result based on the first field version vector and the second field version vector.

[0152] Please continue reading Figure 3 It should be noted that the following comparison methods can be used when comparing version differences:

[0153] 1. Obtaining comprehensive structural change results through nested structure comparison: For data with nested structures (such as JSON and XML formats), a recursive traversal method can be used to compare the version differences before and after the change. Let the original document (first data sample) be D1, and the changed document (second data sample) be D2. The recursive comparison process is as follows:

[0154] Add a new field: If a path exists in D2 but not in D1, then all content under path p is newly added content.

[0155] Delete field: If a path exists in D1 but not in D2, then all content under path p is deleted.

[0156] Modify the field: If both D1 and D2 have path p, but the corresponding values ​​are different:

[0157] 1) If the value is an atomic type (such as a string or number), directly compare the differences between the values. If the value is an array type, use set operations (such as Jaccard similarity) to evaluate the changes in the array elements:

[0158] .

[0159] It should be noted that Jaccard similarity is used to compare the similarity and differences between finite sample sets. It is calculated by dividing the number of elements in the intersection of two sets by the number of elements in the union. The value ranges from 0 to 1, and the closer the value is to 1, the more similar the two sets are.

[0160] 2) If the value is an object type, then recursively compare each field of the object.

[0161] 2. Determining the overall numerical change result through time series data change detection: For time series data (such as log records), the changing trends and statistical characteristics of data points can be analyzed, including calculating the mean, standard deviation, and slope k of the data before and after the change. Hypothesis testing (such as t-test) can be used to determine whether the data distribution has changed significantly.

[0162]

[0163] Based on this, when the t-value exceeds the preset critical value, it is determined that the data distribution has changed significantly.

[0164] 3. Determine the overall version change result through version vector comparison: Maintain a version vector for each document (sample data collected each time). ,in Indicates the first i The version number of each field. When a field changes, the version number corresponding to that field increments.

[0165] Then, by comparing the differences between the two version vectors (e.g., the first data sample and the second data sample), it can be determined whether field changes have occurred, and the changed fields can be quickly located.

[0166] in and These represent the first and second versions of the vector, respectively. i Each element.

[0167] Based on this, by acquiring various information such as structural changes, time-series numerical changes, and version change information before and after the data change as the comparison results of version differences, and using this as the basis for judging the type and content of the data change, the flexibility of data change type detection is further improved.

[0168] In some embodiments, determining the change information of a data change event based on comprehensive semantic change results and version difference comparison results may include one or more of the following:

[0169] 1. If the semantic similarity is lower than the preset first similarity threshold, the keyword change information is characterized as a change in keywords, and the comprehensive version change result is characterized as a field being modified, then the change type of the data change event is determined to be content update.

[0170] Understandably, when the analysis shows that the semantic similarity between two sets of data is low and there are keyword changes, and the version vector shows that the field values ​​have been modified, this data change event can be classified as "content update".

[0171] 2. If, based on the comprehensive numerical change results, the magnitude of the data distribution change is determined to be greater than a preset magnitude threshold, and the semantic similarity is determined to be lower than a preset second similarity threshold, the change type of the data change event is determined to be content update;

[0172] It is understandable that if the distribution of time series data changes significantly (t-value exceeds the critical value), and combined with semantic analysis (such as significant differences in the semantics of the business description text corresponding to the time series data), this data change event can be judged as "content update".

[0173] 3. If, based on the comprehensive numerical change results, it is determined that the change in data distribution is no greater than the magnitude threshold and the semantic similarity is no less than the second similarity threshold, the change type of the data change event is determined to be data correction.

[0174] Understandably, if the change in the distribution of time series data is small (the t-value does not exceed the critical value), or only a few outliers are corrected (the overall distribution does not change significantly), and semantic analysis shows that the overall business meaning is consistent (such as correcting typos in log records, but the trend remains unchanged), this data change event can be judged as "data correction".

[0175] 4. When it is determined that a field has been added or deleted based on the comprehensive structural change results, or when it is determined that the structure of an array or object has changed, the change type of the data change event is determined to be structural adjustment;

[0176] Understandably, when a field is added or deleted, or the structure of an array / object is changed, the change type of the data change event can be determined as "structural adjustment".

[0177] 5. If, based on the comprehensive structural change results, it is determined that a non-critical field addition event has occurred, and the semantic similarity is higher than the preset third similarity threshold, the change type of the data change event is determined to be data supplementation;

[0178] Understandably, when it is detected that the database has added content to a non-critical field, and the semantic similarity before and after the change is high, the change type of the data change event can be determined as "data supplementation".

[0179] 6. If a field modification event is determined to have occurred based on the comprehensive structural change results, and the semantic similarity is determined to be higher than the preset fourth similarity threshold, the change type of the data change event is determined to be data correction.

[0180] Understandably, when an error message is detected that has been modified, but the overall semantics remain consistent (such as correcting a spelling error), the change type of this data change event can be determined as "data correction".

[0181] It should be noted that, depending on the different types of data change events that occur, corresponding response strategies can be set, thereby enabling appropriate business processing of data change events based on different response strategies.

[0182] Based on this, by combining the comprehensive semantic change results and the version difference comparison results to represent different situations, the corresponding data change type can be determined, further improving the flexibility of data change type detection.

[0183] To better illustrate the solutions of the embodiments of this application, specific examples are provided below:

[0184] It should be noted that the embodiments of this application can be applied to systems that require obtaining changed data information across databases. Assume the source database is a MySQL relational database used to store enterprise order information, and the target database is a MongoDB document-oriented database used to provide data support for a data analysis system. The specific steps are as follows:

[0185] 1. Database Connection and Initialization: Establish connections between the detection system and MySQL and MongoDB databases using the appropriate database drivers; obtain the structure of the order table in the MySQL database (including fields such as order number, user ID, product ID, order amount, and order time) and the structure of the comment collection in the MongoDB database (including fields such as comment ID, user ID, product ID, comment content, and comment time), and assign unique identifiers to each database. Initialize the AI ​​model parameters, set the data change detection threshold to 0.6 (i.e., when the model predicts a data change probability greater than 0.6, it determines that data has changed), and set the data sampling period to 5 minutes.

[0186] 2. Data Monitoring and Sampling: In the MySQL database, order data samples are obtained by periodically executing the SQL query "SELECT * FROM order_table"; in the MongoDB database, data samples from the comment collection are obtained using the MongoDB API. After each sampling, the sampling time and the database identifier of the sample are recorded.

[0187] 3. Data Feature Extraction: For order data samples from the MySQL database, the values ​​of fields such as order number, order amount, and order time are extracted as features; for comment data samples from the MongoDB database, natural language processing techniques are used to extract keywords from the comment content as features. The corresponding data feature vectors can be obtained according to preset formulas.

[0188] 4. Data Change Model Training: Collect historical data samples from two databases over the past week and their corresponding change labels (determined manually or in conjunction with business logic) to construct a training dataset. Use the training dataset to train the neural network model, adjusting the model parameters to continuously reduce the loss function L until the model achieves good predictive performance, thus obtaining a well-trained change detection model.

[0189] 5. Data Change Detection: Data samples are collected in real time at a set 5-minute interval and input into the trained neural network model. If the model predicts that the probability of a change for a certain order data sample is greater than 0.6, the field values ​​of this sample are further compared with those of the previous sampled data to determine the corresponding change information, including the change type and change content. The change type includes whether the order data has undergone insertion, deletion, or update operations.

[0190] 6. Data Change Information Push: When a data change is detected, the change information can be encapsulated in JSON format and sent to the data change business response system, for example:

[0191] {

[0192] "database_id": "mysql_001",

[0193] "table_name": "Order Table",

[0194] "field": "order amount",

[0195] "change_type": "Update",

[0196] "change_content": "Changed from 100 to 150",

[0197] "change_time": "2024-01-01 10:00:00"

[0198] }

[0199] Please refer to Figure 4 , Figure 4 A block diagram illustrating the composition of a data change detection apparatus provided in some embodiments of this application is shown. It should be understood that this data change detection apparatus is similar to the one described above. Figure 1 Corresponding to the method embodiments, it is able to perform each step involved in the above method embodiments. The specific functions of the data change detection device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0200] Figure 4 The data change detection device includes at least one software function module that can be stored in a memory or embedded in the data change detection device in the form of software or firmware. The data change detection device includes:

[0201] The data sampling module 410 is used to sample data from the target database to obtain the current data sample of the target database;

[0202] Feature extraction module 420 is used to extract data features of the current data sample;

[0203] The change detection module 430 is used to input data features into a pre-trained change detection model and obtain the data change detection result output by the change detection model; wherein, the data change detection result is used to characterize whether a data change event has occurred in the target database.

[0204] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention. The data change detection device provided by the embodiments of the present invention can implement the data change detection method provided by any one of the method embodiments of the present invention.

[0205] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0206] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement any of the methods included in the above-described data change detection method.

[0207] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0208] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0209] Some embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, describes the method described in the method embodiments.

[0210] Some embodiments of this application also provide a computer program product that, when run on a computer, causes the computer to perform the methods described in the method embodiments.

[0211] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0212] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0213] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0214] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0215] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0216] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0217] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data change detection method, characterized in that, include: Data is sampled from the target database to obtain the current data sample of the target database; Extract the data features of the current data sample; The data features are input into a pre-trained change detection model to obtain the data change detection result output by the change detection model; wherein, the data change detection result is used to characterize whether a data change event has occurred in the target database; The method further includes: If the data change detection result indicates that a data change event has occurred in the target database, the change information of the data change event is obtained; wherein, the change information includes the change type and the change content; The change information will be pushed to the data change business response system; The step of obtaining the change information of the data change event includes: If the target database is determined to be a non-relational database, obtain the first data sample from the previous sampling of the target database; Based on the first data sample and the current data sample, a comprehensive semantic change result of the data change event is determined, and a version difference comparison result of the data change event is determined based on the first data sample and the current data sample. The change information of the data change event is determined based on the comprehensive semantic change results and the version difference comparison results.

2. The data change detection method according to claim 1, characterized in that, The step of sampling data from the target database to obtain the current data sample of the target database includes: Determine the database type of the target database; wherein, the database type includes relational databases and non-relational databases; Data is sampled from the target database based on the target sampling rules corresponding to the database type to obtain the current data sample of the target database.

3. The data change detection method according to claim 1, characterized in that, The extraction of data features from the current data sample includes: Determine the data type of the current data sample; wherein, the data type includes structured data and unstructured data; Based on the feature extraction rules corresponding to the data type, feature extraction is performed on the current data sample to obtain the data features of the current data sample.

4. The data change detection method according to claim 1, characterized in that, The step of obtaining the change information of the data change event includes: If the target database is determined to be a relational database, obtain the first data sample from the previous sampling of the target database; Identify the field differences between the first data sample and the current data sample; The change type and content of the data change event are determined based on the field difference information.

5. The data change detection method according to claim 1, characterized in that, The determination of the comprehensive semantic change result of the data change event based on the first data sample and the current data sample includes: Text content is extracted from the first data sample and the current data sample respectively to obtain the corresponding first text content data and second text content data; Determine the semantic similarity between the first text content data and the second text content data; Extract and statistically analyze the first keyword weight information of the first text content data, and extract and statistically analyze the second keyword weight information of the second text content data; Based on the first keyword weight information and the second keyword weight information, the keyword change information between the first text content data and the second text content data is determined.

6. The data change detection method according to claim 5, characterized in that, The determination of the version difference comparison result of the data change event based on the first data sample and the current data sample includes: Extract the first structural data of the first data sample and the second structural data of the current data sample respectively, and determine the comprehensive structural change result based on the first structural data and the second structural data; The first time-series data of the first data sample and the second time-series data of the current data sample are extracted respectively, and the comprehensive numerical change result is determined based on the first time-series data and the second time-series data; Obtain the first field version vector corresponding to the first data sample and the second field version vector of the current data sample respectively, and determine the comprehensive version change result based on the first field version vector and the second field version vector.

7. The data change detection method according to claim 6, characterized in that, The determination of change information for the data change event based on the comprehensive semantic change result and the version difference comparison result includes: If the semantic similarity is determined to be lower than a preset first similarity threshold, the keyword change information is determined to be a keyword change, and the comprehensive version change result is determined to be a field modification, then the change type of the data change event is determined to be content update. If, based on the comprehensive numerical change results, it is determined that the magnitude of the data distribution change is greater than a preset magnitude threshold, and it is determined that the semantic similarity is lower than a preset second similarity threshold, then the change type of the data change event is determined to be content update; If, based on the comprehensive numerical change results, it is determined that the magnitude of the data distribution change is not greater than the magnitude threshold, and the semantic similarity is not lower than the second similarity threshold, then the change type of the data change event is determined to be data correction. If, based on the comprehensive structural change results, it is determined that a field has been added or deleted, or that the structure of an array or object has changed, the change type of the data change event is determined to be a structural adjustment. If, based on the comprehensive structural change results, it is determined that a non-critical field addition event has occurred, and it is determined that the semantic similarity is higher than the preset third similarity threshold, the change type of the data change event is determined to be data supplementation; If a field modification event is determined to have occurred based on the comprehensive structural change results, and the semantic similarity is determined to be higher than the preset fourth similarity threshold, then the change type of the data change event is determined to be data correction.

8. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the data change detection method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the data change detection method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data change detection method according to any one of claims 1-7.

Images