Enterprise data management method and system based on artificial intelligence
By identifying and evaluating Cartesian products using artificial intelligence-based methods and adjusting the incremental data collection interval, the problems of data bloat and duplication during data fusion are solved, ensuring data integration quality and system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU SIXIANG SOFTWARE CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-12
AI Technical Summary
In enterprise big data management, the Cartesian product phenomenon during data fusion leads to data volume expansion, affecting system performance. There is a lack of effective methods to assess the impact and pinpoint the cause, and it is difficult to optimize the incremental data collection interval, resulting in increased duplication and redundancy.
An artificial intelligence-based approach is used to identify the Cartesian product, assess its impact on data fusion processing, acquire incremental duplicate data in real time, analyze the reasons, and adjust the incremental data acquisition interval to reduce duplication and redundancy.
Effectively integrate full and incremental data, reduce system resource pressure, avoid system crashes, optimize data query and processing capabilities, and reduce system load and storage resource waste.
Smart Images
Figure CN122019596A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of enterprise data management technology, specifically an enterprise data management method and system based on artificial intelligence. Background Technology
[0002] In enterprise big data management processes, data fusion is a core task. Data fusion aims to organically combine incremental data collected from different data sources and at different times with the full dataset to build a comprehensive, accurate, and consistent enterprise data view. The full dataset represents the complete data set of the enterprise at a specific point in time, while incremental data is data newly collected after that point. By fusing incremental and full datasets, enterprises can update their data view in a timely manner to reflect the latest business changes.
[0003] In practical data fusion processing, a series of technical problems urgently need to be solved. First, when comparing incremental and full-scale data, the diversity of data sources, inconsistencies in data collection times, and the complexity of data structures easily lead to the Cartesian product phenomenon. The Cartesian product causes a dramatic increase in data volume, resulting in a fused data size far exceeding expectations. This not only increases the burden of data storage but also severely impacts system processing performance, potentially leading to slow system response or even crashes, seriously affecting the enterprise's real-time data query and analysis capabilities.
[0004] Secondly, when a Cartesian product occurs during data fusion, there is currently a lack of effective means to accurately assess its impact on the data fusion process. Without accurate assessment, enterprises struggle to determine whether the data fusion process will proceed smoothly and whether critical information will be missed. An excessively high impact may indicate serious problems in the data fusion process, potentially leading to ineffective integration of full and incremental data, preventing enterprises from obtaining a complete and accurate data view, and consequently affecting data-driven decision-making. Furthermore, even if a high impact of the Cartesian product on data fusion is identified, current technologies struggle to pinpoint the exact cause. During data acquisition, the timing of data collection is crucial. Inappropriate timing can lead to excessive collection of similar data within a short period, increasing duplication and redundancy, while also increasing system load and wasting storage and network resources. However, enterprises often lack effective means to dynamically adjust the incremental data collection interval based on actual conditions, hindering optimized data collection management and making it difficult to improve data quality and system performance while ensuring data timeliness.
[0005] Therefore, the present invention provides an enterprise data management method and system based on artificial intelligence. Summary of the Invention
[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.
[0007] The technical solution adopted by this invention to solve its technical problem is: An artificial intelligence-based enterprise data management method includes: During the data fusion process of incremental data and full data after the collection of enterprise big data, the incremental data and full data are compared and analyzed to identify whether a Cartesian product occurs. When a Cartesian product occurs during data fusion processing, the impact of the Cartesian product on data fusion processing is assessed by evaluating the degree of expansion and overlap of the fused data. When the Cartesian product causes a high impact on the data fusion process, incremental duplicate data is acquired in real time, and incremental data acquisition duplicate analysis is performed on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. When the Cartesian product is caused by repeated incremental data collection, the collection interval of the overlapping incremental data is analyzed to obtain the value of the overlapping incremental data collection interval. Based on the value of the overlapping incremental data collection interval, the adjustment amount of the current incremental data collection interval is obtained, and the incremental data collection interval is adjusted and managed.
[0008] As a further aspect of the present invention, the process of comparing and analyzing incremental data with full data is as follows: In the same business scenario or related subject scenario, a data fusion processing period is set. Within the data fusion processing period, the timestamps of the collected incremental data and the full data are extracted respectively. The time difference between the timestamps of the collected incremental data and the timestamps of the full data is obtained and the ratio is calculated with the duration of the data fusion processing period as the data timestamp difference. The summation and average are then calculated to obtain the data time error analysis value. In the same business scenario or related subject scenario, extract incremental data corresponding to multiple full data, and calculate the sum and mean of the incremental multi-full data ratios corresponding to each multi-full incremental data to obtain the incremental multi-full analysis value.
[0009] As a further aspect of the present invention, the recognition process of the Cartesian product is as follows: The Cartesian product identification value is obtained by summing the time error analysis value and the incremental multi-dimensional analysis value. If the Cartesian product analysis value is greater than the Cartesian product analysis threshold, it is displayed as a Cartesian product signal.
[0010] As a further aspect of the present invention, the process for evaluating the impact of the Cartesian product on data fusion processing is as follows: In the same business scenario or related subject scenario, obtain the actual amount of data after fusion, and calculate the difference with the reasonable amount of data in the business, and calculate the ratio with the actual amount of data after fusion to obtain the value of the degree of data expansion of fusion. In the same business scenario or related subject scenario, extract the core fields in each fused actual data, and compare the core fields in any fused actual data to see if they overlap. The fused actual data with overlapping core fields are considered as duplicate fused data. The number of duplicate merged data points and their proportion of the total invalid data after merging are used as the value of the degree of overlap in the merged data. The Cartesian product impact value is obtained by summing the fusion data expansion value and the fusion data overlap value. If the Cartesian product impact value is greater than the Cartesian product impact threshold, it is displayed as a Cartesian product strong impact signal.
[0011] As a further aspect of the present invention, the process for obtaining incremental duplicate data is as follows: Within the same business scenario or related subject scenario, during the data fusion processing period, the core fields of the incremental data acquired in real time are extracted and used as incremental analysis data. The core fields corresponding to any two incremental analysis data are compared for overlap, and the incremental analysis data with overlapping core fields are marked as incremental overlapping data.
[0012] As a further aspect of the present invention, the process of incremental data acquisition and repeated analysis of the Cartesian product is as follows: Extract the number of repetitions in the incremental overlapping data, and construct a curve showing the change in the number of repetitions in the incremental overlapping data, with the X-axis representing time and the Y-axis representing the number of repetitions in the incremental overlapping data, according to the time when the data was acquired. Extract the amount of invalid data after fusion, and construct a curve showing the change in the amount of invalid data after fusion, with the X-axis representing time and the Y-axis representing the amount of invalid data after fusion, according to the time when the data was acquired. The local change curve between two adjacent coordinate points on the incremental overlap quantity change curve is taken as an incremental overlap quantity analysis curve, and multiple incremental overlap quantity analysis curves are obtained. The local change curve between two adjacent coordinate points on the curve of invalid data quantity change is taken as an invalid data quantity analysis curve, and multiple invalid data quantity analysis curves are obtained. The slope calculation formula is used to calculate and process each incremental overlap quantity analysis curve and invalid data quantity analysis curve to obtain the incremental overlap quantity analysis slope and invalid data quantity analysis slope. The local trend analysis value is obtained by calculating the ratio of the slope of the incremental overlap analysis and the slope of the invalid data analysis on the same time dimension. The standard deviation of the local trend analysis values is calculated to obtain the correlation values of quantity changes.
[0013] As a further aspect of the present invention, the process for determining whether the Cartesian product is caused by repeated incremental data acquisition is as follows: Extract the core fields of the incremental overlapping data and compare them with the core fields of the duplicate fused data. The duplicate fused data that overlaps with the core fields of the incremental overlapping data is taken as overlapping associated data and the ratio is calculated with the total number of duplicate fused data to obtain the field duplicate data ratio. The ratio of the number of duplicate data in a field to the value related to the change in quantity is calculated to obtain the duplicate collection analysis value. If the duplicate collection analysis value is greater than or equal to the duplicate collection analysis threshold, it is determined that the Cartesian product is caused by duplicate incremental data collection.
[0014] As a further aspect of the present invention, the process for obtaining the incremental overlap acquisition interval value is as follows: Extract incremental overlapping data within multiple data fusion processing periods, and extract the data acquisition node corresponding to each incremental overlapping data within the data fusion processing period. Sort the data according to the chronological order within the data fusion processing period, and obtain the acquisition time interval between adjacent incremental overlapping data corresponding to the data acquisition nodes within the data fusion processing period after sorting, as the incremental overlapping acquisition interval value.
[0015] As a further aspect of the present invention, the process for obtaining the current incremental data acquisition interval adjustment is as follows: Compare the values of all incremental overlapping acquisition intervals, and calculate the average of the sum of the largest and smallest incremental overlapping acquisition intervals to obtain the average incremental overlapping acquisition interval. The difference between the average incremental overlap collection interval and the current incremental data collection interval is taken as the absolute value to obtain the adjustment amount of the current incremental data collection interval.
[0016] An artificial intelligence-based enterprise data management system includes: Cartesian product recognition module: During the data fusion processing of incremental and full data after the collection of enterprise big data, the incremental data and full data are compared and analyzed to identify whether a Cartesian product exists. Impact magnitude assessment module: When a Cartesian product occurs during the data fusion process, the module assesses the impact magnitude of the Cartesian product on the data fusion process based on the degree of expansion and overlap of the fused data. Cartesian product cause analysis module: When the Cartesian product causes a high impact on data fusion processing, incremental duplicate data is acquired in real time, and incremental data acquisition duplicate analysis is performed on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. Data Acquisition Adjustment Management Module: When the Cartesian product is caused by repeated incremental data acquisition, the module analyzes the acquisition interval of the overlapping incremental data, obtains the value of the overlapping incremental acquisition interval, and obtains the current adjustment amount of the incremental data acquisition interval based on the value of the overlapping incremental acquisition interval, thereby managing the adjustment of the incremental data acquisition interval.
[0017] The beneficial effects of this invention are as follows: This invention, in the process of data fusion processing of incremental and full data after enterprise big data collection, compares and analyzes the incremental and full data to identify whether a Cartesian product exists. When a Cartesian product appears during data fusion processing, the degree of expansion and overlap of the fused data is assessed to evaluate the impact of the Cartesian product on the data fusion process. Assessing the impact helps ensure that no key information is missed during the data fusion process. When the impact is within a controllable range, it ensures the effective integration of full and incremental data, giving the enterprise a complete data view. It also helps reduce the pressure of data fusion on system resources, avoiding system crashes or performance degradation due to excessive data volume, and providing enterprises with rapid data query and processing capabilities.
[0018] When the Cartesian product causes a high impact on data fusion processing, this invention acquires incremental duplicate data in real time and performs incremental data acquisition duplication analysis on the Cartesian product based on the incremental duplicate data. If the Cartesian product is caused by incremental data acquisition duplication, the acquisition interval analysis is performed on the incremental overlapping data to obtain the incremental overlapping acquisition interval value. Based on the incremental overlapping acquisition interval value, the current incremental data acquisition interval adjustment amount is obtained, and the incremental data acquisition interval is adjusted and managed. Adjusting the acquisition interval can avoid excessive acquisition of similar data in a short period of time, thereby reducing duplication and redundancy in the data. Moreover, reducing unnecessary data acquisition can significantly reduce the system load, avoid storing a large amount of useless duplicate data, reduce unnecessary data transmission, and optimize the use of network resources. Attached Figure Description
[0019] The invention will now be further described with reference to the accompanying drawings.
[0020] Figure 1 This is a flowchart illustrating the steps of an artificial intelligence-based enterprise data management method according to the present invention. Figure 2 This is a flowchart of a module of an artificial intelligence-based enterprise data management system according to the present invention. Detailed Implementation
[0021] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0022] Example 1 When collecting enterprise big data, a combination of full and incremental collection methods is often used. For example, daily full collection of EPR data and real-time incremental collection of CRM customer behavior data. During the fusion of the incremental and full data, duplicate or mismatched association keys in the incremental data can lead to a Cartesian product. This Cartesian product during the fusion process causes a dramatic increase in the volume of the merged data. Furthermore, this dramatic increase in data volume necessitates scanning more records during queries, resulting in excessively long query response times. Therefore, please refer to [the relevant documentation / provisions]. Figure 1 As shown in the figure, an artificial intelligence-based enterprise data management method according to an embodiment of the present invention includes the following steps: Step 1: During the data fusion process of incremental data and full data after the enterprise big data collection, the incremental data and full data are compared and analyzed to identify whether a Cartesian product exists. It's important to clarify that "full data" refers to the complete collection of all valid data in an enterprise's business system at a specific point in time or within a given period. It's essentially a complete backup of the business data. Examples include: an ERP inventory table collected daily at midnight, containing current inventory and historical inbound / outbound records for all products, representing complete inventory data up to that day; or a CRM customer table synchronized weekly, containing complete data such as basic information, purchase records, and tags for all registered customers. Incremental data, on the other hand, refers to data added, modified, or deleted relative to the previous collection (or a specific baseline time). It only includes the changed portions, acting as a dynamic update log for business data. Examples include: real-time collected CRM customer order data, which only includes orders added that day and excludes historical orders. In some embodiments, under the same business scenario or related subject scenario, a data fusion processing period is set. During the data fusion processing period, the timestamps of the collected incremental data and the full data are extracted respectively, and the time difference between the timestamps of the collected incremental data and the timestamps of the full data is obtained. The ratio between the timestamps of the collected incremental data and the timestamps of the full data is calculated and used as the data timestamp difference ratio. The average of the timestamp difference ratios of all data within the data fusion processing period is calculated to obtain the data time error analysis value. In the same business scenario or related subject scenario, extract incremental data (or full data corresponding to multiple full data) as multiple full incremental data (or multiple full data). For example, taking multiple full incremental data as an example, we count the number of full data corresponding to each multiple full incremental data and the proportion of the total number of full data, which is used as the incremental multiple full data ratio. The incremental multiple-to-full analysis value is obtained by summing and averaging the incremental multiple-to-full ratios corresponding to each multiple-to-full incremental data point. The Cartesian product identification value is obtained by summing the data time error analysis value and the incremental multi-full analysis value. Understandably, the Cartesian product analysis value represents the probability of a Cartesian product occurring during the fusion processing of incremental and full data after enterprise big data collection. On one hand, the data time error analysis value reflects the average degree of difference between incremental and full data in the time dimension during the data fusion processing period. On the other hand, the incremental-to-full analysis value reflects the average complexity of the relationship between incremental and full data under the same business scenario or related subject scenario. Specifically, the larger the Cartesian product analysis value, the greater the probability of a Cartesian product phenomenon occurring between incremental and full data in terms of time and relationship during the data fusion processing. Conversely, the smaller the Cartesian product analysis value, the lower the probability of a Cartesian product phenomenon occurring between incremental and full data in terms of time and relationship during the data fusion processing. If the Cartesian product analysis value is greater than the Cartesian product analysis threshold, it indicates that during the data fusion process, the incremental data and the full data are more likely to exhibit a Cartesian product phenomenon in terms of time and correlation, and this is displayed as a Cartesian product signal. If the Cartesian product analysis value is less than or equal to the Cartesian product analysis threshold, it indicates that during the data fusion process, the incremental data and the full data are more likely to exhibit a Cartesian product phenomenon in terms of time and correlation, and are displayed as non-Cartesian product signals.
[0023] Step 2: When a Cartesian product occurs during the data fusion process, the impact of the Cartesian product is analyzed based on the degree of expansion and overlap of the fused data, and the impact of the Cartesian product on the data fusion process is evaluated. In some embodiments, under the same business scenario or related subject scenario, the actual amount of data after fusion is obtained, and the difference is calculated with the reasonable amount of business data. The ratio of the difference with the actual amount of data after fusion is then calculated to obtain the value of the degree of data expansion of fusion. Similarly, in the same business scenario or related subject scenario, extract the core fields in each merged actual data, and compare the core fields in any merged actual data to see if they overlap. Merged actual data with overlapping core fields are considered as duplicate merged data. The number of duplicate merged data points and their proportion of the total invalid data after merging are used as the value of the degree of overlap in the merged data. It should be noted that the amount of invalid data after merging is obtained by subtracting the actual amount of data after merging from the reasonable amount of data for business purposes. The Cartesian product impact value is obtained by summing the value of the expansion degree of the fused data and the value of the overlap degree of the fused data. Understandably, the Cartesian product impact value represents the quantification of the negative impact of the Cartesian product on the entire data fusion process. On one hand, it reflects the degree of abnormal data expansion caused by the Cartesian product after data fusion. On the other hand, it reflects the degree of data duplication caused by the Cartesian product in the fused data. Specifically, the larger the Cartesian product impact value, the greater the impact of the Cartesian product on the data fusion process, resulting in lower quality and usability of the fused data. Conversely, the smaller the Cartesian product impact value, the smaller the impact of the Cartesian product on the data fusion process, resulting in higher quality and usability of the fused data. If the Cartesian product impact value is greater than the Cartesian product impact threshold, it indicates that the Cartesian product has a strong impact on the data fusion process, and is displayed as a strong Cartesian product impact signal. If the Cartesian product impact value is less than or equal to the Cartesian product impact threshold, it indicates that the impact of the Cartesian product on the data fusion processing is relatively weak, and it is displayed as a weak Cartesian product impact signal. The specific solution in this embodiment is as follows: During the data fusion processing of incremental and full data after the enterprise's big data collection, the incremental and full data are compared and analyzed to identify whether a Cartesian product occurs. When a Cartesian product occurs during the data fusion process, the degree of expansion and overlap of the fused data are assessed to evaluate the impact of the Cartesian product on the data fusion process. Assessing the impact helps ensure that no key information is missed during the data fusion process. When the impact is within a controllable range, it can ensure the effective integration of full and incremental data, giving the enterprise a complete data view. It also helps to reduce the pressure of data fusion on system resources, avoid system crashes or performance degradation due to excessive data volume, and provide the enterprise with rapid data query and processing capabilities.
[0024] Example 2 Please see Figure 1 As shown in the embodiment of the present invention, an enterprise data management method based on artificial intelligence further includes the following steps: Step 3: If the Cartesian product causes a high impact on the data fusion process, then acquire incremental duplicate data in real time, and perform incremental data acquisition duplicate analysis on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. In some embodiments, within the same business scenario or related subject scenario, during the data fusion processing period, the core fields of the incremental data acquired in real time are extracted and used as incremental analysis data. Compare the core fields corresponding to any two incremental analysis data, and mark the incremental analysis data with overlapping core fields as incremental overlapping data. Extract the number of repetitions in the incremental overlapping data, and construct a curve showing the change in the number of repetitions in the incremental overlapping data, with the X-axis representing time and the Y-axis representing the number of repetitions in the incremental overlapping data, according to the time when the data was acquired. Extract the amount of invalid data after fusion, and construct a curve showing the change in the amount of invalid data after fusion, with the X-axis representing time and the Y-axis representing the amount of invalid data after fusion, according to the time when the data was acquired. The local change curve between two adjacent coordinate points on the incremental overlap quantity change curve is taken as an incremental overlap quantity analysis curve, and multiple incremental overlap quantity analysis curves are obtained. Similarly, by taking the local change curve between two adjacent coordinate points on the invalid data quantity change curve as an invalid data quantity analysis curve, multiple invalid data quantity analysis curves can be obtained. The slope calculation formula is used to calculate and process each incremental overlap quantity analysis curve and invalid data quantity analysis curve to obtain the incremental overlap quantity analysis slope and invalid data quantity analysis slope. The local trend analysis value is obtained by calculating the ratio of the slope of the incremental overlap analysis and the slope of the invalid data analysis on the same time dimension. The standard deviation of the local trend analysis values is calculated to obtain the correlation values of quantity changes; Extract the core fields of the incremental overlapping data and compare them with the core fields of the duplicate fused data. The duplicate fused data that overlaps with the core fields of the incremental overlapping data is taken as overlapping associated data and the ratio is calculated with the total number of duplicate fused data to obtain the field duplicate data ratio. The ratio of the number of duplicate data in a field to the correlation value of the quantity change is calculated to obtain the duplicate data collection analysis value; It is understandable that the meaning represented by the duplicate data collection analysis value includes multiple factors such as the overlap of incremental data, the trend of changes in invalid data, and the degree of field duplication during the data fusion process. On the one hand, the field duplication ratio reflects the proportion of core fields in the duplicated fused data that overlap with the incremental overlapping data. On the other hand, the change in quantity reflects the consistency between the change trend of invalid data and incremental overlapping data over time. Specifically, if the duplicate data collection analysis value is larger, it means that the duplicate collection not only leads to a large number of duplicates in the incremental data itself and a large number of invalid data, but also a high degree of field duplication. If the duplicate data collection analysis value is smaller, it means that the duplicate collection not only leads to a small number of duplicates in the incremental data itself and a small number of invalid data, but also a low degree of field duplication. If the value of repeated data collection analysis is greater than or equal to the value of repeated data collection analysis threshold, it indicates that repeated data collection not only leads to a large number of duplicate incremental data and a large number of invalid data, but also that the degree of field duplication is high. It is determined that the Cartesian product is caused by repeated incremental data collection. If the value of repeated data collection analysis is less than the repeated data collection analysis threshold, it means that repeated data collection not only results in a small number of repeated incremental data and a small number of invalid data, but also that the severity of field duplication is low. Therefore, it is determined that the Cartesian product is not caused by repeated incremental data collection. Step 4: If the Cartesian product is caused by repeated incremental data collection, then perform collection interval analysis on the incremental overlapping data, obtain the incremental overlapping collection interval value, and obtain the current incremental data collection interval adjustment amount based on the incremental overlapping collection interval value, and adjust and manage the incremental data collection interval. In some embodiments, incremental overlapping data within multiple data fusion processing periods are extracted, and the data acquisition nodes corresponding to each incremental overlapping data within the data fusion processing period are extracted respectively. The data are then sorted according to the chronological order within the data fusion processing period, and the acquisition time interval between the data acquisition nodes corresponding to adjacent incremental overlapping data within the data fusion processing period is obtained as the incremental overlapping acquisition interval value. Compare the values of all incremental overlapping acquisition intervals, and calculate the average of the sum of the largest and smallest incremental overlapping acquisition intervals to obtain the average incremental overlapping acquisition interval. The difference between the average incremental overlap collection interval and the current incremental data collection interval is taken as the absolute value to obtain the adjustment amount of the current incremental data collection interval. The specific solution in this embodiment is as follows: If the Cartesian product causes a high impact on the data fusion process, incremental duplicate data is acquired in real time, and incremental data acquisition duplication analysis is performed on the Cartesian product based on the incremental duplicate data. If the Cartesian product is caused by incremental data acquisition duplication, the acquisition interval analysis is performed on the incremental overlapping data to obtain the incremental overlapping acquisition interval value. Based on the incremental overlapping acquisition interval value, the current incremental data acquisition interval adjustment amount is obtained, and the incremental data acquisition interval is adjusted and managed. Adjusting the acquisition interval can avoid excessive acquisition of similar data in a short period of time, thereby reducing the duplication and redundancy in the data. Moreover, reducing unnecessary data acquisition can significantly reduce the system load, avoid storing a large amount of useless duplicate data, reduce unnecessary data transmission, and optimize the use of network resources.
[0025] Example 3 like Figure 2 As shown, this embodiment of the invention also provides an enterprise data management system based on artificial intelligence, including the following modules: Cartesian product recognition module: During the data fusion processing of incremental and full data after the collection of enterprise big data, the incremental data and full data are compared and analyzed to identify whether a Cartesian product exists. Impact magnitude assessment module: When a Cartesian product occurs during the data fusion process, the module assesses the impact magnitude of the Cartesian product on the data fusion process based on the degree of expansion and overlap of the fused data. Cartesian product cause analysis module: When the Cartesian product causes a high impact on data fusion processing, incremental duplicate data is acquired in real time, and incremental data acquisition duplicate analysis is performed on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. Data Acquisition Adjustment Management Module: When the Cartesian product is caused by repeated incremental data acquisition, the module analyzes the acquisition interval of the overlapping incremental data, obtains the value of the overlapping incremental acquisition interval, and obtains the current adjustment amount of the incremental data acquisition interval based on the value of the overlapping incremental acquisition interval, thereby managing the adjustment of the incremental data acquisition interval.
[0026] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. An enterprise data management method based on artificial intelligence, characterized in that: include: During the data fusion process of incremental data and full data after the collection of enterprise big data, the incremental data and full data are compared and analyzed to identify whether a Cartesian product occurs. When a Cartesian product occurs during data fusion processing, the impact of the Cartesian product on data fusion processing is assessed by evaluating the degree of expansion and overlap of the fused data. When the Cartesian product causes a high impact on the data fusion process, incremental duplicate data is acquired in real time, and incremental data acquisition duplicate analysis is performed on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. When the Cartesian product is caused by repeated incremental data collection, the collection interval of the overlapping incremental data is analyzed to obtain the value of the overlapping incremental data collection interval. Based on the value of the overlapping incremental data collection interval, the adjustment amount of the current incremental data collection interval is obtained, and the incremental data collection interval is adjusted and managed.
2. The enterprise data management method based on artificial intelligence according to claim 1, characterized in that: The process of comparing and analyzing incremental data with full data is as follows: In the same business scenario or related subject scenario, a data fusion processing period is set. Within the data fusion processing period, the timestamps of the collected incremental data and the full data are extracted respectively. The time difference between the timestamps of the collected incremental data and the timestamps of the full data is obtained and the ratio is calculated with the duration of the data fusion processing period as the data timestamp difference. The summation and average are then calculated to obtain the data time error analysis value. In the same business scenario or related subject scenario, extract incremental data corresponding to multiple full data, and calculate the sum and mean of the incremental multi-full data ratios corresponding to each multi-full incremental data to obtain the incremental multi-full analysis value.
3. The enterprise data management method based on artificial intelligence according to claim 2, characterized in that: The process of recognizing the Cartesian product is as follows: The Cartesian product identification value is obtained by summing the time error analysis value and the incremental multi-dimensional analysis value. If the Cartesian product analysis value is greater than the Cartesian product analysis threshold, it is displayed as a Cartesian product signal.
4. The enterprise data management method based on artificial intelligence according to claim 1, characterized in that: The process of assessing the impact of the Cartesian product on data fusion processing is as follows: In the same business scenario or related subject scenario, obtain the actual amount of data after fusion, and calculate the difference with the reasonable amount of data in the business, and calculate the ratio with the actual amount of data after fusion to obtain the value of the degree of data expansion of fusion. In the same business scenario or related subject scenario, extract the core fields in each fused actual data, and compare the core fields in any fused actual data to see if they overlap. The fused actual data with overlapping core fields are considered as duplicate fused data. The number of duplicate merged data points and their proportion of the total invalid data after merging are used as the value of the degree of overlap in the merged data. The Cartesian product impact value is obtained by summing the fusion data expansion value and the fusion data overlap value. If the Cartesian product impact value is greater than the Cartesian product impact threshold, it is displayed as a Cartesian product strong impact signal.
5. The enterprise data management method based on artificial intelligence according to claim 1, characterized in that: The process of obtaining incremental duplicate data is as follows: Within the same business scenario or related subject scenario, during the data fusion processing period, the core fields of the incremental data acquired in real time are extracted and used as incremental analysis data. The core fields corresponding to any two incremental analysis data are compared for overlap, and the incremental analysis data with overlapping core fields are marked as incremental overlapping data.
6. The enterprise data management method based on artificial intelligence according to claim 5, characterized in that: The process of performing incremental data acquisition and repeatability analysis on the Cartesian product is as follows: Extract the number of repetitions in the incremental overlapping data, and construct a curve showing the change in the number of repetitions in the incremental overlapping data, with the X-axis representing time and the Y-axis representing the number of repetitions in the incremental overlapping data, according to the time when the data was acquired. Extract the amount of invalid data after fusion, and construct a curve showing the change in the amount of invalid data after fusion, with the X-axis representing time and the Y-axis representing the amount of invalid data after fusion, according to the time when the data was acquired. The local change curve between two adjacent coordinate points on the incremental overlap quantity change curve is taken as an incremental overlap quantity analysis curve, and multiple incremental overlap quantity analysis curves are obtained. The local change curve between two adjacent coordinate points on the curve of invalid data quantity change is taken as an invalid data quantity analysis curve, and multiple invalid data quantity analysis curves are obtained. The slope calculation formula is used to calculate and process each incremental overlap quantity analysis curve and invalid data quantity analysis curve to obtain the incremental overlap quantity analysis slope and invalid data quantity analysis slope. The local trend analysis value is obtained by calculating the ratio of the slope of the incremental overlap analysis and the slope of the invalid data analysis on the same time dimension. The standard deviation of the local trend analysis values is calculated to obtain the correlation values of quantity changes.
7. The enterprise data management method based on artificial intelligence according to claim 5, characterized in that: The process for determining whether the Cartesian product is caused by repeated incremental data collection is as follows: Extract the core fields of the incremental overlapping data and compare them with the core fields of the duplicate fused data. The duplicate fused data that overlaps with the core fields of the incremental overlapping data is taken as overlapping associated data and the ratio is calculated with the total number of duplicate fused data to obtain the field duplicate data ratio. The ratio of the number of duplicate data in a field to the value related to the change in quantity is calculated to obtain the duplicate collection analysis value. If the duplicate collection analysis value is greater than or equal to the duplicate collection analysis threshold, it is determined that the Cartesian product is caused by duplicate incremental data collection.
8. The enterprise data management method based on artificial intelligence according to claim 1, characterized in that: The process for obtaining the incremental overlap acquisition interval value is as follows: Extract incremental overlapping data within multiple data fusion processing periods, and extract the data acquisition node corresponding to each incremental overlapping data within the data fusion processing period. Sort the data according to the chronological order within the data fusion processing period, and obtain the acquisition time interval between adjacent incremental overlapping data corresponding to the data acquisition nodes within the data fusion processing period after sorting, as the incremental overlapping acquisition interval value.
9. The enterprise data management method based on artificial intelligence according to claim 8, characterized in that: The process for obtaining the current incremental data collection interval adjustment is as follows: Compare the values of all incremental overlapping acquisition intervals, and calculate the average of the sum of the largest and smallest incremental overlapping acquisition intervals to obtain the average incremental overlapping acquisition interval. The difference between the average incremental overlap collection interval and the current incremental data collection interval is taken as the absolute value to obtain the adjustment amount of the current incremental data collection interval.
10. An enterprise data management system based on artificial intelligence, characterized in that: Includes the following modules: Cartesian product recognition module: During the data fusion processing of incremental and full data after the collection of enterprise big data, the incremental data and full data are compared and analyzed to identify whether a Cartesian product exists. Impact magnitude assessment module: When a Cartesian product occurs during the data fusion process, the module assesses the impact magnitude of the Cartesian product on the data fusion process based on the degree of expansion and overlap of the fused data. Cartesian product cause analysis module: When the Cartesian product causes a high impact on data fusion processing, incremental duplicate data is acquired in real time, and incremental data acquisition duplicate analysis is performed on the Cartesian product based on the incremental duplicate data to determine whether the Cartesian product is caused by incremental data acquisition duplicates. Data Acquisition Adjustment Management Module: When the Cartesian product is caused by repeated incremental data acquisition, the module analyzes the acquisition interval of the overlapping incremental data, obtains the value of the overlapping incremental acquisition interval, and obtains the current adjustment amount of the incremental data acquisition interval based on the value of the overlapping incremental acquisition interval, thereby managing the adjustment of the incremental data acquisition interval.