Intelligent optimization methods applied to data quality

Abstract

The present invention provides an intelligent optimization method for data quality, including the following specific steps: collecting data resources from various data sources; evaluating the data resources collected by the data sources to obtain quality evaluation results, and selecting preferred data resources according to the quality evaluation results of the data resources; obtaining the business needs and goals of the required data, identifying the required data resources from the business needs and goals and marking them as required data resources; optimizing the quality of the required data resources corresponding to the preferred data resources to obtain optimized data resources; applying the optimized data resources to the required data resources to meet business needs. The present invention uses an adversarial network algorithm to carry out deep training on the generator and the discriminator, realizes automatic processing and optimization of data to obtain high-quality data results, and at the same time, relies on repeated iterative optimization operations to effectively guarantee its excellent performance level and stable operating state, meet multiple business needs, and achieve optimization of data quality.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality optimization, and in particular to an intelligent optimization method applied to data quality. Background Art

[0002] With the rapid development of big data technology, enterprises and organizations are facing an increasing amount of data and a complex and ever-changing data environment. These data cover a variety of sources, forming a huge and intricate network of data resources. However, the explosive growth and diversity of this data has brought great challenges to data management and quality control.

[0003] Although there are a variety of data quality management tools and techniques, most methods still face challenges when dealing with large-scale and diverse data sets. Traditional data cleaning and preprocessing techniques often rely on fixed rules and manual intervention, which is not only inefficient but also difficult to adapt to the dynamic changes and complexity of data. In addition, as business needs continue to evolve, the requirements for data quality are also increasing, and more intelligent and automated methods are needed to optimize data quality. Therefore, we propose an intelligent optimization method for data quality to solve the problems encountered above. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an intelligent optimization method for data quality to solve the problems raised in the above background technology.

[0005] The purpose of the present invention can be achieved by the following technical solution: comprising the following specific steps:

[0006] S1: Collect data resources from various data sources;

[0007] S2: Evaluate the data resources collected by the data source, obtain quality evaluation results, and select preferred data resources based on the quality evaluation results of the data resources;

[0008] S3: Obtain the business requirements and goals of the required data, identify the required data resources from the business requirements and goals, and mark them as required data resources;

[0009] S4: Optimize the quality of the required data resources corresponding to the preferred data resources to obtain optimized data resources;

[0010] S5: Apply optimized data resources to demand data resources to meet business needs.

[0011] Preferably, based on step S2, the data resources collected by the data source are evaluated, specifically:

[0012] S21: classify various data sources according to data labels to obtain a number of classified data resources;

[0013] S22: Perform real-time analysis on the data quality in the classified data resources to obtain real-time analysis results; wherein the real-time analysis results include subset score values ​​of subset data in the classified data resources and data score values ​​corresponding to the actual data group and sub-source values ​​of the classified data resources in the data source;

[0014] S23: performing change analysis on the data quality in the classified data resources to obtain a change fluctuation value of the data quality;

[0015] S24: weighting the source value and the change fluctuation value to obtain a quality assessment value of the data source corresponding to the classified data resource;

[0016] S25: marking the source values, change fluctuation values ​​and quality assessment values ​​of the classified data resources in all data sources as quality assessment results;

[0017] S26: Selecting a set number of classified data resources corresponding to quality assessment values ​​as preferred sorting data; marking the preferred sorting data of all data sources as preferred data resources.

[0018] As a preferred method, real-time analysis of data quality in classified data resources is performed, specifically:

[0019] S221: Obtain an actual data group of subset data of classified data resources, set a data quality assessment indicator framework, extract expected values ​​of dimensions corresponding to corresponding data in the actual data group from the data quality assessment indicator framework, perform difference calculation between any data in each actual data group in the subset data and the expected value corresponding to its data dimension, and obtain the expected difference value of the corresponding dimension of the data in the actual data group; wherein the dimensions of data quality include authenticity, accuracy, completeness, timeliness, applicability, economy, comparability, coordination and availability;

[0020] S222: Calculate the expected difference values ​​of all dimensions of any data in the actual data group to obtain a data score value of the actual data group;

[0021] S223: Calculate the data score values ​​of all actual data groups in the subset data to obtain the subset score value;

[0022] S224: Sort the classified data resources in order of the size of the subset score values; calculate the subset data of the classified data resources in the data source according to their sequence and subset score values ​​to obtain the sub-source values ​​of the classified data resources in the data source; mark the subset score values ​​of the subset data in the classified data resources and the data score values ​​of the corresponding actual data groups and the sub-source values ​​of the classified data resources in the data source as real-time analysis results.

[0023] As a preferred method, a change analysis is performed on the data quality in the classified data resources, specifically:

[0024] A line graph of data quality change is established, and the source values ​​and their collection times are input into the line graph. The positions of the source values ​​in the line graph are marked as source points, and adjacent source points are connected to obtain source lines. Any source point is used as the initial source point, and the source point at the next collection time of the initial source point is marked as the second source point. A vertical line is made with the second source point as the starting point, and a horizontal line is made with the first source point as the starting point. A source triangle is formed by connecting the vertical vertical line, the horizontal vertical line and the source line, and the area of ​​the source triangle is calculated; the source triangle below the source line is marked as an increasing source value, and the source triangle above the source line is marked as a decreasing source value; a recent quality time zone is set, and all the increasing source values ​​and decreasing source values ​​in the recent quality time zone are summed up to obtain the total increasing source value and the total decreasing source value; the total increasing source value and the total decreasing source value are weighted to obtain the change fluctuation value.

[0025] Preferably, based on step S4, the quality of the required data resource corresponding to the preferred data resource is optimized as follows:

[0026] S41: preprocessing any original data set in the preferred data resource, wherein the preprocessing includes removing noise, outliers, and filling missing values;

[0027] S42: Select some data sets from the preprocessed preferred data resources as training data sets, use the training data sets to train the generator network G, process the generator network, and use the formula Get the loss function L of the generator G ; where z represents a random noise vector, represents the fake data sample generated by the generator, D represents the discriminator, and E represents the expected value. Represents the discriminator to generate samples judgment of authenticity;

[0028] S43: At the same time, train the discriminator network D and process the discriminator network using the formula Get the loss function L of the discriminator D ; Among them, pdata(x) is used to represent the probability of sample data x appearing in the selected training data set;

[0029] S44: Alternately train the generator and the discriminator until the performance of the discriminator tends to be stable.

[0030] Preferably, based on step S44, the generator and the discriminator are trained alternately, as follows:

[0031] S441: Use the back-propagation algorithm to calculate the gradient of the loss relative to the discriminator network parameters;

[0032] S442: Apply the gradient descent algorithm to update the weights and biases of the discriminator network to reduce the value of the loss function;

[0033] S443: Input the discriminator network into the loss function of the discriminator to obtain the loss value P; mark the preset limit time before the current moment as the restricted time zone, and identify the number of training iterations Y in the restricted time zone; calculate the mean of all loss values ​​in the restricted time zone, and use the formula Get the performance mean Among them, Pb represents the loss value of the bth iteration; the variance of the loss value of the restricted time zone is calculated using the formula Obtain a performance fluctuation value β; perform weighted calculation on the performance mean and the performance fluctuation value to obtain a training evaluation value; set a training evaluation threshold and a number limit threshold, compare the training evaluation value with its training evaluation threshold, if the training evaluation value is less than its training evaluation threshold, it indicates that the discriminator performance tends to be stable, indicating that the training is completed; if the training evaluation value is greater than its training evaluation threshold, the total number of training iterations is increased by one; if the number of training iterations is equal to its number limit threshold, reselect part of the data set in the preprocessed preferred data resource to train the discriminator, and repeat steps S42-S44 until the discriminator indicates that the training is completed;

[0034] S444: After the training is completed, the trained generator network G is used to generate new data, and the new data is marked as new data; the trained discriminator network D is used to perform quality assessment on the new data to obtain a data assessment value; a high-quality assessment index is set, and new data with a data assessment value greater than the high-quality assessment index is marked as high-quality data; the high-quality data is combined with the original data set in the preferred data resource to form a quality data resource;

[0035] S445: Post-process the quality data resources to obtain optimized data resources; the post-processing includes but is not limited to standardization, normalization, and encoding conversion.

[0036] Preferably, based on step S1, the data source is provided by multiple channels including cooperative websites, government agencies, research institutions, and commercial data providers.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention uses an adversarial network algorithm to carry out in-depth training on the generator and the discriminator, realizes automatic data processing and optimization to obtain high-quality data results. At the same time, it relies on repeated iterative optimization operations to effectively guarantee its excellent performance level and stable operating state, meet the needs of multiple businesses, and optimize data quality.

[0039] 2. The present invention can also accurately identify and select high-quality classified data resources from multiple data sources to support business needs and goals by analyzing and evaluating the quality of data sources, thereby improving the integration efficiency of data resources, ensuring the accuracy, completeness and applicability of data, and thus optimizing data quality. At the same time, it reduces manual intervention and improves the degree of automation in data processing, effectively reducing the manpower and time costs of data management, and building a stable and reliable data environment for enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0041] Figure 1 It is a flowchart of the intelligent optimization method for data quality applied in the present invention. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "class", "group" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] See also Figure 1 As shown, the intelligent optimization method applied to data quality includes the following specific steps:

[0046] S1: Collecting data resources from various data sources; it should be noted that the data resources include original or unprocessed data sets collected from various different data sources;

[0047] S2: Evaluate the data resources collected by the data source, obtain quality evaluation results, and select preferred data resources based on the quality evaluation results of the data resources;

[0048] S3: Obtain the business requirements and goals of the required data, identify the required data resources from the business requirements and goals, and mark them as required data resources;

[0049] S4: Optimize the quality of the required data resources corresponding to the preferred data resources to obtain optimized data resources;

[0050] S5: Apply optimized data resources to demand data resources to meet business needs;

[0051] In this application, based on step S2, the data resources collected by the data source are evaluated, specifically:

[0052] S21: classify various data sources according to data labels to obtain a number of classified data resources;

[0053] It should be noted that data tags are used to identify and distinguish data of different types, attributes or characteristics, where data tags include but are not limited to information of various dimensions such as the field to which the data belongs, the type of business, the data format, and the degree of importance;

[0054] S22: Perform real-time analysis on the data quality in the classified data resources to obtain real-time analysis results; wherein the real-time analysis results include subset score values ​​of subset data in the classified data resources and data score values ​​corresponding to the actual data group and sub-source values ​​of the classified data resources in the data source;

[0055] S23: performing change analysis on the data quality in the classified data resources to obtain a change fluctuation value of the data quality;

[0056] S24: weighting the source value and the change fluctuation value to obtain a quality assessment value of the data source corresponding to the classified data resource;

[0057] S25: marking the source values, change fluctuation values ​​and quality assessment values ​​of the classified data resources in all data sources as quality assessment results;

[0058] S26: Selecting a set number of classified data resources corresponding to quality assessment values ​​as preferred sorting data; marking the preferred sorting data of all data sources as preferred data resources.

[0059] It should be noted that step 2 firstly classifies various data sources, which helps to more clearly understand the structure and type of data and improve the pertinence and accuracy of subsequent analysis; secondly, real-time analysis of data quality can promptly discover problems in the data, such as defects in data integrity, accuracy, consistency, etc., so as to take targeted measures to improve them; analysis of changes in data quality can provide insights into the dynamic changes in data quality and provide a basis for timely adjustment of optimization strategies; quality assessment values ​​are obtained through weighted processing, which can comprehensively consider the impact of various factors on the quality of data sources, making the assessment results more reasonable and reliable; marking and selecting quality assessment results is conducive to identifying high-quality data resources and providing a high-quality data foundation for subsequent optimization and application, so as to better meet business needs and improve the overall efficiency and effectiveness of data processing and application.

[0060] In this application, the data quality in the classified data resources is analyzed in real time, specifically:

[0061] S221: Obtain an actual data group of a subset of classified data resources, set a data quality assessment indicator framework, extract the expected value of the dimension corresponding to the corresponding data in the actual data group from the data quality assessment indicator framework, perform a difference calculation between any data in each actual data group in the subset data and the expected value corresponding to its data dimension, and obtain the expected difference of the corresponding dimension of the data in the actual data group, expressed as QS; wherein the dimensions of data quality include authenticity, accuracy, completeness, timeliness, applicability, economy, comparability, coordination and availability;

[0062] S222: Calculate the expected difference of all dimensions of any data in the actual data set, using the formula The data score value QR of the actual data group is obtained; wherein r represents the rth data in the actual data group, e represents the weight influence factor of the eth dimension corresponding to the data in the actual data group, reQS represents the expected difference corresponding to the eth dimension of the rth data in the actual data group, and reχ represents the weight corresponding to the expected difference corresponding to the eth dimension of the rth data in the actual data group;

[0063] S223: Calculate the data scores of all actual data groups in the subset data using the formula Get the subset score value RE; where q represents the total number of data in the actual data set, qQR represents the data score value of the qth data in the actual data set, Represents the weight of the data score value of the qth data in the actual data group;

[0064] S224: Sort the classified data resources in the order of the size of the subset score values ​​as t; calculate the subset data of the classified data resources in the data source according to their sequence and subset score values, using the formula Get the source value of the classified data resource in the data source in, They respectively represent the subset score value of the tth subset data in the nth data source of the mth classified data resource and its corresponding weight; the subset score value of the subset data in the classified data resource and the data score value of the corresponding actual data group and the source value of the classified data resource in the data source are marked as real-time analysis results.

[0065] In this application, the data quality in the classified data resources is analyzed for changes, specifically:

[0066] Establish a data quality change line graph, input the source value and its collection time into the line graph, mark the position of the source value in the line graph as a source point, connect adjacent source points to obtain a source line, take any source point as the initial source point, mark the source point at the next collection time of the initial source point as the second source point, draw a vertical line with the second source point as the starting point, draw a horizontal vertical line with the first source point as the starting point, connect the vertical vertical line, the horizontal vertical line and the source line to form a source triangle, and calculate the area of ​​the source triangle; Mark the source triangle below the source line as the rising source value, and mark the source triangle above the source line as the falling source value; set the recent quality time zone, sum up all the rising source values ​​and falling source values ​​in the recent quality time zone to get the total rising source value and the total falling source value, marked as GF1 and GF2 respectively; perform weighted calculation on the total rising source value and the total falling source value, and use the formula GF=GF1*g1+GF2*g2 to get the change fluctuation value GF; wherein g1 and g2 represent the weights corresponding to the total rising source value and the total falling source value respectively.

[0067] In this application, after the application quality assessment analysis, the quality status assessment index of the preferred data resource is also subjected to historical comparison and analysis, specifically:

[0068] Business needs and goals for obtaining the required data Classification of historical data Quality status assessment index of data resources marked as q represents the index of the current quality status evaluation index of the classified data resources applied to the business needs and goals of the required data. and For comparison, if If , it means that the data quality of actual application has been improved. On the contrary, if This indicates that the data quality of actual applications has declined. Indicates the previous quality status assessment index before the current quality status assessment index;

[0069] Set the number of historical selected classification data resources, and arrange and number the quality status assessment indexes of historical selected classification resources in chronological order. k represents the index of the quality status assessment index;

[0070] Establish a historical quality status assessment line chart, input the quality status assessment index of historical selected classified resources and their generation time into the line chart, mark the position of the quality status assessment index in the line chart as the assessment point, connect the adjacent assessment points to get the assessment line, make a vertical line with the assessment point as the origin and intersect it with the position of the corresponding generation time, and make a quadrilateral with the vertical line of the assessment line and its two end assessment points and its generation time. Calculate the area of ​​the quadrilateral to get the assessment face value expressed as LF; use the variance formula to calculate, and use the formula Get the recent quality fluctuation value XG; where LFv represents the evaluation face value of the quadrilateral with index v, It represents the average value of all assessed face values, and G represents the total number of assessed face values. A fluctuation threshold is set. If the recent quality fluctuation value is greater than the fluctuation threshold, the classified data resource with the second highest quality assessment index in the original preferred sorting data is selected as the new preferred sorting data. If the recent quality fluctuation value is less than or equal to the fluctuation threshold, it means that the preferred sorting data meets the requirements.

[0071] It should be noted that through historical comparison and analysis, the optimal or suboptimal data resources can be intelligently selected to ensure the stability and reliability of data quality, thereby improving the efficiency of data resource utilization. On the premise of ensuring data quality, enterprises can reduce unnecessary data resource investment, effectively control costs, and improve the economic benefits of resource management.

[0072] In this application, based on step S4, the quality of the required data resources corresponding to the preferred data resources is optimized as follows:

[0073] S41: preprocessing any original data set in the preferred data resource, wherein the preprocessing includes removing noise, outliers, and filling missing values;

[0074] S42: Select some data sets from the preprocessed preferred data resources as training data sets, use the training data sets to train the generator network G, process the generator network, and use the formula Get the loss function L of the generator G ; where z represents a random noise vector, represents the fake data sample generated by the generator, D represents the discriminator, and E represents the expected value. Represents the discriminator to generate samples judgment of authenticity;

[0075] S43: At the same time, train the discriminator network D and process the discriminator network using the formula Get the loss function L of the discriminator D ; Among them, pdata(x) is used to represent the probability of sample data x appearing in the selected training data set;

[0076] S44: Alternately train the generator and the discriminator until the performance of the discriminator tends to be stable.

[0077] In this application, based on step S44, the generator and the discriminator are trained alternately, as follows:

[0078] S441: Use the back-propagation algorithm to calculate the gradient of the loss relative to the discriminator network parameters;

[0079] S442: Apply the gradient descent algorithm to update the weights and biases of the discriminator network to reduce the value of the loss function;

[0080] S443: Input the discriminator network into the loss function of the discriminator to obtain the loss value P; mark the preset limit time before the current moment as the restricted time zone, and identify the number of training iterations Y in the restricted time zone; calculate the mean of all loss values ​​in the restricted time zone, and use the formula Get the performance mean Among them, Pb represents the loss value of the bth iteration; the variance of the loss value of the restricted time zone is calculated using the formula Obtain a performance fluctuation value β; perform weighted calculation on the performance mean and the performance fluctuation value to obtain a training evaluation value; set a training evaluation threshold and a number limit threshold, compare the training evaluation value with its training evaluation threshold, if the training evaluation value is less than its training evaluation threshold, it indicates that the discriminator performance tends to be stable, indicating that the training is completed; if the training evaluation value is greater than its training evaluation threshold, the total number of training iterations is increased by one; if the number of training iterations is equal to its number limit threshold, reselect part of the data set in the preprocessed preferred data resource to train the discriminator, and repeat steps S42-S44 until the discriminator indicates that the training is completed;

[0081] S444: After the training is completed, the trained generator network G is used to generate new data, and the new data is marked as new data; the trained discriminator network D is used to perform quality assessment on the new data to obtain a data assessment value; a high-quality assessment index is set, and new data with a data assessment value greater than the high-quality assessment index is marked as high-quality data; the high-quality data is combined with the original data set in the preferred data resource to form a quality data resource;

[0082] S445: Post-process the quality data resources to obtain optimized data resources; the post-processing includes but is not limited to standardization, normalization, and encoding conversion.

[0083] It should be noted that by alternately training the generator and the discriminator, through the alternating training strategy, the generator learns how to generate more and more realistic data samples, while the discriminator learns how to distinguish between true and false data more accurately, until the training evaluation value is less than its training evaluation threshold, indicating that the performance of the discriminator tends to be stable, enabling the generator to produce high-quality data, which is similar to the real data set in statistical characteristics and meets specific business needs or data analysis goals.

[0084] In the present application, based on step S1, the data source is provided by multiple channels, including cooperative websites, government agencies, research institutions, and commercial data providers.

[0085] It should be noted that by selecting multiple channels as data sources, the diversity and comprehensiveness of the data are ensured, thereby improving the quality and credibility of the data. Multi-channel data sources help reduce data dependence and reduce the impact of unavailability or failure of single-channel data on the system. It can improve the stability and reliability of the system and provide a better foundation for subsequent optimization of data quality.

[0086] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0087] The above are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An intelligent optimization method applied to data quality, characterized in that: The specific steps include: S1: Collect data resources from various data sources; S2: Evaluate the data resources collected by the data source to obtain quality evaluation results, and select the preferred data resources based on the quality evaluation results of the data resources, specifically: S21: classify various data sources according to data labels to obtain a number of classified data resources; S22: Perform real-time analysis on the data quality in the classified data resources to obtain real-time analysis results; wherein the real-time analysis results include subset score values ​​of subset data in the classified data resources and data score values ​​corresponding to the actual data group and sub-source values ​​of the classified data resources in the data source; wherein the real-time analysis on the data quality in the classified data resources is specifically as follows: S221: Obtain an actual data group of subset data of classified data resources, set a data quality assessment indicator framework, extract expected values ​​of dimensions corresponding to corresponding data in the actual data group from the data quality assessment indicator framework, perform difference calculation between any data in each actual data group in the subset data and the expected value corresponding to its data dimension, and obtain the expected difference value of the corresponding dimension of the data in the actual data group; wherein the dimensions of data quality include authenticity, accuracy, completeness, timeliness, applicability, economy, comparability, coordination and availability; S222: Calculate the expected difference values ​​of all dimensions of any data in the actual data group to obtain a data score value of the actual data group; S223: Calculate the data score values ​​of all actual data groups in the subset data to obtain the subset score value; S224: sorting the classified data resources in the order of the size of the subset score values; calculating the subset data of the classified data resources in the data source according to their sequence and subset score values ​​to obtain the sub-source value of the classified data resources in the data source; marking the subset score value of the subset data in the classified data resources and the data score value of the corresponding actual data group and the sub-source value of the classified data resources in the data source as real-time analysis results; S23: performing change analysis on the data quality in the classified data resources to obtain a change fluctuation value of the data quality; S24: weighting the source value and the change fluctuation value to obtain a quality assessment value of the data source corresponding to the classified data resource; S25: marking the source values, change fluctuation values ​​and quality assessment values ​​of the classified data resources in all data sources as quality assessment results; S26: Selecting a set number of classified data resources corresponding to quality evaluation values ​​as preferred sorting data; marking the preferred sorting data of all data sources as preferred data resources; S3: Obtain the business requirements and goals of the required data, identify the required data resources from the business requirements and goals, and mark them as required data resources; S4: Optimize the quality of the required data resources corresponding to the preferred data resources to obtain optimized data resources; S5: Apply optimized data resources to demand data resources to meet business needs.

2. The intelligent optimization method for data quality according to claim 1, characterized in that: Perform change analysis on data quality in classified data resources, specifically: Establish a data quality change line graph, input the source value and its collection time into the line graph, mark the position of the source value in the line graph as a source point, connect adjacent source points to obtain a source line, take any source point as the initial source point, mark the source point at the next collection time of the initial source point as the second source point, draw a vertical line with the second source point as the starting point, draw a horizontal vertical line with the first source point as the starting point, connect the vertical vertical line, the horizontal vertical line and the source line to form a source triangle, and calculate the area of ​​the source triangle; mark the source triangle below the source line as an increasing source value, and mark the source triangle above the source line as a decreasing source value; Set a recent quality time zone, and sum up all the upgrading source values ​​and degrading source values ​​in the recent quality time zone to obtain the total upgrading source value and the total degrading source value; perform weighted calculation on the total upgrading source value and the total degrading source value to obtain the change fluctuation value.

3. The intelligent optimization method for data quality according to claim 1, characterized in that: Based on step S4, the quality of the preferred data resources corresponding to the required data resources is optimized as follows: S41: preprocessing any original data set in the preferred data resource, wherein the preprocessing includes removing noise, outliers, and filling missing values; S42: Select some data sets from the preprocessed preferred data resources as training data sets, use the training data sets to train the generator network G, process the generator network, and use the formula Get the loss function L of the generator G ; where z represents a random noise vector, represents the fake data sample generated by the generator, D represents the discriminator, and E represents the expected value. Represents the discriminator to generate samples judgment of authenticity; S43: At the same time, train the discriminator network D and process the discriminator network using the formula Get the loss function L of the discriminator D ; Among them, pdata(x) is used to represent the probability of sample data x appearing in the selected training data set; S44: Alternately train the generator and the discriminator until the performance of the discriminator tends to be stable.

4. The intelligent optimization method for data quality according to claim 3, characterized in that: Based on step S44, the generator and the discriminator are trained alternately, as follows: S441: Use the back-propagation algorithm to calculate the gradient of the loss relative to the discriminator network parameters; S442: Apply the gradient descent algorithm to update the weights and biases of the discriminator network to reduce the value of the loss function; S443: Input the discriminator network into the loss function of the discriminator to obtain a loss value P; mark the preset restricted time before the current moment as a restricted time zone, and identify the number of training iterations Y within the restricted time zone; Calculate the average of all loss values ​​in the restricted time zone using the formula Get the performance mean Among them, Pb represents the loss value of the bth iteration; the variance of the loss value of the restricted time zone is calculated using the formula Obtain a performance fluctuation value β; perform weighted calculation on the performance mean and the performance fluctuation value to obtain a training evaluation value; set a training evaluation threshold and a number limit threshold, compare the training evaluation value with its training evaluation threshold, if the training evaluation value is less than its training evaluation threshold, it indicates that the discriminator performance tends to be stable, indicating that the training is completed; if the training evaluation value is greater than its training evaluation threshold, the total number of training iterations is increased by one; if the number of training iterations is equal to its number limit threshold, reselect part of the data set in the preprocessed preferred data resource to train the discriminator, and repeat steps S42-S44 until the discriminator indicates that the training is completed; S444: After the training is completed, the trained generator network G is used to generate new data, and the new data is marked as new data; the trained discriminator network D is used to perform quality assessment on the new data to obtain a data assessment value; a high-quality assessment index is set, and new data with a data assessment value greater than the high-quality assessment index is marked as high-quality data; the high-quality data is combined with the original data set in the preferred data resource to form a quality data resource; S445: Post-process the quality data resources to obtain optimized data resources; the post-processing includes but is not limited to standardization, normalization, and encoding conversion.

5. The intelligent optimization method for data quality according to claim 1, characterized in that: Based on step S1, the data source is provided by multiple channels including cooperative websites, government agencies, research institutions, and commercial data providers.

Images