Big data optimization method and system for improving data quality

Abstract

The present application relates to the technical field of electric digital data processing, in particular to a big data optimization method and system for improving data quality. The method comprises the following steps: obtaining full data generated in the process of different user operations on a software client; determining data generated by each functional module in the full data, and performing data consistency analysis on each functional module to determine data difference functional modules; based on the data of each data difference functional module, constructing a training data set of each data difference functional module, training the version conversion functional module respectively to obtain a trained version conversion functional module and a version mapping knowledge base of each data difference functional module, and the version mapping knowledge base of each data difference functional module comprises an optimal format conversion instruction sequence of a historical version, and then the historical version data of each data difference functional module is converted in format to obtain historical version data in the latest version format.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and specifically to a big data optimization method and system for improving data quality. Background Technology

[0002] In big data-driven business decision-making, data consistency is a core element in ensuring the accuracy of analysis. However, because data is generated by different users using software clients, and these users update their software clients at different times, the data received by the server at the same time from different users may be data from different software client versions (including the latest version data and historical version data). This leads to data format heterogeneity issues, which severely restrict data quality.

[0003] While existing technologies can automatically identify version differences, they cannot learn the format evolution patterns between versions. Each version update requires manual reconfiguration of conversion rules, resulting in slow response times and high maintenance costs. Furthermore, different functional modules (such as user information, orders, and logs) have varying data update frequencies and stability, making it difficult for a unified processing strategy to take into account the characteristics of each functional module. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a big data optimization method and system for improving data quality. The specific technical solution adopted is as follows: In a first aspect, the present invention provides a big data optimization method for improving data quality, comprising the following steps: 1) Obtain all data generated during the operation of the software client by different users, wherein the software client includes multiple functional modules; 2) Determine the data generated by each functional module in the full dataset, and perform data consistency analysis on each functional module to identify the functional modules with data discrepancies; 3) Based on the data of each data difference function module, construct the training dataset of each data difference function module, train the version conversion function module respectively, and obtain the trained version conversion function module and version mapping knowledge base of each data difference function module. The version mapping knowledge base of each data difference function module includes the optimal format conversion instruction sequence of historical versions. 4) Using the version conversion function module and version mapping knowledge base trained by each data difference function module, the historical version data of each data difference function module is converted into the latest version format of the historical version data.

[0005] In conjunction with the first aspect mentioned above, among some possible implementation methods, the following methods involve performing data consistency analysis on each functional module to determine the functional modules with data discrepancies: Parse the data of any functional module to determine the attribute groups of each data item in that functional module; Based on the attribute groups of each data point, cluster analysis is performed on all data of this functional module to obtain multiple clusters; The consistency coefficient of a functional module is obtained by calculating the ratio of the number of data entries in the cluster with the most data entries to the total number of data entries in that functional module. Based on the consistency coefficient of each functional module, the functional modules with data differences are identified.

[0006] In conjunction with the first aspect mentioned above, in some possible implementations, the attribute group includes a software version vector, a time attribute feature, a field type distribution vector, and a field name distribution vector.

[0007] In conjunction with the first aspect mentioned above, among some possible implementation methods, the methods for constructing the training dataset for each data difference functional module based on the data from each data difference functional module include: Determine the latest version of the software version among all data entries in any data difference function module, and then mark the user with the latest version of the data as the target user; For each latest version data of each target user, identify the historical version data in the data difference function module that is the same user as the latest version data, has the smallest time difference and the time difference is less than the set time difference limit, and then use the latest version data and the historical version data to construct a mapping data group; Calculate the validity score of each mapped data group, and select the valid mapped data groups as training samples to construct the training dataset for the data difference function module.

[0008] In conjunction with the first aspect mentioned above, among some possible implementations, methods for calculating the validity score of each mapped data group include: The validity score of any mapped data group is obtained by fusing the validity score of the time difference between the latest version data and the historical version data, the validity score of the version difference, the semantic similarity of the data content, and the structural similarity of the data content.

[0009] In conjunction with the first aspect mentioned above, among some possible implementation methods, the method of using the version conversion function module trained by each data difference function module and the version mapping knowledge base to perform format conversion on any historical version data of each data difference function module includes: For any historical version data in any data difference function module, determine the historical version of the historical version data, and determine whether there is an optimal format conversion instruction sequence for the historical version in the version mapping knowledge base of the data difference function module. If it exists, use the optimal format conversion instruction sequence to convert the format of the historical version data. If it does not exist, use the version conversion function module trained by the data difference function module to generate the optimal format conversion instruction sequence for the historical version, and then convert the format of the historical version data.

[0010] In conjunction with the first aspect mentioned above, among some possible implementation methods, the methods for obtaining the version mapping knowledge base for each data difference functional module include: For any mapping data group in the training dataset of any data difference function module, the optimal format conversion instruction sequence obtained after training to convert historical version data to the latest version is used as the optimal format conversion instruction sequence for that historical version, thereby obtaining the version mapping knowledge base of that data difference function module.

[0011] In conjunction with the first aspect mentioned above, in some possible implementations, the version conversion function module is a reinforcement learning neural network with an Actor-Critic architecture.

[0012] In conjunction with the first aspect mentioned above, in some possible implementations, the reward function of the reinforcement learning neural network of the Actor-Critic architecture is the fusion result of the semantic similarity and structural similarity between the data content of historical versions of data and the data content of the latest version of data in any training sample.

[0013] Secondly, the present invention also provides a big data optimization system for improving data quality, including a server. The server includes a data acquisition module and a processing module. The data acquisition module is used to collect all data generated by the user during the use of the software client and send it to the processing module. The processing module implements the big data optimization method for improving data quality of the present invention based on the received full data.

[0014] The present invention has the following beneficial effects: Based on the data of each functional module, the present invention performs data consistency analysis on each functional module, thereby accurately identifying the functional modules with data differences. This allows for subsequent format conversion of the data in these functional modules. Furthermore, using the data from the functional modules with data differences, a training dataset is constructed to train the version conversion functional module, resulting in a trained version conversion functional module and a version mapping knowledge base. This version mapping knowledge base can convert the format of each historical version of the data to the format of the latest version, maintaining data consistency. Moreover, the present invention, considering the different characteristics of the data in different functional modules, constructs separate training datasets for each functional module and trains them separately, thereby improving the accuracy of format conversion for functional modules with data differences. Attached Figure Description

[0015] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0016] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0017] This invention relates to a big data optimization system for improving data quality, comprising a server and multiple software clients. The software clients include a user center module, an order management module, and a software log module. These multiple software clients are installed on different users' devices (computers, smartphones, etc.).

[0018] The server-side includes a data acquisition module, a processing module, and a data lake. The data acquisition module collects all data generated during user interaction with the software client. The data lake stores all data generated during the software client's operation. The processing module implements the big data optimization method for improving data quality based on the all data generated during software client usage. The specific steps of this method are as follows: Figure 1 As shown, it includes the following steps: Step 1: Based on the full data generated during the operation of each software client, determine the functional module to which each piece of data belongs, thereby obtaining the data of each functional module and sending it to the server.

[0019] Each piece of data in the full dataset generated during the software client's operation includes data content, user tag, software version tag, timestamp tag, and associated functional module tag. The user tag identifies the software client from which the data originates; the software version tag, including major version number, minor version number, and revision number, identifies the version of the software client from which the data originates; the timestamp tag determines the time the data was generated; and the associated functional module tag identifies the functional module to which the data belongs.

[0020] In this embodiment, the functional module labels are automatically generated based on the scene data, for example: In scenarios such as user registration, login, and personal information modification, the corresponding data will be labeled as belonging to the "User Center Function Module"; in scenarios such as order creation, payment, and refund, the corresponding data will be labeled as belonging to the "Order Management Function Module"; and in scenarios such as page access, button clicks, and error reporting, the corresponding data will be labeled as belonging to the "Software Log Function Module".

[0021] The data acquisition module collects every piece of data generated during the operation of the software client, encapsulates the collected data, and sends it to the server. Specifically, the data acquisition module uses a unified JSON Schema for data encapsulation to ensure that different versions of data have a consistent outer structure, and transmits the tagged data to the server's data lake (such as Hadoop HDFS or cloud object storage) in real time through the message queue Kafka.

[0022] Step 2: The server-side processing module performs data consistency analysis on each functional module based on the data stored in the data lake, and identifies the functional modules with data discrepancies.

[0023] Because different users update the software client at different times, different software client versions may differ at the same time. This leads to different data formats sent to the server by the data acquisition module, affecting data quality. Furthermore, in addition to format differences due to version variations, the data content within any functional module is also not entirely the same, further amplifying data discrepancies between functional modules. Therefore, this invention first categorizes the data according to functional modules, and then performs consistency analysis on the data from different functional modules in terms of version, content, and format to identify functional modules with significant data discrepancies. To focus limited computing resources on functional modules with genuine consistency issues, this step performs big data analysis on the acquired data from each functional module, quantitatively assessing the consistency of the data across the software's functional modules. This identifies functional modules with data discrepancies requiring intervention, and subsequent steps convert the format of the data in these modules to ensure that the data from these modules is unaffected by software version differences.

[0024] For any functional module, the specific methods for performing consistency analysis include: 2.1) For any functional module, parse the JSON structure of each piece of data, determine the data content of each piece of data, and then calculate the total number of fields in each piece of data. field name collection Field type collection .in They are the first, the second, ..., up to the 1st. The name of the last field. They are the first, the second, ..., up to the 1st. The last field has six data types: string: used to represent text data (e.g., user name, order number); int: used to represent integer values ​​without decimals (e.g., age, quantity); float: used to represent floating-point numbers with decimals (e.g., amount, latitude and longitude); bool: used to represent true / false logical values ​​(e.g., whether paid, whether enabled); object: used to represent nested key-value pair structures (e.g., a detailed dictionary of user address information); array: used to represent an ordered collection of data (e.g., a list of multiple purchased items).

[0025] 2.2) Based on the set of field names, field types, software version label, and timestamp label for each data entry, construct attribute groups for each data entry. Specific methods include: The software version labels are normalized to obtain the software version vector: In the formula, For the first The software version vector of each data point. For the first The numerical encoding result of the major version number of the data item. The numerical encoding result of the preset maximum major version number is, for example, 9. For the first The numerical encoding result of the minor version number of the data. The numerical encoding result is the preset upper limit of the minor version number, for example, 9. For the first The numerical encoding result of the revision number of the data item. The numerical encoding result is the preset revision number upper limit, for example, 9. In this embodiment, the numerical encoding method adopts weighted sum encoding. Version numbers with only numbers can be directly encoded; for version numbers containing letters or special symbols, the letters or special symbols need to be converted into numbers before encoding. Converting letters or special symbols includes establishing a mapping table of letters or special symbols, for example, A=1, B=2, C=3, ... or It converts letters or special symbols into numbers by constructing a mapping table.

[0026] Based on the timestamp label and the current time, construct time attribute features: In the formula, For the first The time attribute characteristics of each data point For the current time, For the first The timestamp label of each data item. This indicates normalization processing. In this embodiment, maximum and minimum value normalization processing is used. The maximum and minimum values ​​in the normalization process are taken from all data of the same functional module. The maximum and minimum values.

[0027] Based on the set of field names, a 64-bit binary fingerprint is generated using the SimHash algorithm, which in turn generates a fingerprint vector of a set dimension, resulting in a field name distribution vector. In this embodiment, the dimension is set to 8 as an example. The 64-bit binary fingerprint is divided into eight 8-bit binary codes, which are then converted to decimal and normalized for maximum and minimum values ​​to obtain an 8-dimensional fingerprint vector. During the normalization process, the maximum value is 255, the maximum value of the 8-bit binary code after conversion to decimal, and the minimum value is 0, the minimum value of the 8-dimensional binary code after conversion to decimal. The 8-dimensional fingerprint vector is as follows: In the formula, For the first Field name distribution vector of the data. For the first An 8-dimensional fingerprint vector of the set of field names of a data item.

[0028] Construct a field type distribution vector based on the field type set: In the formula, For the first Field type distribution vector of the data. For the first The number of fields of type string in a data entry. For the first The number of fields with type int in the data. For the first The number of fields with a float type in the data. For the first The number of fields with a boolean type in the data. For the first The number of fields of type object in a data record. For the first The number of fields of type array in a data record. For the first The total number of fields in a data entry.

[0029] Based on the software version vector, time attribute features, field name distribution vector, and field type distribution vector of each data entry, construct attribute groups for each data entry. .

[0030] 2.3) Attribute groups based on each data entry A consistency analysis is performed on the functional module to determine whether it is a data discrepancy module.

[0031] In this embodiment, the consistency analysis method includes: attribute groups based on each data entry. The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) density analysis is used to perform density clustering on each data corresponding to the same functional module. During the density analysis, the distance is Euclidean distance, the neighborhood radius is a preset neighborhood radius, such as 0.5, and the minimum number of nodes is determined based on a set proportion of the total number of data. In this embodiment, the set proportion is, for example, 10%.

[0032] After cluster analysis, multiple clusters are obtained. The ratio of the number of data entries in the cluster with the most data entries to the total number of data entries in the entire functional module is used as the consistency coefficient of that functional module. If the consistency coefficient of a functional module is less than a set consistency coefficient threshold, then the functional module is determined to be a data discrepancy functional module.

[0033] The above steps are performed on each functional module to filter out the data difference functional modules from all functional modules.

[0034] In this embodiment, the consistency coefficient is determined based on the first set quantile among the consistency coefficients of all functional modules. The first set quantile should be less than or equal to the 0.5 quantile (median). In this embodiment, the quantile is set to 0.4 quantile as an example.

[0035] Step 3: Based on the data difference functional modules determined in Step 2, construct training datasets for each data difference functional module, and then use the training datasets of each data difference functional module to train the version conversion functional module respectively, thereby obtaining the version mapping knowledge base of each data difference functional module.

[0036] The data difference module obtained in Step Two has a significant data format difference problem, which is caused by different software client versions used by different users. To eliminate this difference, this step uses reinforcement learning to incrementally learn the data format differences in the target functional module caused by version differences, thereby understanding the mapping relationship between data formats under different versions. This allows for format processing of historical version data in the target functional module through reinforcement learning. Specific steps include: 3.1) Based on the data of each data difference function module, construct the mapping data group of each data difference function module.

[0037] Based on the software version tags of each data entry in the data difference function module, the latest software version is determined, and the users of each data entry with the latest software version are marked as target users. Thus, the target user set is obtained. .

[0038] For the target user set Each target user in It retrieves all data in the data difference function module and divides it into the latest version data and historical version data of the target user according to the software version tag.

[0039] For a latest version of data (Corresponding timestamp label is) The software version is tagged as The data content is From the target user's historical version data, filter out historical version data with the smallest time difference from the latest version data and a time difference less than the set time difference limit. (timestamp label is) The software version is tagged as The data content is ).

[0040] The formula for determining if the time difference is less than the set upper limit of the time difference is as follows: In the formula To set a maximum time difference, such as 30 days.

[0041] Based on the latest version data Version tags Data content and this historical version data Version tags Data content Build from historical version data Up to the latest version of data Mapping data group .

[0042] The latest version data and historical version data with the smallest time difference may be data from the same user within a short period of time and may have similarities. For example, the status information of the same user does not change before and after the update, only the data structure changes. Therefore, this invention selects the latest version data and historical version data with the smallest time difference and the time difference is less than the set time difference upper limit to construct a mapping data group, which can focus on the changes in data format before and after the software version update.

[0043] For the same target user, there may be multiple records of the latest version data. Each record defines a mapping data group, and there may also be multiple corresponding mapping data groups.

[0044] For all target users in the target user set The historical version data of all versions were obtained using the steps described above. Up to the latest version of data Mapping data group This yields the mapping data group for the data difference function module.

[0045] For each data difference function module, the mapping data group of that data difference function module is obtained through the above steps.

[0046] 3.2) Filter the mapping data groups of each data difference function module to obtain the effective mapping data groups of each data difference function module.

[0047] Since the latest version data and historical version data with the smallest time difference and a time difference less than the set time difference upper limit may also be due to changes in data content rather than changes in data format, this invention filters out effective mapping data groups with similar data content by calculating the validity score of each mapping data group.

[0048] For any data difference function module, the effective methods for filtering the mapped data groups include: Calculate the validity score for each mapped data group, and then define all mapped data groups whose validity scores for the data difference function module are greater than a set validity score threshold as valid mapped data groups for the data difference function module. In this embodiment, the method for calculating the validity score of each mapped data group includes: In the formula, For mapping data groups The validity score, For mapping data groups The time difference between the latest version data and historical version data. The time decay coefficient is set as the upper limit of the time difference in this embodiment. Half of that (i.e., 15 days). For mapping data groups The version difference between the latest version data and historical version data For mapping data groups The semantic similarity between the latest version data and historical version data is calculated in this embodiment using keywords in the data content (such as user ID, order number, etc.). For mapping data groups In this embodiment, the structural similarity between the latest version data and historical version data is calculated using the Jaccard similarity of the set of field names in the data content. , , and These are the first, second, third, and fourth weighting coefficients, respectively. ,and In this embodiment, .

[0049] For mapping data groups The main version number of the latest version of the data. For mapping data groups The minor version number of the latest version of the data. For mapping data groups The revision number of the latest version of the data; For mapping data groups The major version number in the historical version data. For mapping data groups The minor version number in the historical version data. For mapping data groups Revision number in historical version data; , and These are the fifth, sixth, and seventh weighting coefficients, representing the weights of the major version number, minor version number, and revision number, respectively. ,and In this embodiment, , =0.3, =0.1.

[0050] It is the inverse mapping result of the ratio of time difference to time decay coefficient, used to represent the effectiveness score of time difference. The smaller the time difference, the larger the effectiveness score of time difference. Its value range is (0,1]. ,therefore The value range is (0,1]. The validity score is used to represent the version difference; the smaller the version difference, the higher the validity score.

[0051] In this embodiment, the validity score threshold is set as the second set quantile of the validity score of all mapped data groups of the data difference function module. The second set quantile is greater than or equal to 0.5 quantile. In this embodiment, the second set quantile is 0.7 quantile.

[0052] This invention uses the constraint of the same user and the minimum time difference when constructing the mapping data group to ensure that the latest version data and historical version data in a mapping data group are consistent as much as possible. Furthermore, by calculating the validity score of the mapping data group, it can filter out historical version data and latest version data with small differences in data content as valid mapping data groups and construct training datasets. This allows the differences in users and data content to be ignored during training, focusing on the format differences between different versions and improving the accuracy of format conversion.

[0053] 3.3) Based on all valid mapping data groups of each data difference function module, construct training datasets for each data difference function module, and use the training datasets of each data difference function module to train the version conversion function module respectively, so as to obtain the trained version conversion function module and version mapping knowledge base of each data difference function module.

[0054] For a certain data difference function module, each effective mapping data set is a set of training data for that data difference function module. The historical version data of the effective mapping data set is the state input sample for reinforcement learning, and the corresponding latest version data is the reference benchmark for calculating the reward for that sample, thus constructing the training dataset of the data difference function module.

[0055] In this embodiment, the version conversion function module is a reinforcement learning neural network with an Actor-Critic architecture. The input of the reinforcement learning neural network is the historical version data of the data difference function module, and the output is the optimal format conversion instruction sequence for the corresponding historical version. A version mapping knowledge base for the data difference function module is constructed based on the output of the reinforcement learning neural network. Specific methods include: Construct the state space, action space, and reward function of a reinforcement learning neural network.

[0056] The state space is a space composed of feature vectors formed by historical version data. The feature vectors are obtained by the feature extraction network of the reinforcement learning neural network converting the historical version data into vector format.

[0057] The action space includes actions that convert the format of historical version data. The actions in the action space are sequences of operation instructions output by the reinforcement learning neural network. By executing these sequences, historical version data can be converted from its historical format to the latest version format. The operation instruction sequence includes the operation instruction type and operation parameters. In this embodiment, the instruction types include: field renaming, type conversion, field splitting, merging, filling, deletion, and keeping as is. The operation parameters of the operation instruction sequence specify the exact location or target of the operation.

[0058] The reward function is based on the similarity between historical versions of data after format conversion and the latest versions of data in the same valid mapped data set. Specific calculation methods include: In the formula, This measures the similarity between historical version data after format conversion and the latest version data of the same valid mapped data group. For mapping data groups The semantic similarity between the latest version data and historical version data is calculated in this embodiment using keywords in the data content (such as user ID, order number, etc.). For mapping data groups In this embodiment, the structural similarity between the latest version data and historical version data is calculated using the Jaccard similarity of the set of field names in the data content. and These are the weighting coefficients for semantic similarity and structural similarity, respectively. In this embodiment .

[0059] In this embodiment, the Actor-Critic architecture reinforcement learning neural network includes a feature extraction network, an Actor network, and a Critic network. The feature extraction network encodes the state of each data point (including the latest version data and historical version data) input to the reinforcement learning neural network, obtaining feature vectors for each data point in the training dataset. Each feature vector is composed of three parts: a software version vector, a field type distribution vector, and a word sequence feature vector. The software version vector and field type distribution vector have already been constructed in step 2.2). The method for constructing the word sequence feature vector includes: For the data content of each data item or The algorithm parses JSON field names, flattens the resulting token sequence, and converts it into a fixed-dimensional token sequence feature vector. The specific methods include: recursively traversing all keys in the JSON data, converting nested structures into path strings; further splitting each path string into tokens; concatenating all tokens in hierarchical order into a one-dimensional sequence; truncating or padding the sequence to achieve a fixed length, and then converting it into a fixed-dimensional token sequence feature vector.

[0060] Specifically, the training method for reinforcement learning neural networks is existing technology and will not be described in detail in this embodiment. The iterative training process of reinforcement learning neural networks is to continuously select actions from the action space and execute these actions so that the historical version data in the effective mapping data set is transformed to approach the latest version data.

[0061] After the reinforcement learning neural network of the data difference function module is trained, the parameters of the reinforcement learning neural network are retained, resulting in the reinforcement learning neural network (version conversion function module) of the data difference function module. Simultaneously, the actions selected in the action space during each iteration of the training process are recorded, forming the version mapping knowledge base of the data difference function module. For each valid mapped data group in the training dataset, the optimal action sequence after training convergence or when the reward function value reaches a set threshold is output, and the version of the historical version data of the valid mapped data group is marked (historical version), forming the optimal format conversion instruction sequence for that historical version. The data is stored in the version mapping knowledge base of the data difference function module. These are the actions for step 1, step 2, ..., step K, respectively, and each action belongs to the action space of the reinforcement learning neural network.

[0062] Step 4: Using the version conversion function module and version mapping knowledge base completed by training each data difference function module, convert all historical version data of each data difference function module to the latest version format.

[0063] In this embodiment, in order to improve the speed and efficiency of format conversion of historical version data, the historical version data is split into multiple streams. The version mapping knowledge base is used first for format conversion, and the trained version conversion function module is used for conversion if the version mapping knowledge base cannot be used.

[0064] For any data difference function module, the specific methods include: 4.1) Based on the software version tags of all data in the data difference function module, filter out all historical version data (the latest version data does not need to be processed).

[0065] 4.2) For any historical version data, determine its software version (historical version), and determine whether there is an optimal format conversion instruction sequence for the historical version in the version mapping knowledge base of the data difference function module. If it exists, use the optimal format conversion instruction sequence to convert the format of the historical version data. If it does not exist, use the version conversion function module trained by the data difference function module to generate the optimal format conversion instruction sequence for the historical version, and then convert the format of the historical version data.

[0066] Since the data in different functional modules have significant differences in content and structure, in order to adapt to these differences, this invention constructs a separate training dataset for each functional module and trains a separate reinforcement learning neural network to obtain a reinforcement learning neural network dedicated to each functional module. This improves the optimization effect on the historical version data of each functional module and thus improves data consistency.

Claims

1. A big data optimization method for improving data quality, characterized in that, Includes the following steps: 1) Obtain all data generated during the operation of the software client by different users, wherein the software client includes multiple functional modules; 2) Determine the data generated by each functional module in the full dataset, and perform data consistency analysis on each functional module to identify the functional modules with data discrepancies; Specific methods include: Parse the data of any functional module to determine the attribute groups of each data item in that functional module; Based on the attribute groups of each data point, cluster analysis is performed on all data of this functional module to obtain multiple clusters; The consistency coefficient of a functional module is obtained by calculating the ratio of the number of data entries in the cluster with the most data entries to the total number of data entries in that functional module. Based on the consistency coefficient of each functional module, identify the functional modules with data differences. 3) Based on the data of each data difference function module, construct the training dataset of each data difference function module, train the version conversion function module respectively, and obtain the trained version conversion function module and version mapping knowledge base of each data difference function module. The version mapping knowledge base of each data difference function module includes the optimal format conversion instruction sequence of historical versions. The specific methods for constructing the training datasets for each data difference functional module include: Determine the latest version of the software version among all data entries in any data difference function module, and then mark the user with the latest version of the data as the target user; For each latest version data of each target user, identify the historical version data in the data difference function module that is the same user as the latest version data, has the smallest time difference and the time difference is less than the set time difference limit, and then use the latest version data and the historical version data to construct a mapping data group; Calculate the validity score of each mapped data group and select the valid mapped data groups as training samples to construct the training dataset for the data difference function module. 4) Using the version conversion function module and version mapping knowledge base trained by each data difference function module, the historical version data of each data difference function module is converted into the latest version format of the historical version data.

2. The big data optimization method for improving data quality according to claim 1, characterized in that, The attribute group includes a software version vector, a time attribute feature, a field type distribution vector, and a field name distribution vector.

3. The big data optimization method for improving data quality according to claim 2, characterized in that, Methods for calculating the validity score of each mapped data group include: The validity score of any mapped data group is obtained by fusing the validity score of the time difference between the latest version data and the historical version data, the validity score of the version difference, the semantic similarity of the data content, and the structural similarity of the data content.

4. The big data optimization method for improving data quality according to claim 1, characterized in that, The methods for format conversion of any historical version data from each data difference functional module, using the version conversion functional module trained by the version mapping knowledge base, include: For any historical version data in any data difference function module, determine the historical version of the historical version data, and determine whether there is an optimal format conversion instruction sequence for the historical version in the version mapping knowledge base of the data difference function module. If it exists, use the optimal format conversion instruction sequence to convert the format of the historical version data. If it does not exist, use the version conversion function module trained by the data difference function module to generate the optimal format conversion instruction sequence for the historical version, and then convert the format of the historical version data.

5. The big data optimization method for improving data quality according to claim 2, characterized in that, Methods for obtaining the version mapping knowledge base for each data difference functional module include: For any mapping data group in the training dataset of any data difference function module, the optimal format conversion instruction sequence obtained after training to convert historical version data to the latest version is used as the optimal format conversion instruction sequence for that historical version, thereby obtaining the version mapping knowledge base of that data difference function module.

6. The big data optimization method for improving data quality according to claim 1, characterized in that, The version conversion function module is a reinforcement learning neural network with an Actor-Critic architecture.

7. The big data optimization method for improving data quality according to claim 6, characterized in that, The reward function of the reinforcement learning neural network in the Actor-Critic architecture is the fusion result of the semantic similarity and structural similarity between the data content of historical versions of data and the data content of the latest version of data in any training sample.

8. A big data optimization system for improving data quality, comprising a server, characterized in that, The server includes a data acquisition module and a processing module. The data acquisition module is used to collect all the data generated by the user during the use of the software client and send it to the processing module. The processing module implements the big data optimization method for improving data quality as described in any one of claims 1-7 based on the received full data.

Images