Data aggregation method, device, equipment and computer readable medium

Abstract

The present application relates to a data aggregation method, device, equipment and computer-readable medium. The method includes: obtaining raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node; inputting the raw data into a preset data cleaning model to clean the raw data into metadata, wherein the metadata is used to represent the attribute characteristics of the raw data; a data deconstruction module, which is used to input the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data; using a preset dictionary model to classify and store the structured data to aggregate the data. The present application solves the technical problem that it is difficult to construct basic associations for data from multiple channels.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data aggregation method, device, equipment and computer-readable medium. Background Art

[0002] With the development of science and technology and the progress of society, whether it is a traditional enterprise or an Internet enterprise, the means of production are no longer limited to physical objects, but also include data. With the advent of the DT (Data Technology, data processing technology) era, the sources of data channels are becoming more and more diverse. Basic product data, product marketing data, consumer data, after-sales experience data..., while the amount of various types of data is growing exponentially, it is also facing the situation of "data islands". Data is an indispensable part of the means of production. Only by adding all the data together in production can the maximum value of data be brought into play, which poses a great challenge to the aggregation of multi-channel and massive data.

[0003] At present, in the relevant technologies, due to the huge differences between different fields and different businesses, the data forms from different channels are different. It is difficult to overcome the difference gap to further mine the data in depth on the basis of meeting the needs of breadth aggregation. As a result, the difference gap cannot be broken after aggregation, and the data between channels cannot establish basic connections, which means that the data between channels are still isolated and the maximum value of the data cannot be brought into play.

[0004] There is currently no effective solution to the problem that it is difficult to build basic associations for data from multiple sources. Summary of the invention

[0005] The present application provides a data aggregation method, apparatus, device and computer-readable medium to solve the technical problem that it is difficult to construct basic associations with data from multiple channels.

[0006] According to one aspect of an embodiment of the present application, the present application provides a data aggregation method, including:

[0007] Acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node;

[0008] Inputting the original data into a preset data cleaning model to clean the original data into metadata, wherein the metadata is used to represent attribute characteristics of the original data;

[0009] Inputting the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data;

[0010] The structured data is classified and stored using a preset dictionary model to aggregate the data.

[0011] Optionally, obtaining the original data from at least one channel includes obtaining the original data from one of the service nodes in the following manner:

[0012] Link to the target service interface and / or target data query interface, wherein the target service interface is the actual production line interface called when the target service node provides the corresponding business service, and the target data query interface is the data query interface provided by the target service node;

[0013] The original data is intercepted from at least one of the target service interface and the target data interface according to a preset data range, wherein the preset data range is the collection range of the original data in the target service node determined according to the configuration parameters of the target business model, and the target business model is the downstream application model after data aggregation.

[0014] Optionally, the method further includes obtaining the data cleaning model in the following manner:

[0015] Acquire an initial training set, wherein the training data in the initial training set all carry labeling information, and the labeling information is used to label the cleaning data corresponding to the training data;

[0016] Iteratively train the initial model using the initial training set, until the average value of the overlap rate between the first predicted cleaning result output by the initial model and the cleaning data in the annotation information is greater than or equal to a first threshold, and then stop the iteration to obtain an intermediate model, wherein the data size of the training set used in each iteration is gradually reduced during the iteration process, the initial model is a deep neural network model, the first predicted cleaning result includes the predicted cleaning results of each of the training data, and the average value of the overlap rate is used to represent the prediction accuracy of the intermediate model;

[0017] Re-inputting the initial training set into the intermediate model, and obtaining a second prediction cleaning result of the intermediate model on the initial training set;

[0018] When it is determined based on the second prediction cleaning result that the prediction accuracy of the intermediate model is greater than or equal to the first threshold, the intermediate model is determined as the data cleaning model.

[0019] Optionally, one of the iterative processes of iteratively training the initial model using the initial training set includes:

[0020] Obtaining a first intermediate training set used in a previous iteration, and obtaining the first prediction cleaning result output by the initial model for predicting the first intermediate training set in the previous iteration;

[0021] Determine an average value of the difference rate between the first predicted cleaning result and the cleaning data in the annotation information;

[0022] Determine the loss value of the objective function of the initial model using the average value of the difference rate;

[0023] Adjusting the model parameters of the initial model according to the loss value;

[0024] The model parameters are used as new initial parameters of the initial model, and the data size of the first intermediate training set is reduced to obtain a second intermediate training set, wherein the initial parameters used when the initial model is first trained are preset initial parameters;

[0025] The second intermediate training set is input into the initial model to perform this iterative training.

[0026] Optionally, reducing the data size of the first intermediate training set to obtain the second intermediate training set includes:

[0027] sorting the training data in the first intermediate training set according to the difference rate between the first predicted cleaning result and the cleaning data in the labeling information;

[0028] The training data located before the target sorting position is extracted from the sorting result to form the second intermediate training set.

[0029] Optionally, after classifying and storing the structured data using a preset dictionary model, the method further includes:

[0030] When receiving a data retrieval request from a target object for a target classification, extracting a first key carried in the data retrieval request;

[0031] In case the first key is paired with the second key of the target classification, a data access interface of the target classification is sent to the target object.

[0032] Optionally, after the structured data is classified and stored using a preset dictionary model, the method further includes:

[0033] Acquire usage frequency and / or feature extraction frequency of the structured data by the target business model;

[0034] Determining a first parameter adjustment amount for the data cleaning model, a second parameter adjustment amount for the deconstruction model, and a third parameter adjustment amount for the dictionary model based on the usage frequency and / or the feature extraction frequency;

[0035] Iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the data with a usage frequency greater than or equal to the second threshold is greater than or equal to a third threshold, and / or, iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the features with a feature extraction frequency greater than or equal to a fourth threshold is greater than or equal to a fifth threshold, wherein the model parameters of the data cleaning model are adjusted with the first parameter adjustment amount as an adjustment step in each iteration;

[0036] Iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the data with a usage frequency greater than or equal to a sixth threshold is greater than or equal to a seventh threshold, and / or, iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the features with a feature extraction frequency greater than or equal to an eighth threshold is greater than or equal to a ninth threshold, wherein the model parameters of the deconstruction model are adjusted using the second parameter adjustment amount as an adjustment step in each iteration;

[0037] The dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the data with a usage frequency greater than or equal to the tenth threshold is greater than or equal to the eleventh threshold, and / or, the dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the features with a feature extraction frequency greater than or equal to the twelfth threshold is greater than or equal to the thirteenth threshold, wherein the model parameters of the dictionary model are adjusted using the third parameter adjustment amount as an adjustment step in each iteration.

[0038] According to another aspect of an embodiment of the present application, the present application provides a data aggregation device, including:

[0039] A data acquisition module, used to acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node;

[0040] A data cleaning module, used for inputting the raw data into a preset data cleaning model to clean the raw data into metadata, wherein the metadata is used to represent the attribute characteristics of the raw data;

[0041] A data deconstruction module, used for inputting the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data;

[0042] The classification storage module is used to classify and store the structured data using a preset dictionary model to aggregate the data.

[0043] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0044] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0045] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0046] The technical solution of the present application is to obtain raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is the production data on the service node; input the raw data into a preset data cleaning model to clean the raw data into metadata, wherein the metadata is used to represent the attribute characteristics of the raw data; a data deconstruction module is used to input the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data; and a preset dictionary model is used to classify and store the structured data to aggregate the data. The present application uses a data cleaning model, a deconstruction model, and a dictionary model to clean, structure and intelligently classify data from different channels, build basic associations for data from multiple channels, and make the data resonate synergistically to form data assets, thereby maximizing the value of the data and solving the technical problem that it is difficult to build basic associations for data from multiple channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of a hardware environment for an optional data aggregation method provided according to an embodiment of the present application;

[0050] Figure 2 A flowchart of an optional data aggregation method provided according to an embodiment of the present application;

[0051] Figure 3 A block diagram of an optional data aggregation device provided according to an embodiment of the present application;

[0052] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0055] In related technologies, due to the huge differences between different fields and different businesses, the data forms from different channel sources are different. It is difficult to overcome the difference gap to further mine the data in depth on the basis of meeting the needs of breadth aggregation. As a result, the difference gap cannot be broken after aggregation, and the data between channels cannot establish basic connections, which means that the data between channels are still isolated and the maximum value of the data cannot be realized.

[0056] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a data aggregation method is provided.

[0057] Optionally, in the embodiment of the present application, the above data aggregation method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services (such as data collection, data cleaning, data deconstruction, data classification, and data storage, etc.) for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0058] A data aggregation method in the embodiment of the present application can be executed by the server 103, or can be executed by the server 103 and the terminal 101 together, such as Figure 2 As shown, the method may include the following steps:

[0059] Step S202, obtaining original data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the original data is production data on the service node.

[0060] In the embodiments of the present application, the data sources can be different businesses and different fields. Each business consists of a business chain composed of multiple service nodes. For example, the transaction business involves order placement, commodity transfer, logistics distribution, and user evaluation. Each link is a service node, and multiple service nodes form a business chain. The original data from the same service node may also be different, such as commodity basic data, marketing data, inventory data, user purchasing power data, etc. The original data includes structured data and unstructured data such as pictures, documents, files, tables, videos, and audio.

[0061] Step S204: inputting the original data into a preset data cleaning model to clean the original data into metadata, wherein the metadata is used to represent the attribute characteristics of the original data.

[0062] In the embodiment of the present application, the amount of raw data is huge, the types are diverse, and there is "dirty" data. The raw data is cleaned through the data cleaning model to filter out missing, erroneous, duplicated, abnormal and other dirty data, and retain valid data, i.e. metadata. Data cleaning can effectively reduce the space occupied by the amount of data.

[0063] Step S206: input the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data.

[0064] In the embodiment of the present application, the received metadata contains data in various formats, including many unstructured data such as pictures, documents, files, tables, videos, audio, etc. It is very important to structure the unstructured data. The present application deconstructs the unstructured data through the deconstruction model, so that all the data is deconstructed into compilable, storable and readable data, and finally obtains structured data. At this point, the original data from all channels are converted into machine-compilable and identifiable structured data, so that the data from different channels have the basis for establishing associations.

[0065] Step S208, classifying and storing the structured data using a preset dictionary model to aggregate the data.

[0066] In the embodiment of the present application, data features can be extracted from structured data through a dictionary model, and the structured data can be stored and classified based on the data features, thereby constructing a basic association of data from multiple sources. This not only achieves data aggregation and enables synergistic resonance between data to maximize the value of data, but also makes the correlation between multi-source data stronger, thereby improving the efficiency of data retrieval and viewing.

[0067] By adopting the technical solution of the present application, data from different sources are cleaned, structured and intelligently classified through data cleaning models, deconstruction models, and dictionary models, and basic associations of data from multiple channels are constructed, so that data can be coordinated and resonated to form data assets, thereby maximizing the value of data and solving the technical problem that it is difficult to construct basic associations for data from multiple channels.

[0068] Optionally, obtaining the original data from at least one channel includes obtaining the original data from one of the service nodes in the following manner:

[0069] Step 1, link to the target service interface and / or target data query interface, wherein the target service interface is the actual production line interface called when the target service node provides corresponding business services, and the target data query interface is the data query interface provided by the target service node.

[0070] In an embodiment of the present application, when acquiring data, it is necessary to link to the target service interface called by the target service node, or to link to the data query interface provided by the target service node. The target service interface is an interface on the actual production line, which can intercept the actual original data generated in real time. The data query interface is provided by the target service node, and the original data intercepted through this interface is more targeted.

[0071] Step 2: intercept original data from at least one of the target service interface and the target data interface according to a preset data range, wherein the preset data range is the collection range of original data in the target service node determined according to the configuration parameters of the target business model, and the target business model is the downstream application model after data aggregation.

[0072] In the embodiment of the present application, the purpose of data aggregation is to provide a data basis for the downstream application model. The downstream application model corresponds to the actual problem to be solved, or corresponds to the business needs of the downstream application. The downstream application model is configured according to the business needs or the actual problems to be solved, and the data range of the data to be collected during data aggregation is determined by the configuration parameters of the downstream application model.

[0073] The present application also provides a training method for the above-mentioned data cleaning model, which is described below.

[0074] Optionally, the method further includes obtaining the data cleaning model in the following manner:

[0075] Step 1: Obtain an initial training set, wherein the training data in the initial training set all carry annotation information, and the annotation information is used to annotate the cleaning data corresponding to the training data;

[0076] Step 2, iteratively training the initial model using the initial training set, until the average value of the overlap rate between the first predicted cleaning result output by the initial model and the cleaning data in the annotation information is greater than or equal to the first threshold, and then the iteration is stopped to obtain an intermediate model, wherein the data size of the training set used in each iteration is gradually reduced during the iteration process, the initial model is a deep neural network model, the first predicted cleaning result includes the predicted cleaning results of each training data, and the average value of the overlap rate is used to represent the prediction accuracy of the intermediate model;

[0077] Step 3, re-inputting the initial training set into the intermediate model, and obtaining a second prediction cleaning result of the intermediate model on the initial training set;

[0078] Step 4: When it is determined based on the second prediction cleaning result that the prediction accuracy of the intermediate model is greater than or equal to the first threshold, the intermediate model is determined as the data cleaning model.

[0079] In the embodiment of the present application, the data scale of the initial training set is the largest, and the data scale of the training set gradually decreases during the iteration process. This is because after each iterative training, it is necessary to eliminate training data whose prediction results are too far away from the actual annotations. In this way, in the process of gradually reducing the training data, the prediction accuracy of the model is also gradually improved.

[0080] In the embodiment of the present application, after the iterative training is completed, the trained intermediate model needs to be applied to the full amount of data in the initial training set for verification. Only when the same prediction accuracy is achieved, the intermediate model is determined as a data cleaning model that can be put into use.

[0081] Optionally, one of the iterative processes of iteratively training the initial model using the initial training set includes:

[0082] Step 1, obtaining a first intermediate training set used in a previous iteration, and obtaining a first prediction cleaning result of the initial model predicting the first intermediate training set in the previous iteration;

[0083] Step 2, determining an average value of the difference rate between the first predicted cleaning result and the cleaning data in the annotation information;

[0084] Step 3, using the average value of the difference rate to determine the loss value of the objective function of the initial model;

[0085] Step 4, adjusting the model parameters of the initial model according to the loss value;

[0086] Step 5, using the model parameters as new initial parameters of the initial model, and reducing the data size of the first intermediate training set to obtain a second intermediate training set, wherein the initial parameters used when the initial model is first trained are the preset initial parameters;

[0087] Step 6: Input the second intermediate training set into the initial model for this iterative training.

[0088] Optionally, reducing the data size of the first intermediate training set to obtain the second intermediate training set includes:

[0089] Step 1, sorting the training data in the first intermediate training set according to the difference rate between the first predicted cleaning result and the cleaning data in the annotation information;

[0090] Step 2: extract the training data before the target sorting position from the sorting results to form a second intermediate training set.

[0091] In the embodiment of the present application, before each iteration, it is necessary to remove the training data whose prediction results are too far away from the actual annotations from the training set of the previous iteration, so as to gradually improve the prediction accuracy of the model.

[0092] Since this application breaks the data gap between different channels, it is more convenient and faster to retrieve specific categories of data in a targeted manner, so data security needs to be guaranteed.

[0093] Optionally, after the structured data is classified and stored using a preset dictionary model, the method further includes:

[0094] When receiving a data retrieval request from a target object for a target classification, extracting a first key carried in the data retrieval request;

[0095] When the first key is paired with the second key of the target classification, a data access interface of the target classification is sent to the target object.

[0096] In an embodiment of the present application, data security is ensured by pairing data keys. Pairing keys are generated in advance, one of which is associated with a specific data category, and the other is assigned to a user who needs to access the data under that category. Each time the user retrieves the data under that category, he needs to send a data retrieval request carrying the key. The system will only open the data access interface of the category to the user when the key pairing is successful, thereby ensuring data security.

[0097] In the embodiments of the present application, not only can data security be guaranteed from the data category level, but also data can be classified more finely, and users need to reach the corresponding authority level to access the corresponding data.

[0098] The data cleaning model, deconstruction model and dictionary model provided in this application can analyze the data flow behavior in the actual application process, further self-iterate, and ultimately improve the aggregation efficiency and accuracy of data from multiple channels.

[0099] Optionally, after the structured data is classified and stored using a preset dictionary model, the method further includes:

[0100] Obtaining the usage frequency and / or feature extraction frequency of the target business model for structured data;

[0101] determining a first parameter adjustment amount for the data cleaning model, a second parameter adjustment amount for the deconstruction model, and a third parameter adjustment amount for the dictionary model based on the usage frequency and / or the feature extraction frequency;

[0102] Iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for data with a usage frequency greater than or equal to the second threshold is greater than or equal to a third threshold, and / or, iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for features with a feature extraction frequency greater than or equal to a fourth threshold is greater than or equal to a fifth threshold, wherein each iteration adjusts the model parameters of the data cleaning model using the first parameter adjustment amount as an adjustment step;

[0103] Iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for data with a usage frequency greater than or equal to a sixth threshold is greater than or equal to a seventh threshold, and / or, iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for features with a feature extraction frequency greater than or equal to an eighth threshold is greater than or equal to a ninth threshold, wherein each iteration adjusts the model parameters of the deconstruction model using the second parameter adjustment amount as an adjustment step;

[0104] The dictionary model is iteratively trained based on the third parameter adjustment amount until the dictionary model's recognition accuracy for data with a usage frequency greater than or equal to a tenth threshold is greater than or equal to an eleventh threshold, and / or, the dictionary model is iteratively trained based on the third parameter adjustment amount until the dictionary model's recognition accuracy for features with a feature extraction frequency greater than or equal to a twelfth threshold is greater than or equal to a thirteenth threshold, wherein each iteration adjusts the model parameters of the dictionary model using the third parameter adjustment amount as an adjustment step.

[0105] In the embodiment of the present application, the first threshold to the thirteenth threshold mentioned above can be set according to actual needs.

[0106] According to another aspect of the embodiment of the present application, Figure 3 As shown, a data aggregation device is provided, comprising:

[0107] The data acquisition module 301 is used to acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node;

[0108] The data cleaning module 303 is used to input the original data into a preset data cleaning model to clean the original data into metadata, wherein the metadata is used to represent the attribute characteristics of the original data;

[0109] The data deconstruction module 305 is used to input the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data;

[0110] The classification storage module 307 is used to classify and store the structured data using a preset dictionary model to aggregate the data.

[0111] It should be noted that the data acquisition module 301 in this embodiment can be used to execute step S202 in the embodiment of the present application, the data cleaning module 303 in this embodiment can be used to execute step S204 in the embodiment of the present application, the data deconstruction module 305 in this embodiment can be used to execute step S206 in the embodiment of the present application, and the classification storage module 307 in this embodiment can be used to execute step S208 in the embodiment of the present application.

[0112] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0113] Optionally, the data acquisition module is specifically used to acquire the original data of one of the service nodes in the following manner:

[0114] Link to the target service interface and / or target data query interface, wherein the target service interface is the actual production line interface called when the target service node provides the corresponding business service, and the target data query interface is the data query interface provided by the target service node;

[0115] The original data is intercepted from at least one of the target service interface and the target data interface according to a preset data range, wherein the preset data range is the collection range of the original data in the target service node determined according to the configuration parameters of the target business model, and the target business model is the downstream application model after data aggregation.

[0116] Optionally, the data aggregation device further includes a data cleaning model training module, which is used to:

[0117] Acquire an initial training set, wherein the training data in the initial training set all carry labeling information, and the labeling information is used to label the cleaning data corresponding to the training data;

[0118] Iteratively train the initial model using the initial training set, until the average value of the overlap rate between the first predicted cleaning result output by the initial model and the cleaning data in the annotation information is greater than or equal to a first threshold, and then stop the iteration to obtain an intermediate model, wherein the data size of the training set used in each iteration is gradually reduced during the iteration process, the initial model is a deep neural network model, the first predicted cleaning result includes the predicted cleaning results of each of the training data, and the average value of the overlap rate is used to represent the prediction accuracy of the intermediate model;

[0119] Re-inputting the initial training set into the intermediate model, and obtaining a second prediction cleaning result of the intermediate model on the initial training set;

[0120] When it is determined based on the second prediction cleaning result that the prediction accuracy of the intermediate model is greater than or equal to the first threshold, the intermediate model is determined as the data cleaning model.

[0121] Optionally, the data cleaning model training module further includes an iterative training unit, which is specifically used to:

[0122] Obtaining a first intermediate training set used in a previous iteration, and obtaining the first prediction cleaning result output by the initial model for predicting the first intermediate training set in the previous iteration;

[0123] Determine an average value of the difference rate between the first predicted cleaning result and the cleaning data in the annotation information;

[0124] Determine the loss value of the objective function of the initial model using the average value of the difference rate;

[0125] Adjusting the model parameters of the initial model according to the loss value;

[0126] The model parameters are used as new initial parameters of the initial model, and the data size of the first intermediate training set is reduced to obtain a second intermediate training set, wherein the initial parameters used when the initial model is first trained are preset initial parameters;

[0127] The second intermediate training set is input into the initial model to perform this iterative training.

[0128] Optionally, the iterative training unit is further used to:

[0129] sorting the training data in the first intermediate training set according to the difference rate between the first predicted cleaning result and the cleaning data in the labeling information;

[0130] The training data located before the target sorting position is extracted from the sorting result to form the second intermediate training set.

[0131] Optionally, the data aggregation device further includes a data security control module, specifically configured to:

[0132] When receiving a data retrieval request from a target object for a target classification, extracting a first key carried in the data retrieval request;

[0133] In case the first key is paired with the second key of the target classification, a data access interface of the target classification is sent to the target object.

[0134] Optionally, the data aggregation device further includes a model optimization module, specifically configured to:

[0135] Acquire usage frequency and / or feature extraction frequency of the structured data by the target business model;

[0136] Determining a first parameter adjustment amount for the data cleaning model, a second parameter adjustment amount for the deconstruction model, and a third parameter adjustment amount for the dictionary model based on the usage frequency and / or the feature extraction frequency;

[0137] Iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the data with a usage frequency greater than or equal to the second threshold is greater than or equal to a third threshold, and / or, iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the features with a feature extraction frequency greater than or equal to a fourth threshold is greater than or equal to a fifth threshold, wherein the model parameters of the data cleaning model are adjusted with the first parameter adjustment amount as an adjustment step in each iteration;

[0138] Iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the data with a usage frequency greater than or equal to a sixth threshold is greater than or equal to a seventh threshold, and / or, iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the features with a feature extraction frequency greater than or equal to an eighth threshold is greater than or equal to a ninth threshold, wherein the model parameters of the deconstruction model are adjusted using the second parameter adjustment amount as an adjustment step in each iteration;

[0139] The dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the data with a usage frequency greater than or equal to the tenth threshold is greater than or equal to the eleventh threshold, and / or, the dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the features with a feature extraction frequency greater than or equal to the twelfth threshold is greater than or equal to the thirteenth threshold, wherein the model parameters of the dictionary model are adjusted using the third parameter adjustment amount as an adjustment step in each iteration.

[0140] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0141] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0142] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0143] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0144] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above embodiments.

[0145] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to perform the following steps:

[0146] Acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node;

[0147] Inputting the original data into a preset data cleaning model to clean the original data into metadata, wherein the metadata is used to represent the attribute characteristics of the original data;

[0148] Inputting the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data;

[0149] The structured data is classified and stored using a preset dictionary model to aggregate the data.

[0150] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0151] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0152] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0153] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0154] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0160] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A data aggregation method, characterized in that: include: Acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node; Inputting the original data into a preset data cleaning model to clean the original data into metadata, wherein the metadata is used to represent attribute characteristics of the original data; Inputting the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data; Classifying and storing the structured data using a preset dictionary model to aggregate the data; The method also includes obtaining the data cleaning model in the following manner: obtaining an initial training set, wherein the training data in the initial training set all carry annotation information, and the annotation information is used to annotate the cleaning data corresponding to the training data; iteratively training the initial model using the initial training set until the average value of the overlap rate between the first predicted cleaning result output by the initial model and the cleaning data in the annotation information is greater than or equal to a first threshold, and the iteration is stopped to obtain an intermediate model, wherein the data size of the training set used in each iteration gradually decreases during the iteration process, the initial model is a deep neural network model, the first predicted cleaning result includes the predicted cleaning results for each of the training data, and the average value of the overlap rate is used to represent the prediction accuracy of the intermediate model; re-inputting the initial training set into the intermediate model, and obtaining a second predicted cleaning result of the intermediate model for the initial training set; when it is determined based on the second predicted cleaning result that the prediction accuracy of the intermediate model is greater than or equal to the first threshold, the intermediate model is determined as the data cleaning model; One of the iterative processes of iteratively training the initial model using the initial training set includes: obtaining the first intermediate training set used in the previous iteration, and obtaining the first predicted cleaning result output by the initial model for the first intermediate training set in the previous iteration; determining the average value of the difference rate between the first predicted cleaning result and the cleaning data in the annotation information; determining the loss value of the objective function of the initial model using the average value of the difference rate; adjusting the model parameters of the initial model according to the loss value; using the model parameters as new initial parameters of the initial model, and reducing the data scale of the first intermediate training set to obtain a second intermediate training set, wherein the initial parameters used when the initial model is first trained are preset initial parameters; inputting the second intermediate training set into the initial model for this iterative training; The reducing the data size of the first intermediate training set to obtain the second intermediate training set comprises: sorting the training data in the first intermediate training set according to the difference rate between the first predicted cleaning result and the cleaning data in the annotation information; extracting the training data located before the target sorting position from the sorting result to form the second intermediate training set; After classifying and storing the structured data using a preset dictionary model, the method also includes: upon receiving a data retrieval request for a target classification from a target object, extracting a first key carried in the data retrieval request; and when the first key is paired with a second key of the target classification, sending the data access interface of the target classification to the target object.

2. The method according to claim 1, characterized in that Acquiring the original data from at least one channel includes acquiring the original data from one of the service nodes in the following manner: Link to the target service interface and / or target data query interface, wherein the target service interface is the actual production line interface called when the target service node provides the corresponding business service, and the target data query interface is the data query interface provided by the target service node; The original data is intercepted from at least one of the target service interface and the target data interface according to a preset data range, wherein the preset data range is the collection range of the original data in the target service node determined according to the configuration parameters of the target business model, and the target business model is the downstream application model after data aggregation.

3. The method according to claim 2, characterized in that After classifying and storing the structured data using a preset dictionary model, the method further includes: Acquire usage frequency and / or feature extraction frequency of the structured data by the target business model; Determining a first parameter adjustment amount for the data cleaning model, a second parameter adjustment amount for the deconstruction model, and a third parameter adjustment amount for the dictionary model based on the usage frequency and / or the feature extraction frequency; Iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the data with a usage frequency greater than or equal to the second threshold is greater than or equal to a third threshold, and / or, iteratively training the data cleaning model based on the first parameter adjustment amount until the recognition accuracy of the data cleaning model for the features with a feature extraction frequency greater than or equal to a fourth threshold is greater than or equal to a fifth threshold, wherein the model parameters of the data cleaning model are adjusted with the first parameter adjustment amount as an adjustment step in each iteration; Iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the data with a usage frequency greater than or equal to a sixth threshold is greater than or equal to a seventh threshold, and / or, iteratively training the deconstruction model based on the second parameter adjustment amount until the recognition accuracy of the deconstruction model for the features with a feature extraction frequency greater than or equal to an eighth threshold is greater than or equal to a ninth threshold, wherein the model parameters of the deconstruction model are adjusted using the second parameter adjustment amount as an adjustment step in each iteration; The dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the data with a usage frequency greater than or equal to the tenth threshold is greater than or equal to the eleventh threshold, and / or, the dictionary model is iteratively trained based on the third parameter adjustment amount until the recognition accuracy of the dictionary model for the features with a feature extraction frequency greater than or equal to the twelfth threshold is greater than or equal to the thirteenth threshold, wherein the model parameters of the dictionary model are adjusted using the third parameter adjustment amount as an adjustment step in each iteration.

4. A data aggregation device, characterized in that: include: A data acquisition module, used to acquire raw data from at least one channel, wherein each channel includes at least one service node on at least one business chain, and the raw data is production data on the service node; A data cleaning module, used for inputting the raw data into a preset data cleaning model to clean the raw data into metadata, wherein the metadata is used to represent the attribute characteristics of the raw data; A data deconstruction module, used for inputting the metadata into a preset deconstruction model to deconstruct the metadata into structured data, wherein the structured data is compilable, storable and readable data; A classification storage module, used to classify and store the structured data using a preset dictionary model to aggregate the data; A data cleaning model training module is used to: obtain an initial training set, wherein the training data in the initial training set all carry annotation information, and the annotation information is used to annotate the cleaning data corresponding to the training data; iteratively train the initial model using the initial training set until the average value of the overlap rate between the first predicted cleaning result output by the initial model and the cleaning data in the annotation information is greater than or equal to a first threshold, and then stop the iteration to obtain an intermediate model, wherein the data size of the training set used in each iteration gradually decreases during the iteration process, the initial model is a deep neural network model, the first predicted cleaning result includes the predicted cleaning results for each of the training data, and the average value of the overlap rate is used to represent the prediction accuracy of the intermediate model; re-input the initial training set into the intermediate model, and obtain the second predicted cleaning result of the intermediate model for the initial training set; when it is determined based on the second predicted cleaning result that the prediction accuracy of the intermediate model is greater than or equal to the first threshold, the intermediate model is determined as the data cleaning model; The data cleaning model training module also includes an iterative training unit, which is specifically used to: obtain the first intermediate training set used in the previous iteration, and obtain the first predicted cleaning result output by the initial model for the first intermediate training set in the previous iteration; determine the average value of the difference rate between the first predicted cleaning result and the cleaning data in the annotation information; use the average value of the difference rate to determine the loss value of the objective function of the initial model; adjust the model parameters of the initial model according to the loss value; use the model parameters as new initial parameters of the initial model, and reduce the data scale of the first intermediate training set to obtain a second intermediate training set, wherein the initial parameters used when the initial model is first trained are preset initial parameters; input the second intermediate training set into the initial model for this iterative training; The iterative training unit is further used to: sort the training data in the first intermediate training set according to the difference rate between the first predicted cleaning result and the cleaning data in the annotation information; extract the training data located before the target sorting position from the sorting result to form the second intermediate training set; The data security control module is specifically used to: upon receiving a data retrieval request from a target object for a target classification, extract the first key carried in the data retrieval request; and when the first key is paired with the second key of the target classification, send the data access interface of the target classification to the target object.

5. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 3.

Images