A multi-source data processing method based on a low-code platform

Abstract

The present invention proposes a multi-source data processing method based on a low-code platform, which relates to the technical field of data interactive processing. Data from multiple data sources are obtained, and a low-code data management platform is used to convert all data in a target database into data to be parsed in a target format; the data to be parsed in the target format is divided into fragment data sequences, and each fragment data sequence is closed-loop parsed to obtain multiple parsed data sets, and cluster analysis is performed on the multiple parsed data sets to form multiple data subsets; the intersection of the multiple data subsets is taken to obtain the highest frequency intersection data sequence, thereby obtaining a high frequency intersection data sequence, and the data item with the largest number of repetitions in the highest frequency intersection data sequence is output as the data item of focus, which helps to simplify the data processing process and improve the parsing efficiency, and can also enhance the accuracy and flexibility of data parsing.

Description

Technical Field

[0001] The present invention relates to the technical field of data interaction processing, and in particular to a multi-source data processing method based on a low-code platform. Background Art

[0002] Multi-source data processing methods mainly involve integrating, cleaning, transforming and analyzing data from different sources, formats and structures to provide consistent, accurate and useful information. These data sources may include databases, APIs, files, social media, IoT devices, etc.

[0003] Low-code platforms usually provide a variety of pre-built data connectors or adapters that can easily connect to various data sources (such as databases, APIs, cloud storage services, third-party applications, etc.). Users can define data source connections, data synchronization frequency, and data formats through simple configuration or drag-and-drop operations. This enables even users with non-technical backgrounds to quickly and efficiently obtain data from various sources.

[0004] Since the data structures and formats of different data sources may vary greatly, data mapping and conversion are important steps when processing multi-source data. Low-code platforms provide intuitive data mapping tools that enable users to easily map data fields from different sources to a unified data model. In addition, conversion tools can help users handle data cleaning, aggregation, formatting and other conversion operations to ensure data consistency and accuracy.

[0005] Low-code platforms integrate a variety of powerful data analysis tools that can help users conduct exploratory data analysis, generate reports, and create visual charts and dashboards. Users do not need to write complex queries or scripts, but can create a variety of customized data views and reports by dragging and dropping, configuring, etc. These tools not only improve the efficiency of data processing, but also make data analysis more intuitive and easy to understand, helping users better understand data and make more informed decisions.

[0006] However, in the existing technology, the multi-source data processing method based on the low-code platform still needs to be customized and optimized according to specific business needs and data characteristics, and further simplify the data processing process, improve the analysis efficiency, and enhance the accuracy and flexibility of data analysis. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a multi-source data processing method based on a low-code platform, comprising the following steps:

[0008] S1. Obtain data from multiple data sources and use the low-code data management platform to convert all data in the target database into data to be parsed in the target format;

[0009] S2, dividing the data to be parsed in the target format into fragment data sequences, performing closed-loop parsing on each fragment data sequence to obtain multiple parsed data sets, and performing cluster analysis on the multiple parsed data sets to form multiple data subsets;

[0010] S3. Take the intersection of the multiple data subsets to obtain multiple intersection data sequences, take the union of the multiple data subsets to obtain a union data sequence, export the intersection data sequence with the highest frequency of occurrence in the union data sequence, and count the data items with the largest number of repetitions in the intersection data sequence with the highest frequency as key focus data items.

[0011] Furthermore, in step S2, closed-loop analysis of the fragment data sequence includes:

[0012] Calculate the relationship between the data in the fragment data sequence, and extract data features representing the relationship between the data from the fragment data sequence;

[0013] Set segment limit Y th , when the data feature exceeds the segment limit Y th , the fragment data sequence is taken as a parsed dataset.

[0014] Furthermore, the clustering value S between the two parsed data sets is calculated:

[0015] ;

[0016] In the formula, x i Indicates the i-th data in the parsed data set x, y j Indicates parsing the jth data in the dataset y. and Represent the average values ​​of the data in the parsed data sets x and y, respectively. A and B represent the total number of data in the parsed data sets x and y, respectively.

[0017] Multiple parsed data sets with clustering values ​​greater than a threshold are grouped into one category, ultimately forming multiple data subsets.

[0018] Furthermore, the fragment limit Y th for:

[0019] ;

[0020] Among them, Y is the data feature of the fragment data sequence, and M is the number of data in the fragment data sequence.

[0021] Furthermore, the step S3 comprises:

[0022] S31: performing an intersection operation on the multiple data subsets to obtain an intersection data sequence;

[0023] S32: after obtaining the intersection data sequence, calculating the occurrence frequency of the intersection data sequence in the union of multiple data subsets;

[0024] S33: sorting the intersection data sequences according to the occurrence frequencies, and selecting the intersection data sequence with the highest occurrence frequency;

[0025] S34: Counting a plurality of data items whose occurrence times in the intersection data sequence with the highest occurrence frequency are higher than a preset target number, and taking the plurality of data items as key focus data items.

[0026] Further, calculate the performance value X of each focused data item N (T):

[0027] X N (T) = (1-b) K D *q*c;

[0028] Among them, D represents the set of data items that are of key concern, c is the number of repetitions of the data items of key concern in the intersection data sequence with the highest occurrence frequency, and K D represents the number of data items in the focused data item set, q represents the frequency of occurrence of the intersection data sequence where the focused data item T is located, b represents the efficiency coefficient, 0≤b<1;

[0029] Calculate the sum of the performance values ​​of all the focused data items as the cumulative performance value of the focused data item set, and determine whether the cumulative performance value of the focused data item set is greater than the performance threshold. When the cumulative performance value is greater than the performance threshold, determine the focused data item set as the target focused data item set.

[0030] Furthermore, in the step S1, the data from multiple data sources are sent to a target database of a corresponding data format on a low-code data management platform for storage, and all the data in the target database are converted into data to be parsed in a target format through a preset conversion format code.

[0031] Furthermore, the data feature is one of the mean, median, mode, variance and standard deviation of the fragment data sequence data.

[0032] Compared with the prior art, the present invention has the following beneficial technical effects:

[0033] Obtain data from multiple data sources, and use a low-code data management platform to convert all data in the target database into data to be parsed in the target format; parse the data in the target database to obtain data in the required storage format, reducing learning costs and shortening the development cycle.

[0034] The data to be parsed in the target format is divided into fragment data sequences, each fragment data sequence is closed-loop parsed to obtain multiple parsed data sets, and multiple parsed data sets are clustered to form multiple data subsets; the intersection of multiple data subsets is taken to obtain the intersection data sequence with the highest frequency, thereby obtaining the intersection data sequence with the highest frequency, and the data item with the largest number of repetitions in the intersection data sequence with the highest frequency is output as the data item of focus. This helps to simplify the data processing process, improve the parsing efficiency, and also enhance the accuracy and flexibility of data parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0036] Figure 1 It is a flowchart of the multi-source data processing method based on the low-code platform of the present invention;

[0037] Figure 2 It is a schematic flow chart of a method for outputting the data items of focus in step S3 of the present invention;

[0038] Figure 3 This is a schematic diagram of the steps for generating the data items that are the focus of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and cannot constitute a limitation on the signal transmission direction, connection sequence and size, dimensions and shape of the components or structures.

[0041] Example 1

[0042] like Figure 1 As shown, it is a schematic diagram of the multi-source data processing method based on the low-code platform of the present invention, and the multi-source data processing method includes the following steps:

[0043] Step S1: Obtain data from multiple data sources, and use the low-code data management platform to convert the data from multiple data sources in the target database into data to be parsed in the target format.

[0044] The data source server reads multiple data source data transmitted by all data interfaces it manages, and obtains multiple data formats of multiple data source data. The low-code data management platform adopts a relational database management system, a database system built on a relational model, which uses a two-dimensional table format to store information. Multi-source data is stored in the form of a table, which reduces complex data structures to simple binary relationships, that is, a two-dimensional table format, and stores multi-source data in the form of rows and columns for easy user understanding. A series of rows and columns is called a table. A group of tables constitutes a database. The data records between tables are represented by a relational model.

[0045] Relational database management systems support the use of structured query language (SQL) for data operations, can ensure data consistency and integrity, and ensure data security and isolation through transaction processing.

[0046] Send data from multiple data sources to the target database of the corresponding data format on the low-code data management platform for storage, so that the low-code data management platform can read and parse the data in the target database, parse the data in the target database to obtain the data in the required storage format, reduce learning costs, and shorten the development cycle.

[0047] In a preferred embodiment, the conversion format code of the target database to be read is pre-written. In the target database, according to the preset conversion format code, after all the data in the target database is obtained, all the data in the target database are converted into data to be parsed in the target format through the preset conversion format code.

[0048] Step S2: Divide the data to be parsed in the target format into fragment data sequences, perform closed-loop parsing on each fragment data sequence to obtain multiple parsed data sets, and perform cluster analysis on the multiple parsed data sets to form multiple data subsets.

[0049] Step S21: Perform closed-loop analysis on each fragment data sequence to obtain multiple analysis data sets.

[0050] First, the target format data to be parsed is divided into multiple fragment data sequences, and then each fragment data sequence is closed-loop parsed. This method helps to simplify the data processing process, improve the parsing efficiency, and also enhance the accuracy and flexibility of data parsing.

[0051] First, determine the division criteria of the data to be parsed, which can be the format, structure, length, etc. of the data. Then, according to the determined division criteria, cut the data to be parsed into multiple fragment data sequences, define corresponding parsing rules for each fragment data sequence, and parse each fragment data sequence according to the defined parsing rules. These rules can be based on specific algorithms, logic and other technical means.

[0052] After parsing is complete, the results are verified to ensure their accuracy and validity. This can be achieved by comparing with other reliable data sources, using business rules for verification, or performing statistical analysis. If errors or inconsistencies are found during the verification process, the parsing rules need to be adjusted and the parsing process is repeated. This parsing process is a closed loop, which means that parsing and verification are repeated until a satisfactory result is obtained.

[0053] According to an embodiment of the present invention, in the closed-loop parsing process of the segment data sequence, by performing closed-loop parsing of the segment data sequence for each part, a plurality of segment data sequence parsing result data sets can be obtained. th .

[0054] The closed-loop analysis of the fragment data sequence specifically includes the following steps:

[0055] First, feature extraction is performed. Feature extraction is a crucial step in closed-loop parsing and is essential for discovering hidden information and structures in the data.

[0056] Calculate the relationship between data in the fragment data sequence, and extract data features representing the relationship between data from the fragment data sequence, such as mean, median, mode, variance, standard deviation, etc., to reveal the distribution and change of the data.

[0057] Preferably, for text data, features such as word frequency, bag-of-words model, TF-IDF, etc. can be extracted; for image data, features such as edge, texture, shape, etc. can be extracted; for time series data, features such as periodicity, seasonality, trend, autocorrelation, etc. can be extracted.

[0058] Next, set the fragment limit Y th . Fragment limit Y th It is a criterion for evaluating the parsing results of a fragment data sequence. thWhen the segmentation result of the fragment data sequence is meaningful, the fragment data sequence is taken as a parsing data set and included in the subsequent overall analysis.

[0059] ;

[0060] Among them, Y is the data feature of the fragment data sequence, M is the number of data in the fragment data sequence, and according to the defined fragment limit Y th The segmentation results of the fragment data sequence are screened and evaluated to ensure that the segmentation results meet the standard requirements.

[0061] In the process of closed-loop analysis of fragment data sequences, closed-loop analysis of each fragment data sequence is a key step. Closed-loop analysis is to discover patterns, trends or associations through in-depth analysis of data. By processing the fragment data sequences one by one, a series of data sets are generated, each of which represents the analysis result of the fragment data sequence.

[0062] Set segment limit Y th The purpose is to filter and screen the results of data analysis to ensure that only data patterns that meet certain standards are included in the final analysis results.

[0063] Step S22: performing association analysis on the multiple parsed data sets obtained in step S21 to form multiple data subsets.

[0064] After completing the closed-loop analysis of all fragment data sequences, multiple analysis data sets are obtained, which need to be further associated with each other to form multiple data subsets.

[0065] Calculate the clustering value S between two parsed data sets. The specific formula is:

[0066] ;

[0067] In the formula, x i Indicates the i-th data in the parsed data set x, y j Indicates parsing the jth data in the dataset y. and Represent the average values ​​of the data in the parsed data sets x and y, respectively. A and B represent the total number of data in the parsed data sets x and y, respectively.

[0068] Multiple parsed data sets with clustering values ​​greater than a threshold are grouped into one category, ultimately forming multiple data subsets.

[0069] Step S3: Intersect the multiple data subsets to obtain multiple intersection data sequences, union the multiple data subsets to obtain a union data sequence, derive the intersection data sequence with the highest frequency in the union data sequence, and count the data items with the largest number of repetitions in the intersection data sequence with the highest frequency as the key data items. Figure 2 As shown, the following steps are included:

[0070] Step S31: performing an intersection operation on the multiple data subsets formed in step S2 to obtain multiple intersection data sequences, that is, only the data sequences that appear in all data subsets at the same time will be retained.

[0071] In a preferred embodiment, if the data subsets are complex data structures, such as containing nested lists, the intersection operation requires more complex logic to process, so as to find out the common features or patterns in all data subsets.

[0072] Step S32: taking a union of multiple data subsets to obtain a union data sequence, calculating the occurrence frequencies of multiple intersection data sequences in the union data sequence, and thus determining the actual occurrence frequencies of the intersection data sequences in the entire data set.

[0073] Step S33: sorting the multiple intersection data sequences according to the calculated occurrence frequencies, and selecting the intersection data sequence with the highest occurrence frequency.

[0074] Step S34: Count multiple data items whose repetition times in the intersection data sequence with the highest occurrence frequency are higher than the preset target times, and use the multiple data items as the multi-source data processing results, that is, the data items of focus, such as Figure 3 shown.

[0075] Example 2

[0076] In the technical solution of implementation 1, in step S3, the performance value of each key focus data item is further calculated based on the number of repetitions of each key focus data item, and the sum of the calculated performance values ​​of each key focus data item is used as the cumulative performance value of the key focus data item set.

[0077] Specifically, for each focused data item, the performance value of the focused data item is calculated using the following formula according to the frequency of occurrence of the intersection data sequence in which the focused data item is located:

[0078] X N (T) = (1-b) K D *q*c;

[0079] Among them, D is used to represent the set of data items of focus, c is the number of repetitions of the data items of focus in the intersection data sequence with the highest frequency, and K is D It is used to represent the number of data items in the data item set of the focus, and q is used to represent the frequency of occurrence of the intersection data sequence of the data item T of the focus. T is used to represent the data item of the focus, and X N (T) is used to represent the effectiveness value of the key data item T, b is used to represent the effectiveness coefficient, 0≤b<1, the effectiveness coefficient b is determined by the impact of the historical data of the key data item, and a effectiveness coefficient b between 0 and 1 is determined based on the analysis results. This effectiveness coefficient can reflect the degree of important influence of the key data item T. If the key data item T has a great impact, then b should be close to 1; if the impact is small or not significant, then b should be close to 0.

[0080] Calculate the sum of the performance values ​​of all the focused data items as the cumulative performance value of the focused data item set, and determine whether the cumulative performance value of the focused data item set is greater than the performance threshold; when the cumulative performance value is greater than the performance threshold, determine the focused data item set as the target focused data item set.

[0081] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0082] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-source data processing method based on a low-code platform, characterized in that: The steps include: S1. Obtain data from multiple data sources, and use the low-code data management platform to convert the data from multiple data sources in the target database into data to be parsed in the target format; S2. Divide the data to be parsed in the target format into fragment data sequences, and perform closed-loop parsing on each fragment data sequence. The closed-loop parsing of the fragment data sequence includes: Calculate the relationship between the data in the fragment data sequence, and extract data features representing the relationship between the data from the fragment data sequence; Set segment limit Y th , when the data feature exceeds the segment limit Y th When , the fragment data sequence is taken as a parsed data set; Perform cluster analysis on the obtained multiple analytical data sets to form multiple data subsets; S3. Take the intersection of the multiple data subsets to obtain multiple intersection data sequences, take the union of the multiple data subsets to obtain a union data sequence, export the intersection data sequence with the highest frequency of occurrence in the union data sequence, and count the data items with the largest number of repetitions in the intersection data sequence with the highest frequency as key focus data items.

2. The multi-source data processing method based on a low-code platform according to claim 1 is characterized in that: Calculate the clustering value S between two parsed datasets: ； In the formula, x i Indicates the i-th data in the parsed data set x, y j Indicates parsing the jth data in the dataset y. and Represent the average values ​​of the data in the parsed data sets x and y, respectively. A and B represent the total number of data in the parsed data sets x and y, respectively. Multiple parsed data sets with clustering values ​​greater than a threshold are grouped into one category, ultimately forming multiple data subsets.

3. The multi-source data processing method based on a low-code platform according to claim 1 is characterized in that: Fragment limit Y th for: ； Among them, Y is the data feature of the fragment data sequence, and M is the number of data in the fragment data sequence.

4. The multi-source data processing method based on a low-code platform according to claim 2 is characterized in that: The step S3 comprises: S31: performing an intersection operation on the multiple data subsets to obtain multiple intersection data sequences; S32: Taking a union of multiple data subsets to obtain a union data sequence, and calculating the occurrence frequency of multiple intersection data sequences in the union data sequence; S33: sorting the multiple intersection data sequences according to the occurrence frequencies, and selecting the intersection data sequence with the highest occurrence frequency; S34: Counting a plurality of data items whose repetition times are higher than a preset target number in the intersection data sequence with the highest occurrence frequency, and taking the plurality of data items as key data items.

5. The multi-source data processing method based on a low-code platform according to claim 4 is characterized in that: Calculate the performance value X for each focused data item N (T): X N (T)=(1-b)K D *q*c; Among them, D represents the set of data items that are of key concern, c is the number of repetitions of the data items of key concern in the intersection data sequence with the highest occurrence frequency, and K D represents the number of data items in the focused data item set, q represents the frequency of occurrence of the intersection data sequence where the focused data item T is located, b represents the efficiency coefficient, 0≤b<1; Calculate the sum of the performance values ​​of all the focused data items as the cumulative performance value of the focused data item set, and determine whether the cumulative performance value of the focused data item set is greater than the performance threshold. When the cumulative performance value is greater than the performance threshold, determine the focused data item set as the target focused data item set.

6. The multi-source data processing method based on a low-code platform according to claim 1 is characterized in that: In step S1, data from multiple data sources are sent to a target database of corresponding data format on a low-code data management platform for storage, and all data in the target database are converted into data to be parsed in a target format using a preset conversion format code.

7. The multi-source data processing method based on a low-code platform according to claim 3 is characterized in that: The data feature is one of the mean, median, mode, variance, and standard deviation of the data in the segment data sequence.

Images