Data conversion and storage method based on adaptive matrix operation
Through the definition of data and automatic database tables of data through the adaptive matrix operation method, the problem of low data conversion storage efficiency and high cost in the existing technology, and realized high -efficiency and automated data conversion and storage procedures.
Patent Information
- Application Number
- CN202411394408.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-01-24
AI Technical Summary
The existing data conversion storage method is inefficient and costly, and requires manual writing of conversion job files and lack of a unified data storage format, resulting in high development costs and high database table changes.
Adopting a data conversion and storage method based on adaptive matrix operations, by obtaining original data, defining data attributes, performing adaptive format conversion, extracting attribute values and data value matrices, performing matrix multiplication and straightening, automatically creating database tables and synchronously storing them data.
The efficiency of data conversion and storage has been improved, the cost of changing the conversion script and database table is reduced, and automated data conversion and storage procedures are realized.
Smart Images
Figure CN119357262A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data processing, and in particular to a data conversion and storage method based on adaptive matrix operations. Background Art
[0002] The current real-time conversion and storage of Internet data uses a solution that combines program scripts with ETL (Extract-Transform-Load) tools or big data frameworks, which is mainly divided into three steps: collection, conversion, and loading. The main ETL tools involved are DataPipline, Talend, Informatica, kettle, etc., and big data frameworks are Spark, Flink, etc. After the data is downloaded from the target data set through the script, it is further cleaned, converted, and loaded through the ETL tool, and finally stored in the database or data warehouse as required for further use.
[0003] The defects of the existing technology are mainly reflected in the following two aspects: First, the tools used by technologies such as DataPipline, Talend, Informatica, and kettle require manual writing of conversion job files. Changes in the input data format or fields will cause corresponding changes in the conversion job, and the existing job files need to be modified repeatedly, reducing the efficiency of service deployment. Second, the existing technology does not define a unified data storage format, and there is no standard for database storage table fields. In the development process, it is often necessary to build tables for storage of data in different formats, which has a high development cost. When the data source format changes, the cost of changing the database table is also high. Summary of the invention
[0004] The present application aims to solve the problems of low efficiency and high cost in existing data conversion and storage methods, and proposes a data conversion and storage method based on adaptive matrix operations.
[0005] The technical solution adopted by this application to solve the above technical problems is:
[0006] The present application provides a data conversion and storage method based on adaptive matrix operation, the method comprising:
[0007] Acquire original data, wherein the original data includes at least two attribute values and their corresponding data values;
[0008] Defining data attributes according to attribute values in the original data, and performing adaptive format conversion on the original data according to the data attributes to obtain at least one corresponding original matrix;
[0009] Extracting the attribute value submatrix corresponding to each data attribute from the original matrix, and performing matrix multiplication operation on the attribute value submatrix corresponding to each data attribute to obtain an attribute value matrix;
[0010] Extracting a data value matrix from the original matrix, making one-to-one correspondence between the attribute values in the attribute value matrix and the data values in the data value matrix, respectively straightening the attribute value matrix and the data value matrix, and splitting each element in the straightened attribute value matrix;
[0011] A database table is adaptively created according to the defined data attributes, and the data in the split attribute value matrix and the straightened data value matrix are synchronously stored in the database table.
[0012] Furthermore, the original data includes two attribute values and their corresponding data values, and the corresponding original matrix is as follows:
[0013]
[0014]
[0015]
[0016]
[0017] Among them, A represents the original matrix, X and Y represent the attribute value sub-matrices corresponding to the two data attributes, A0 represents the data value matrix, T represents the matrix transpose, and x m Represents the mth attribute value of the first data attribute, y n Indicates the nth attribute value of the second data attribute, A mn Represents the data value corresponding to the mth attribute value of the first data attribute and the nth attribute value of the second data attribute.
[0018] Furthermore, matrix multiplication is performed on the attribute value submatrix corresponding to each data attribute to obtain the attribute value matrix, and its calculation formula is as follows:
[0019]
[0020] Among them, R nm Represents a matrix of attribute values.
[0021] Furthermore, the attribute value matrix after straightening is as follows:
[0022]
[0023] The data value matrix after straightening is as follows:
[0024]
[0025] Among them, R represents the attribute value matrix after straightening, A r Represents the matrix of data values after straightening.
[0026] Furthermore, the attribute value matrix after splitting is as follows:
[0027]
[0028] Among them, R r Represents the attribute value matrix after splitting.
[0029] Furthermore, the original data includes three attribute values and their corresponding data values, and adaptive format conversion is performed on the original data according to the data attributes, specifically including:
[0030] Performing adaptive format conversion on the original data according to a certain data attribute to obtain a plurality of original matrices corresponding to the attribute value of the data attribute, wherein the number of the original matrices is the same as the type of the attribute value of the data attribute;
[0031] The attribute value submatrix corresponding to the data attributes extracted from the original matrix is as follows:
[0032]
[0033]
[0034]
[0035] The matrix of data values obtained by extracting them from the original matrix is as follows:
[0036]
[0037] Among them, X, Y and Z represent the attribute value sub-matrices corresponding to the three data attributes, A0 represents the data value matrix, x m Represents the mth attribute value of the first data attribute, y n Represents the nth attribute value of the second data attribute, z k Represents the kth attribute value of the third data attribute, A mn Represents the data values corresponding to the mth attribute value of the first data attribute, the nth attribute value of the second data attribute, and the kth attribute value of the third attribute.
[0038] Furthermore, matrix multiplication is performed on the attribute value submatrix corresponding to each data attribute to obtain the attribute value matrix, and its calculation formula is as follows:
[0039]
[0040] Among them, Rnm Represents a matrix of attribute values.
[0041] Furthermore, the attribute value matrix after straightening is as follows:
[0042]
[0043] The data value matrix after straightening is as follows:
[0044]
[0045] Among them, R represents the attribute value matrix after straightening, A r Represents the matrix of data values after straightening.
[0046] Furthermore, the attribute value matrix after splitting is as follows:
[0047]
[0048] Among them, R r Represents the attribute value matrix after splitting.
[0049] The beneficial effects of the present application are as follows: the data conversion and storage method based on adaptive matrix operation provided by the present application can convert data into one dimension based on the adaptive matrix operation method, and automatically complete the database table definition, and finally use the data conversion tool to convert and store Internet data in real time, so as to facilitate the subsequent data visualization display. For data sources with frequently changed formats, the present application can automatically perform corresponding data conversion and create table storage by configuring the defined data attributes, thereby improving the efficiency of data conversion and storage and reducing the cost of changing conversion scripts and database tables. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A flowchart of a data conversion and storage method based on adaptive matrix operations provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0052] The technical solution of the embodiment of the present application is applicable to the data conversion and loading and storage stage. Since the current data conversion and storage method is basically to convert and load data through ETL tools, but this method requires manual writing of conversion job files, data in different formats or fields need to use different conversion job files, and the conversion job files need to be modified repeatedly, and the data conversion efficiency is low. In addition, the prior art does not define a unified data storage format, and there is no standard for database storage table fields. In the development process, it is often necessary to build tables for data in different formats, resulting in high development costs, and when the data source format changes, the cost of changing the database table is high.
[0053] Based on this, the technical solution of the present application is proposed. In an embodiment of the present application, original data is obtained, wherein the original data includes at least two attribute values and their corresponding data values; data attributes are defined according to the attribute values in the original data, and the original data is adaptively formatted according to the data attributes to obtain at least one corresponding original matrix; an attribute value submatrix corresponding to each data attribute is extracted from the original matrix, and a matrix multiplication operation is performed on the attribute value submatrix corresponding to each data attribute to obtain an attribute value matrix; a data value matrix is extracted from the original matrix, and after one-to-one correspondence between the attribute values in the attribute value matrix and the data values in the data value matrix, the attribute value matrix and the data value matrix are straightened respectively, and each element in the straightened attribute value matrix is split; a database table is adaptively created according to the defined data attributes, and the data in the split attribute value matrix and the straightened data value matrix are synchronously stored in the database table.
[0054] Specifically, after obtaining the original data, the present application first defines the data attributes according to the attribute values in the original data, and performs adaptive format conversion on the original data to obtain at least one original matrix, which is used to represent a two-dimensional table. If the original data contains two data attributes, there is only one original matrix; then the attribute value matrix and the data value matrix are extracted from the original matrix, and the attribute value matrix and the data value matrix are adaptively processed accordingly; finally, the data in the processed matrix is synchronously stored in a database table pre-created according to the data attributes to complete the conversion and storage of the data. For data sources that frequently change formats, the present application can automatically perform corresponding data conversion and create tables for storage by configuring the defined data attributes, thereby improving the efficiency of data conversion and storage and reducing the cost of changing conversion scripts and database tables.
[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.
[0056] See also Figure 1 , the data conversion and storage method based on adaptive matrix operation provided in the embodiment of the present application comprises the following steps:
[0057] S1. Acquire original data, wherein the original data includes at least two attribute values and their corresponding data values;
[0058] The original data may be downloaded Internet data. In practical applications, the method of obtaining the original data may include: obtaining the address of the target site, analyzing the page data interface and tag structure, writing a download script, and then downloading the data to a local specified path and storing it in a unified JSON or Excel format.
[0059] In an embodiment of the present application, the original data may include at least two attribute values and their corresponding data values. For example, the original data is the consumption weight of multiple energy sources in different years, such as the consumption weight of coal, oil and natural gas in 2019, 2020 and 2021, where the specific year and the specific energy type are two attribute values, that is, 2019, 2020 and 2021 are the first attribute value, coal, oil and natural gas are the second attribute value, and the specific consumption weight is the data value.
[0060] S2, defining data attributes according to attribute values in the original data, and performing adaptive format conversion on the original data according to the data attributes to obtain at least one corresponding original matrix;
[0061] In the embodiment of the present application, if the original data includes two attribute values and their corresponding data values, the corresponding original matrix is as follows:
[0062]
[0063]
[0064]
[0065]
[0066] Among them, A represents the original matrix, X and Y represent the attribute value sub-matrices corresponding to the two data attributes, A0 represents the data value matrix, T represents the matrix transpose, and x m Represents the mth attribute value of the first data attribute, y n Indicates the nth attribute value of the second data attribute, A mn Represents the data value corresponding to the mth attribute value of the first data attribute and the nth attribute value of the second data attribute.
[0067] In an embodiment of the present application, if the original data includes three attribute values and their corresponding data values, adaptive format conversion is performed on the original data according to the data attributes, specifically including:
[0068] Performing adaptive format conversion on the original data according to a certain data attribute to obtain a plurality of original matrices corresponding to the attribute value of the data attribute, wherein the number of the original matrices is the same as the type of the attribute value of the data attribute;
[0069] The attribute value submatrix corresponding to the data attributes extracted from the original matrix is as follows:
[0070]
[0071]
[0072]
[0073] The matrix of data values obtained by extracting them from the original matrix is as follows:
[0074]
[0075] Among them, X, Y and Z represent the attribute value sub-matrices corresponding to the three data attributes, A0 represents the data value matrix, x m Represents the mth attribute value of the first data attribute, y n Represents the mth attribute value of the second data attribute, z k Represents the kth attribute value of the third data attribute, A mn Represents the data values corresponding to the mth attribute value of the first data attribute, the nth attribute value of the second data attribute, and the kth attribute value of the third attribute.
[0076] S3, extracting the attribute value submatrix corresponding to each data attribute from the original matrix, and performing matrix multiplication operation on the attribute value submatrix corresponding to each data attribute to obtain an attribute value matrix;
[0077] In the embodiment of the present application, if the original data includes two attribute values and their corresponding data values, the calculation formula of the attribute value matrix is as follows:
[0078]
[0079] Among them, R nm Represents a matrix of attribute values.
[0080] In the embodiment of the present application, if the original data includes three attribute values and their corresponding data values, the calculation formula of the attribute value matrix is as follows:
[0081]
[0082] S4, extracting a data value matrix from the original matrix, making one-to-one correspondence between the attribute values in the attribute value matrix and the data values in the data value matrix, straightening the attribute value matrix and the data value matrix respectively, and splitting each element in the straightened attribute value matrix;
[0083] In the embodiment of the present application, if the original data includes two attribute values and their corresponding data values, the attribute value matrix after straightening processing is as follows:
[0084]
[0085] The attribute value matrix after splitting is as follows:
[0086]
[0087] The data value matrix after straightening is as follows:
[0088]
[0089] In the embodiment of the present application, if the original data includes three attribute values and their corresponding data values, the attribute value matrix after straightening processing is as follows:
[0090]
[0091] The attribute value matrix after splitting is as follows:
[0092] The data value matrix after straightening is as follows:
[0093]
[0094] Among them, R represents the attribute value matrix after straightening, R r represents the attribute value matrix after splitting, A r Represents the matrix of data values after straightening.
[0095] S5. Adaptively create a database table according to the defined data attributes, and synchronously store the data in the split attribute value matrix and the straightened data value matrix into the database table.
[0096] The embodiment of the present application passes the defined data attributes as parameters into the corresponding program, and then generates table fields according to the data attributes, and automatically generates and runs the SQL table creation statement corresponding to the table to complete the database table creation. For example, if the defined data attributes are "time" and "energy type", then "time" and "energy type" are written into the corresponding field creation statement in the SQL statement. Finally, the data in the split attribute value matrix and the straightened data value matrix can be stored in the corresponding data table library by running the ETL tool to complete the conversion and storage of the data.
[0097] The following uses two embodiments in combination with specific data to illustrate the specific implementation methods of data conversion and storage when the original data includes two attribute values and three attribute values.
[0098] Embodiment 1:
[0099] When the original data includes two attribute values, it is assumed that the original table data corresponding to the original data is as follows (unit: 10,000 tons):
[0100] 2019 2020 2021 coal 100 130 180 oil 20 80 40 LNG (Liquefied Natural Gas) 300 200 280
[0101] The first attribute value in the original data is: 2019, 2020, 2021, and the second attribute value is: coal, oil, LNG, and the corresponding data values are 100, 20, 300, 130, 80, 200, 180, 40, 280.
[0102] The data attribute defined according to the first attribute value is "time", and the data attribute defined according to the second attribute value is "energy type".
[0103] There is only one original matrix A corresponding to the original data, as follows:
[0104]
[0105] The attribute value sub-matrices X and Y corresponding to the two attribute values in the original matrix A, and the data value matrix A0 are as follows:
[0106]
[0107]
[0108]
[0109] Attribute value matrix R nm The calculation formula is as follows:
[0110]
[0111] The attribute value matrix R after straightening is as follows:
[0112]
[0113] The attribute value matrix after splitting is as follows:
[0114]
[0115] The data value matrix after straightening is as follows:
[0116]
[0117] The data in the split attribute value matrix and the straightened data value matrix are synchronously stored in the database table, and the resulting table is as follows:
[0118]
[0119]
[0120] Embodiment 2:
[0121] When the original data includes three attribute values, it is assumed that the original table data corresponding to the original data is as follows (unit: 10,000 tons):
[0122] Energy imports:
[0123] import 2019 2020 2021 coal 100 130 180 oil 20 80 40 LNG (Liquefied Natural Gas) 300 200 280
[0124] Energy exports:
[0125] exit 2019 2020 2021 coal 50 100 100 oil 60 90 30 LNG (Liquefied Natural Gas) 200 400 500
[0126] The above two tables can be combined into a three-dimensional table, which contains the original table data with three attribute values. Among them, the first attribute value in the original data is: 2019, 2020, 2021, the second attribute value is: coal, oil, LNG, and the third attribute is: import, export, and the corresponding data values are 100, 20, 300, 130, 80, 200, 180, 40, 280, 50, 60, 200, 100, 90, 400, 100, 30, 500.
[0127] The data attribute defined according to the first attribute value is "time", the data attribute defined according to the second attribute value is "energy type", and the data attribute defined according to the third attribute value is "import and export".
[0128] There are two original matrices A corresponding to the original data, corresponding to the import and export of the third attribute value. The original matrix A1 corresponding to the import is as follows:
[0129]
[0130] The data value matrix A in the original matrix A1 01 as follows:
[0131]
[0132] The original matrix A2 corresponding to the export is as follows:
[0133]
[0134] The data values in the original matrix A2 are in the matrix A 02 as follows:
[0135]
[0136] The attribute value sub-matrices X, Y, and Z corresponding to the three attribute values extracted from the two original matrices, as well as the data value matrix A0 are as follows:
[0137]
[0138]
[0139]
[0140]
[0141] Attribute value matrix R nm The calculation formula is as follows:
[0142]
[0143] The attribute value matrix R after straightening is as follows:
[0144]
[0145] The attribute value matrix after splitting is as follows:
[0146] The data value matrix after straightening is as follows:
[0147]
[0148] The data in the split attribute value matrix and the straightened data value matrix are synchronously stored in the database table, and the resulting table is as follows:
[0149]
[0150]
[0151] In summary, the data conversion and storage method based on adaptive matrix operation provided by the embodiment of the present application can convert data into one dimension based on the adaptive matrix operation method, automatically complete the database table definition, and finally use the data conversion tool to convert and store Internet data in real time to facilitate subsequent data visualization. For data sources that frequently change formats, the present application can automatically perform corresponding data conversion and create tables for storage by configuring defined data attributes, thereby improving the efficiency of data conversion and storage and reducing the cost of changing conversion scripts and database tables.
Claims
1. A data conversion and storage method based on adaptive matrix operation, characterized in that: The method comprises: Acquire original data, wherein the original data includes at least two attribute values and their corresponding data values; Defining data attributes according to attribute values in the original data, and performing adaptive format conversion on the original data according to the data attributes to obtain at least one corresponding original matrix; Extracting the attribute value submatrix corresponding to each data attribute from the original matrix, and performing matrix multiplication operation on the attribute value submatrix corresponding to each data attribute to obtain an attribute value matrix; Extracting a data value matrix from the original matrix, making one-to-one correspondence between the attribute values in the attribute value matrix and the data values in the data value matrix, straightening the attribute value matrix and the data value matrix respectively, and splitting each element in the straightened attribute value matrix; A database table is adaptively created according to the defined data attributes, and the data in the split attribute value matrix and the straightened data value matrix are synchronously stored in the database table.
2. The table data conversion and storage method based on adaptive matrix operation according to claim 1 is characterized in that: The original data includes two attribute values and their corresponding data values, and the corresponding original matrix is as follows: Among them, A represents the original matrix, X and Y represent the attribute value sub-matrices corresponding to the two data attributes, A0 represents the data value matrix, T represents the matrix transpose, and x m Represents the mth attribute value of the first data attribute, y n Indicates the nth attribute value of the second data attribute, A mn Represents the data value corresponding to the mth attribute value of the first data attribute and the nth attribute value of the second data attribute.
3. The table data conversion and storage method based on adaptive matrix operation according to claim 2 is characterized in that: Perform matrix multiplication on the attribute value submatrix corresponding to each data attribute to obtain the attribute value matrix. The calculation formula is as follows: Among them, R nm Represents a matrix of attribute values.
4. The table data conversion and storage method based on adaptive matrix operation according to claim 3 is characterized in that: The attribute value matrix after straightening is as follows: The data value matrix after straightening is as follows: Among them, R represents the attribute value matrix after straightening, A r Represents the matrix of data values after straightening.
5. The table data conversion and storage method based on adaptive matrix operation according to claim 4 is characterized in that: The attribute value matrix after splitting is as follows: Among them, R r Represents the attribute value matrix after splitting.
6. The table data conversion and storage method based on adaptive matrix operation according to claim 1, characterized in that: The original data includes three attribute values and their corresponding data values. The original data is adaptively formatted according to the data attributes, specifically including: Performing adaptive format conversion on the original data according to a certain data attribute to obtain a plurality of original matrices corresponding to the attribute value of the data attribute, wherein the number of the original matrices is the same as the type of the attribute value of the data attribute; The attribute value submatrix corresponding to the data attributes extracted from the original matrix is as follows: The matrix of data values obtained by extracting them from the original matrix is as follows: Among them, X, Y and Z represent the attribute value sub-matrices corresponding to the three data attributes, A0 represents the data value matrix, x m Represents the mth attribute value of the first data attribute, y n Represents the nth attribute value of the second data attribute, z k Represents the kth attribute value of the third data attribute, A mn Represents the data values corresponding to the mth attribute value of the first data attribute, the nth attribute value of the second data attribute, and the kth attribute value of the third attribute.
7. The table data conversion and storage method based on adaptive matrix operation according to claim 6 is characterized in that: Perform matrix multiplication on the attribute value submatrix corresponding to each data attribute to obtain the attribute value matrix. The calculation formula is as follows: Among them, R nm Represents a matrix of attribute values.
8. The table data conversion and storage method based on adaptive matrix operation according to claim 7 is characterized in that: The attribute value matrix after straightening is as follows: The data value matrix after straightening is as follows: Among them, R represents the attribute value matrix after straightening, A r Represents the matrix of data values after straightening.
9. The table data conversion and storage method based on adaptive matrix operation according to claim 8, characterized in that: The attribute value matrix after splitting is as follows: Among them, R r Represents the attribute value matrix after splitting.