A relational time series data processing method, device, equipment and storage medium

Abstract

The application discloses a relational time series data processing method and device, equipment and a storage medium, relates to the technical field of relational data processing, and comprises the following steps: obtaining a standard format data file from a file directory corresponding to data source coding; obtaining a variable value corresponding to a data block name in the standard format data file; judging whether a first substituted expression obtained by replacing a corresponding string in a target expression with the variable value exists a custom processing function; if the custom processing function exists, replacing the custom processing function in the first substituted expression with a numerical value obtained by executing the custom processing function to obtain a second substituted expression; and inserting data in the standard format data file into a target database through an SQL statement generated based on calculation results obtained by executing the second substituted expression and other field information in a configuration table constructed in advance, thereby solving the problems that relational time series data is poor in adaptability during implementation and delivery, and cannot be separated from source code and research and development participation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of relational data processing, and in particular relates to a relational time series data processing method, device, equipment and storage medium. BACKGROUND

[0002] At present, data processing develops with the development of IT, and is a mutually promoting relationship. Large data processing requires stronger computing processing capability, and stronger computing processing capability can meet the requirements of large data processing. However, low code is to abstract the standardizable business logic into a fixed group, and to open the customized part for code programming. The ETL (Extract-Transform-Load) process of data processing is a use scenario of low code, especially the aggregation function operation based on SQL (Structured Query Language). Although there are various commercial products or open source products for data processing on the market, some problems will be encountered when selecting and using, such as low efficiency when the data volume is large, which cannot meet the time delay requirement. Many products can quickly build a data processing framework and provide basic functions, but when the data volume is large, they cannot be distributedly deployed, cannot improve performance, and cannot meet the timeliness requirement of data. There is also a lack of flexibility, which can only meet specific use scenarios. In the extraction (collection), conversion (data meaning corresponding processing) and loading (insertion into the database) of data processing, there are many cases to be processed due to a plurality of cases in each link. Although these products have a certain openness, most of them cannot support low code development in the delivery stage. The products provided on the market are based on the knowledge and data processing experience of a certain industry, and form fixed functions for data conversion. If individualization is required, secondary development is required, which is complicated and requires high skills.

[0003] In summary, how to efficiently process relational time series data and how to adapt to customer customization requirements through low code are problems to be solved in the field. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a relational time series data processing method, device, equipment and storage medium, which can solve the problem that the relational time series data is poor in adaptability during implementation and delivery, and cannot be separated from the source code and the mode of research and development participation, improve the delivery capability and processing efficiency, improve the function module reuse rate, and shorten the new data source delivery cycle. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a relational time series data processing method, comprising:

[0006] determine a data source code corresponding to a current data source, and obtain a standard format data file from a file directory corresponding to the data source code;

[0007] obtain a variable value corresponding to a data block name in the data block according to the data block name, and replace a corresponding string in a target expression with the variable value to obtain a first replaced expression;

[0008] determine whether the first replaced expression includes a custom processing function;

[0009] if the first replaced expression includes the custom processing function, execute the custom processing function to obtain a corresponding numerical value, and replace the custom processing function in the first replaced expression with the numerical value to obtain a second replaced expression;

[0010] generate a corresponding SQL statement based on a calculation result of executing the second replaced expression and other field information in a pre-constructed configuration table, and insert corresponding data in the standard format data file into a target database through the SQL statement.

[0011] Optionally, the relational time series data processing method further includes:

[0012] extract data from a data source with a set data source code to obtain a corresponding original format data file;

[0013] convert the original format data file into a standard format data file by using a pre-determined format conversion program;

[0014] store the standard format data file in a file directory corresponding to the data source code.

[0015] Optionally, the converting the original format data file into a standard format data file by using a pre-determined format conversion program includes:

[0016] filter the original format data file based on a file name keyword of the original format data file to obtain a filtered file;

[0017] convert the filtered file into a standard format data file by using a pre-determined format conversion program.

[0018] Optionally, the determining whether the first replaced expression includes a custom processing function includes:

[0019] traverse each string in the first replaced expression to determine whether a preset leading character is included in the first replaced expression to obtain a corresponding determination result;

[0020] determine whether the first replaced expression exists the custom processing function based on the determination result.

[0021] Optionally, the determining whether the first replaced expression exists the custom processing function based on the determination result comprises:

[0022] If the determination result indicates that the first replaced expression exists the preset leading character, it is determined that the first replaced expression exists the custom processing function.

[0023] If the determination result indicates that the first replaced expression does not exist the preset leading character, it is determined that the first replaced expression does not exist the custom processing function.

[0024] Optionally, the relational time-series data processing method further comprises:

[0025] The target expression required for converting the variable into a corresponding field value is constructed through basic operators.

[0026] Optionally, after the corresponding data in the standard format data file is inserted into the target database through the SQL statement, the method further comprises:

[0027] determining whether the primary key in the target database is duplicated;

[0028] If the primary key in the target database is duplicated, it is determined that the current data insertion fails, and a corresponding update operation is performed.

[0029] If the primary key in the target database is not duplicated, it is determined that the current data insertion is successful.

[0030] In a second aspect, the application discloses a relational time-series data processing device, comprising:

[0031] A data source encoding determination module is configured to determine a data source encoding corresponding to a current data source.

[0032] A file acquisition module is configured to acquire a standard format data file from a file directory corresponding to the data source encoding.

[0033] A variable value acquisition module is configured to acquire a variable value corresponding to a data block name according to a data block corresponding to the data block name in the standard format data file currently traversed.

[0034] A string replacement module is configured to replace a corresponding string in a target expression with the variable value to obtain a first replaced expression.

[0035] The expression judgment module is configured to judge whether the first replaced expression has a custom processing function.

[0036] The function execution module is configured to, when the first replaced expression has the custom processing function, execute the custom processing function to obtain a corresponding numerical value, and replace the custom processing function in the first replaced expression with the numerical value to obtain a second replaced expression.

[0037] The SQL statement generation module is configured to generate a corresponding SQL statement based on a calculation result obtained by executing the second replaced expression and other field information in a configuration table constructed in advance, and insert corresponding data in the standard format data file into a target database through the SQL statement.

[0038] In a third aspect, the present application discloses an electronic device, comprising:

[0039] The memory is configured to save a computer program.

[0040] The processor is configured to execute the computer program to implement the steps of the relational time-series data processing method disclosed above.

[0041] In a fourth aspect, the present application discloses a computer readable storage medium configured to store a computer program; wherein the computer program is executed by a processor to implement the steps of the relational time-series data processing method disclosed above.

[0042] As can be seen, this application provides a relational time-series data processing method, including: determining the data source code corresponding to the current data source, and obtaining a standard format data file from the file directory corresponding to the data source code; obtaining a variable value corresponding to the data block name according to the data block name corresponding to the data block currently traversed in the standard format data file, and using the variable value to replace the corresponding string in the target expression to obtain a first replaced expression; determining whether the first replaced expression has a custom processing function; if the first replaced expression has the custom processing function, executing the custom processing function to obtain a corresponding value, and using the value to replace the custom processing function in the first replaced expression to obtain a second replaced expression; generating a corresponding SQL statement based on the calculation result obtained by executing the second replaced expression and other field information in a pre-built configuration table, and inserting the corresponding data in the standard format data file into the target database through the SQL statement. Therefore, this application obtains the standard format data file in the corresponding file directory through the data source code corresponding to the data source, and then inserts the corresponding data in the standard format data file into the target database through the generated SQL statement. In other words, the technical solution of this application can solve the problem that relational time series data has poor adaptability during implementation and delivery and cannot be separated from the source code and R&D participation mode, improve delivery capability and processing efficiency, increase the reusability of functional modules, and shorten the delivery cycle of new data sources. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a relational time-series data processing method disclosed in this application;

[0045] Figure 2 This is a schematic diagram of a specific relational time-series data processing framework disclosed in this application;

[0046] Figure 3 This is a flowchart of a specific relational time-series data processing method disclosed in this application;

[0047] Figure 4 This is a schematic diagram of the structure of a relational time-series data processing device disclosed in this application;

[0048] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, while various commercial and open-source data processing tools are available on the market, several issues arise when choosing to use them. For example, they are inefficient with large datasets, failing to meet latency requirements. Many products can quickly build data processing frameworks and provide basic functionality, but with large datasets, they cannot be deployed in a distributed manner, cannot improve performance, and cannot meet the timeliness requirements of data processing. They also lack flexibility, only meeting specific use cases. In the three stages of data processing—extraction (collection), transformation (data meaning-based processing), and loading (insertion into the database)—each stage presents numerous scenarios requiring handling. Although these products offer some openness, most do not support low-code development during the delivery phase. Market-available products are based on industry knowledge and data processing experience, forming fixed data transformation functions. Adapting to specific needs requires complex secondary development with high skill requirements. Therefore, this application provides a relational time-series data processing solution that addresses the poor adaptability of relational time-series data during implementation and delivery, its reliance on source code and R&D involvement, improves delivery capabilities and processing efficiency, increases the reusability of functional modules, and shortens the delivery cycle for new data sources.

[0051] This invention discloses a relational time-series data processing method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0052] Step S11: Determine the data source code corresponding to the current data source, and obtain the standard format data file from the file directory corresponding to the data source code.

[0053] It's important to note that data processing is fundamental to IT systems, especially monitoring systems, which handle large amounts of relational time-series data. This type of data is characterized by the row and column concepts found in relational models, with field values ​​typically being strings, numbers, or timestamps. Furthermore, this relational data is characterized by its large volume and strong regularity. Relational data refers to a set of data with a fixed model, usually using columns to represent the attributes of the data model and rows to represent attribute data that changes over time or with managed entities. Time-series data is a combination of data recorded in chronological order for a unified indicator. Time-series data can be period data or point-in-time data, requiring comparability. The data automatically collected by the operations and maintenance system is a typical example of relational time-series data. By building year-on-year, month-on-month, or predictive models using this data, it can be used to discover the operational status of monitored objects, such as their operational status and efficiency. Common practice is to programmatically split the collected data according to its meaning and insert it into relational data tables using SQL statements. However, handling scenarios such as data from different manufacturers, different models from the same manufacturer, and different times for the same model, while improving code reusability and implementation usability, presents significant challenges.

[0054] Understandably, to unify the management of object data, data table structures are generally set according to business types to achieve normalized configuration. That is, data of the same type should be stored in the same data table. For example, the switch table manages equipment from different manufacturers such as Cisco, Huawei, H3C, and Inspur. These manufacturers may define different variables for the same business field. For instance, the traffic of inbound and outbound ports may correspond to different variables from different manufacturers. The data table structure is shown in Table 1.

[0055] Table 1

[0056]

[0057]

[0058] In this embodiment, a data source code is assigned to each data source with different variables or expressions. The data source code must be consistent with the data block name and variable expression. That is, the data source code, as a key parameter, must correspond to a specific directory. The data file in this directory contains the required data block name and variable name. The data file in this directory can use the configured corresponding information. The data source code distinguishes different data sources by configuring the directory where the standard format data file is located. Different data sources are normalized through variable expression configuration. Data is extracted from the data source with the assigned data source code to obtain the corresponding raw format data file. The raw format data file is converted to a standard format data file using a pre-determined format conversion program. The standard format data file is then stored in the file directory corresponding to the data source code. In other words, the standard format data file is a file obtained by pre-converting the raw format data file extracted from the data source corresponding to the data source code, and this standard format data file is stored in the fixed file directory corresponding to the data source code. Therefore, by determining the data source code corresponding to the current data source, the standard format data file can be obtained from the file directory corresponding to the data source code. The data structure template of the standard format data file is shown below:

[0059] {BLOCK}blockname1 / / This flag starts a new block with {BLOCK} as the keyword, followed by the name.

[0060] {EXTEND} / / Extended variable information for this data block, not the original collected information, added during the standardization process.

[0061] STARTDATE|PERIOD|NENAME|REGION / / Extended variable names, separated by '|'

[0062] 2021-11-11|5|vm2021|JN / / Values ​​corresponding to extended variables, separated by '|'

[0063] {ENDEXTEND} / / End marker for extended variable information, usually two lines

[0064] {ROW} / / Data start marker

[0065] COL1|COL2|COL3|… / / Column names, separated by '|'

[0066] 2|3|4|… / / Values ​​for each row and column, which can have one or more rows.

[0067] 7|8|9|…

[0068] {ENDROW} / / End of data marker

[0069] {ENDBLOCK}

[0070] {BLOCK}blockname2

[0071] …

[0072] {ENDBLOCK}

[0073] It should be noted that the process of converting raw format data files to standard format data files may also include: filtering the raw format data files based on filename keywords to obtain filtered files; and using a pre-determined format conversion program to convert the filtered files into standard format data files. For example, raw data (rawfile) is extracted according to data source requirements, and then a format conversion program is used to convert the rawfile into a standard format file (stdfile). The format conversion program can be determined through a configuration file, i.e., filtering filenames containing a certain keyword. Filenames are marked according to the format, and filenames can be manually specified during the extraction process. Files matching these filename keywords are processed using a specific format conversion program, which is pre-developed. If the extracted format is special, a custom format conversion program needs to be customized. The relevant parameter configuration can be: {filename="*-vmcpu-*"; tranprog="snmp2std"; blockname="vmcpu"}. During the format conversion process, set the blockname (i.e., the data block name) as required for use when retrieving variables. Configure it in Table 1 above according to the corresponding formula obtained from the manufacturer or other authoritative sources.

[0074] Furthermore, for data sources with large volumes, a multi-server model can be used for deployment, or a dedicated directory can be established for data that requires priority processing. This allows for prioritizing specific data and ensuring timely data processing. It also supports the normalization management of business data models through expressions. For example, the correspondence between the target business data model and the data source can be configured, avoiding the need to modify the source code when changes occur. The target business data model may include, but is not limited to, tables and fields, while the data source may include, but is not limited to, data blocks and expressions. The expressions can support low-code descriptions, where standardized business logic is abstracted into fixed components, and the parts requiring customization based on customer needs are separated and processed using code. The tables and data blocks can have a many-to-many relationship, meaning the business data model can adapt to multiple data sources, improving the reusability of the business data model and shortening the delivery and implementation cycle.

[0075] Step S12: Obtain the variable value corresponding to the data block name according to the data block name of the data block currently traversed in the standard format data file, and use the variable value to replace the corresponding string in the target expression to obtain the first replaced expression.

[0076] In this embodiment, after obtaining the standard format data file from the file directory corresponding to the data source encoding, the variable value corresponding to the data block name is obtained according to the data block name corresponding to the data block currently traversed in the standard format data file. Then, the variable value is used to replace the corresponding string in the target expression to obtain the first replaced expression. The target expression is the target expression that converts the variable into the value required by the corresponding field by constructing it through basic operators. It is understandable that a target expression is formed by basic operators such as addition, subtraction, multiplication, division, and parentheses. This target expression can convert relevant variables into the numerical values ​​required by the corresponding fields. Furthermore, if a user-defined function is embedded in the expression, the user-defined function can also process the relevant variables into the numerical values ​​required by the corresponding fields. That is, the original variables are calculated using mixed arithmetic operations to obtain the corresponding numerical values. User-defined functions can also be specified for special requirements, such as a division function with a non-zero denominator. The user-defined function is calculated using simple lexical analysis based on compiler principles. Then, the user-defined function is executed, and the execution result is put into the original expression. Finally, the mixed arithmetic operations are performed uniformly to obtain the corresponding calculation result. For example, if the expression is (a+b) / c, and the variable values ​​are a=2, b=6, and c=4, then the replaced expression is (2+6) / 4, and the calculated numerical value is 2. During delivery and implementation, by writing functions to extend variables and process them into patterns that require numerical values, we can improve the ability to adapt to different project customizations. Moreover, we can perform distributed processing to meet the low latency requirements when dealing with large volumes. More importantly, we can meet the requirements of more complex data meaning processing through low-skill, low-code technology during delivery and implementation, and achieve the need for data to be unified in terms of business meaning.

[0077] It's important to note that during implementation, on-site personnel can modify expressions or implement special functions by writing low-code, based on their understanding of the business requirements. Low-code can use the same development language as the R&D version, which can be either interpreted or compiled, such as Perl. The way to make the low-code available in the development language is to place keywords like "import" or "include" at the beginning of the program. Whether interpreted or compiled, the low-code part simply needs to be placed in the specified directory. This is relatively easy to implement and has low technical requirements. For example, if the program name of the expression is exe-exp and the low-code extension program name is ext-program, then "import ext-program" or "include ext-program" can be used at the beginning of the exe-exp program. In other words, different keywords can be chosen depending on the development language. If strings, numbers, or even other functions reserved by the development language are needed, they can be encapsulated as custom functions to achieve special data processing in low-code programming. This addresses the need for special processing based on raw data, reducing the involvement of R&D personnel.

[0078] Step S13: Determine whether the first replaced expression has a custom processing function.

[0079] In this embodiment, after replacing the corresponding string in the target expression with the variable value to obtain the first replaced expression, it is determined whether the first replaced expression contains a custom processing function. Specifically, each string in the first replaced expression is traversed to determine whether a preset leading character exists in the first replaced expression, and a corresponding judgment result is obtained; based on the judgment result, it is determined whether the first replaced expression contains a custom processing function. If the judgment result indicates that the first replaced expression contains the preset leading character, then it is determined that the first replaced expression contains the custom processing function; if the judgment result indicates that the first replaced expression does not contain the preset leading character, then it is determined that the first replaced expression does not contain the custom processing function. It is understood that, in order to distinguish it from reserved functions and variables in the development language, the custom special processing function is required to meet the requirement of starting with "&", having "()" and a parameter string, such as &GetHostName(Hostid). If it is necessary to use reserved functions in the development language, such as string processing functions, it can be done by redefining the custom function for convenient and unified processing.

[0080] Step S14: If the first replaced expression contains the custom processing function, execute the custom processing function to obtain the corresponding value, and use the value to replace the custom processing function in the first replaced expression to obtain the second replaced expression.

[0081] In this embodiment, when determining whether the first replaced expression has a custom processing function, if the first replaced expression has the custom processing function, the custom processing function is executed to obtain the corresponding value, and the value is used to replace the custom processing function in the first replaced expression to obtain the second replaced expression.

[0082] For example, a complete expression configuration is shown in the following string:

[0083] var1-(var2+var3) / var4+&GetPercent(var5);

[0084] The values ​​of each variable can be set as follows: var1 = 5; var2 = 4; var3 = 2; var4 = 3; var5 = 6;

[0085] The expression processing procedure can be as follows: During data processing, the value of the blockname variable is obtained according to the process, and the data that meets the conditions is filtered from the configuration table based on the value of the blockname variable. The expression of each row of data that meets the conditions is processed accordingly: First, the variable value pair in blockname, i.e., variable name and variable value, is used to replace the string in the expression. The regular expression used for replacement has word independence and will not replace part of a string as a variable value. For example, the string after the expression replacement is "5-(4+2) / 3+&GetPercent(6)". The replaced string is traversed character by character to determine whether the preamble character "&" of the custom function exists in the replaced string. If it exists, the complete expression of the function is extracted. If it does not exist, the process ends. The embedded statement of the development language is called to execute the function on the processed string to obtain the result.

[0086] Step S15: Generate a corresponding SQL statement based on the calculation result obtained by executing the second replaced expression and other field information in the pre-built configuration table, and insert the corresponding data in the standard format data file into the target database through the SQL statement.

[0087] It is understandable that the results obtained from the execution function and the information from other fields in the configuration table are combined to form the SQL statement used when inserting data into the database. The corresponding data in the standard format data file is then inserted into the target database through the SQL statement. For example, the configuration information of the data table is used to form the statement: insert into tablename(colname)values(colexp).

[0088] In this embodiment, after inserting data into the target database, it can be determined whether the primary key in the target database is duplicated. If the primary key in the target database is duplicated, the current data insertion is determined to have failed, and a corresponding update operation is performed. If the primary key in the target database is not duplicated, the current data insertion is determined to have succeeded. This enables the automatic correction of primary key duplication errors during data entry, meaning that an update operation can be automatically performed based on the keyword when data insertion fails, preventing data loss. It should be noted that the configuration information of a data table must have at least one field as a keyword, which can be used as a keyword during insertion or for updates after insertion failure.

[0089] For example, according to the data processing ETL process, the relevant system architecture is as follows: Figure 2As shown, based on business analysis, standardized components are first established, while low-code capabilities are reserved for personalized parts to accommodate customized needs. In the data extraction stage, different technical protocols can be used to extract data from the data source to the local machine, forming raw format data files. This process preserves the original data values ​​without any changes, so a fixed data extraction or collection program can be formed based on the data source's technical protocol. The data transformation stage is the core of data processing, and low-code support also occurs in this stage. Depending on business needs, this can be further subdivided into: conversion from raw data format to standard data format (i.e., for the convenience of subsequent low-code function processing, converting source data of different formats to standard format data); and function processing based on standard format data files to derive new values ​​or variables. This system enables the derivation of new variables or values ​​through mixed arithmetic expressions. It supports deriving new variables or values ​​from existing numerical variables using arithmetic operations, and also allows for more complex variable derivation through embedded functions. It supports value mapping, special numerical operations, string processing, constants, and allows for custom processing functions according to the given development language syntax, as well as generating SQL statements for database insertion based on the database platform's SQL language requirements. In the data loading phase, data is transformed into SQL statements, which are then connected to and executed to insert data into the corresponding database. It can also determine if primary keys are duplicated, and update them if found. Based on the characteristics of relational time-series data, the data transformation and loading phases support configurability and low-code support. This means that heterogeneous data can be unified into the business data model through configuration, enabling data loading. Furthermore, configuration allows for consistent data entry information from different data sources within the business data model, adapting to different manufacturers, different models from the same manufacturer, and different versions of the same model, resulting in different formulas.

[0090] In practical project applications, it can be found that low-code implementation can reduce the workload of R&D personnel from 85% to 35%. Moreover, after adopting low-code customization, R&D personnel mainly participate in the development of new format conversion, new protocol interfaces, etc. The technical solution of this application changes the practice of implementing data acquisition, format conversion and data storage in a large program, realizes the same data processing function, improves the reusability of functional modules, thereby reducing the total R&D cost by 23%. In the multi-manufacturer and multi-model management of the same technical protocol and the same file format, the delivery speed is significantly improved. The mode of R&D personnel participating in coding data acquisition and format conversion is transformed into reusing modules of the same technical type and achieving unification through configuration map. The new management time is reduced from 5 working days for a single device to only 1 or half a working day.

[0091] As can be seen, in this embodiment, a standard format data file is obtained in the corresponding file directory through the data source code corresponding to the data source, and then the corresponding data in the standard format data file is inserted into the target database through the generated SQL statement. In other words, the technical solution of this application can solve the problem that relational time series data has poor adaptability during implementation and delivery and cannot be separated from the source code and R&D participation mode, improve delivery capability and processing efficiency, increase the reusability of functional modules, and shorten the delivery cycle of new data sources.

[0092] For example, such as Figure 3 As shown, the corresponding file directory is located according to the data source code ruleid. The standard format data file is obtained from the file directory, and then the file name list filelist is obtained from the data file. The configuration information maplist corresponding to the data source code ruleid is obtained from the pre-built configuration table. The data is grouped according to the data table name tablename and data block name blockname, and a field name collename is formed, which is the value pair string of the variable expression colexp. Then, the data block name is used as the first-level key and the data table name tablename is used as the second-level key, and it is stored as a hash array. The process checks if the filename list is empty. If it is not empty, it iterates through the contents of each file in the list and retrieves the value of the `blockname` variable in the currently accessed file. If the current configuration information `maplist{blockname}->{tablename}` is empty, it iterates through the next block. If the file ends, it checks if the filename list is empty again. If `maplist{blockname}->{tablename}` is not empty or the file has not ended, it replaces `colname` (the placeholder string for the variable in `colexp`) with the variable from the current block, splits `colname` to obtain each individual `colexp`, iterates through the variable expression `colexp` to check if a custom processing function exists. If it exists, it executes the custom processing function, obtains the corresponding value, and uses this value to replace `colexp`. If it does not exist, it directly executes `colexp` to obtain the final result and combines it with the corresponding `colname` to form an SQL statement.

[0093] Accordingly, embodiments of this application also disclose a relational time-series data processing apparatus, see [link to relevant documentation]. Figure 4 As shown, the device includes:

[0094] The data source encoding determination module 11 is used to determine the data source encoding corresponding to the current data source.

[0095] The file acquisition module 12 is used to acquire a standard format data file from the file directory corresponding to the data source encoding;

[0096] The variable value acquisition module 13 is used to acquire the variable value corresponding to the data block name based on the data block name corresponding to the data block in the currently traversed standard format data file;

[0097] String replacement module 14 is used to replace the corresponding string in the target expression with the variable value to obtain the first replaced expression;

[0098] Expression judgment module 15 is used to determine whether the first replaced expression has a custom processing function;

[0099] The function execution module 16 is used to execute the custom processing function to obtain the corresponding value when the first replaced expression contains the custom processing function, and use the value to replace the custom processing function in the first replaced expression to obtain the second replaced expression;

[0100] The SQL statement generation module 17 is used to generate corresponding SQL statements based on the calculation results obtained by executing the second replaced expression and other field information in the pre-built configuration table, and to insert the corresponding data in the standard format data file into the target database through the SQL statements.

[0101] As can be seen from the above, in this embodiment of the application, a standard format data file is obtained in the corresponding file directory through the data source code corresponding to the data source, and then the corresponding data in the standard format data file is inserted into the target database through the generated SQL statement. In other words, the technical solution of this application can solve the problem that relational time series data has poor adaptability during implementation and delivery and cannot be separated from the source code and R&D participation mode, improve delivery capability and processing efficiency, increase the reusability of functional modules, and shorten the delivery cycle of new data sources.

[0102] In some specific embodiments, the relational time-series data processing apparatus may further include:

[0103] The data extraction module is used to extract data from a data source with a pre-defined data source encoding to obtain the corresponding raw format data file;

[0104] The format conversion module is used to convert the original format data file into a standard format data file using a pre-determined format conversion program;

[0105] The file saving module is used to store the standard format data file in the file directory corresponding to the data source encoding.

[0106] In some specific embodiments, the format conversion module may specifically include:

[0107] The file filtering unit is used to filter the original format data file based on the file name keywords of the original format data file to obtain the filtered file;

[0108] The format conversion unit is used to convert the filtered file into a standard format data file using a predetermined format conversion program.

[0109] In some specific embodiments, the expression judgment module 15 may specifically include:

[0110] The judgment unit is used to traverse each string in the first replaced expression to determine whether there is a preset leading character in the first replaced expression and obtain the corresponding judgment result;

[0111] The determining unit is used to determine whether a custom processing function exists in the first replaced expression based on the judgment result.

[0112] In some specific embodiments, the determining unit may specifically include:

[0113] The first determination subunit is used to determine that the first replaced expression contains the preset leading character if the determination result indicates that the first replaced expression contains the custom processing function.

[0114] The second determination subunit is used to determine that the first replaced expression does not contain the preset leading character if the determination result indicates that the first replaced expression does not contain the custom processing function.

[0115] In some specific embodiments, the relational time-series data processing apparatus may further include:

[0116] An expression building module is used to construct the target expression that converts variables into the numerical values ​​required by the corresponding fields using basic operators.

[0117] In some specific embodiments, after inserting the corresponding data from the standard format data file into the target database using the SQL statement, the process may further include:

[0118] The primary key determination module is used to determine whether the primary key in the target database is duplicated;

[0119] The first determination module is used to determine that the current data insertion has failed if the primary key in the target database is duplicated, and to perform the corresponding update operation.

[0120] The second determination module is used to determine that the current data insertion was successful if the primary key in the target database is not duplicated.

[0121] Furthermore, embodiments of this application also provide an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0122] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the relational time-series data processing method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0123] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0124] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0125] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the relational time-series data processing method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0126] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the steps of the relational time-series data processing method disclosed in any of the foregoing embodiments.

[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0128] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] The foregoing has provided a detailed description of a relational time-series data processing method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A relational time-series data processing method, characterized in that, include: Determine the data source code corresponding to the current data source, and obtain the standard format data file from the file directory corresponding to the data source code; Based on the data block name corresponding to the data block in the currently traversed standard format data file, obtain the variable value corresponding to the data block name, and use the variable value to replace the corresponding string in the target expression to obtain the first replaced expression; Determine whether the first replaced expression contains a custom processing function; If the first replaced expression contains the custom processing function, then the custom processing function is executed to obtain the corresponding value, and the value is used to replace the custom processing function in the first replaced expression to obtain the second replaced expression; Based on the calculation result obtained by executing the second replacement expression and other field information in the pre-built configuration table, a corresponding SQL statement is generated, and the corresponding data in the standard format data file is inserted into the target database through the SQL statement.

2. The relational time-series data processing method according to claim 1, characterized in that, Also includes: Extract data from a data source with pre-defined data source encoding to obtain the corresponding raw format data file; The original format data file is converted into a standard format data file using a pre-determined format conversion program; Store the standard format data file in the file directory corresponding to the data source encoding.

3. The relational time-series data processing method according to claim 2, characterized in that, The step of converting the original format data file into a standard format data file using a pre-determined format conversion program includes: The original format data file is filtered based on the file name keywords to obtain the filtered file; The filtered file is converted into a standard format data file using a pre-defined format conversion program.

4. The relational time-series data processing method according to claim 1, characterized in that, The step of determining whether the first replaced expression has a custom processing function includes: Iterate through each string in the first replaced expression to determine whether there is a preset leading character in the first replaced expression and obtain the corresponding judgment result; Based on the judgment result, it is determined whether the first replaced expression contains a custom processing function.

5. The relational time-series data processing method according to claim 4, characterized in that, Determining whether a custom processing function exists in the first replaced expression based on the judgment result includes: If the determination result indicates that the first replaced expression contains the preset leading character, then it is determined that the first replaced expression contains the custom processing function; If the determination result indicates that the first replaced expression does not contain the preset leading character, then it is determined that the first replaced expression does not contain the custom processing function.

6. The relational time-series data processing method according to claim 1, characterized in that, Also includes: The target expression is constructed using basic operators to convert variables into the numerical values ​​required for the corresponding fields.

7. The relational time-series data processing method according to any one of claims 1 to 6, characterized in that, After inserting the corresponding data from the standard format data file into the target database using the SQL statement, the process further includes: Determine whether the primary key in the target database is duplicated; If the primary key in the target database is duplicated, the current data insertion is determined to have failed, and the corresponding update operation is performed. If the primary key in the target database is unique, the current data insertion is considered successful.

8. A relational time-series data processing device, characterized in that, include: The data source encoding determination module is used to determine the data source encoding corresponding to the current data source; The file acquisition module is used to acquire standard format data files from the file directory corresponding to the data source encoding. The variable value acquisition module is used to acquire the variable value corresponding to the data block name based on the data block name corresponding to the data block in the currently traversed standard format data file; The string replacement module is used to replace the corresponding string in the target expression with the variable value to obtain the first replaced expression; The expression judgment module is used to determine whether the first replaced expression contains a custom processing function; The function execution module is used to execute the custom processing function to obtain the corresponding value when the first replaced expression contains the custom processing function, and use the value to replace the custom processing function in the first replaced expression to obtain the second replaced expression; The SQL statement generation module is used to generate corresponding SQL statements based on the calculation results obtained by executing the second replaced expression and other field information in the pre-built configuration table, and to insert the corresponding data in the standard format data file into the target database through the SQL statements.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the relational time-series data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of the relational time-series data processing method as described in any one of claims 1 to 7.

Images