Method and system for constructing multi-source and multi-scale sample database for sodium carbonate production

Abstract

The invention relates to the technical field of soda ash chemical industry and big data, in particular to a multi-source multi-scale sample database building method and system for soda ash production, and the method comprises the steps: unifying the format of data related to soda ash production into a CSV lightweight format; identifying and correcting parameter abnormal data; automatically subtracting the flow accumulated value into an hour difference value; checking and processing the flow accumulated value re-metering condition of the instrument; on the basis of the current integral coefficient, dividing the parameter values taking 24 hours or 8 hours as the updating frequency into parameter values taking 1 hour as the updating frequency; and extracting and matching parameter description information and numerical values, forming a two-dimensional standard data format, and constructing a multi-source multi-scale sample database. According to the method, a two-dimensional standard data format is formed, a multi-source and multi-scale sample database is constructed, and multi-source and multi-scale data association is realized; and real-time updating of the sodium carbonate production sample database is realized.

Description

Technical Field

[0001] This invention relates to the fields of soda ash chemical industry and big data technology, specifically to a method and system for building a multi-source, multi-scale sample database for soda ash production. Background Technology

[0002] In the production of soda ash, optimizing production operations is a core requirement for reducing energy consumption and increasing profits. Intelligent optimization methods can effectively improve the economic efficiency of soda ash production, and constructing a precise proxy model is a key step in achieving this goal, with data being the core of establishing such a model. However, existing production data management suffers from several problems: data sources for process parameters, raw material consumption, energy use, and product output are scattered, recording formats are inconsistent, and multi-source data is difficult to integrate; actual collected data contains outliers, reducing data quality; cumulative flow values ​​from various instruments are inconvenient to use and require recounting; and data recording frequency scales are inconsistent, failing to fully extract effective information from low-frequency data. These problems severely restrict the accuracy of data-driven models, thus affecting the practical application effect of intelligent optimization systems. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a method and system for building a multi-source, multi-scale sample database for soda ash production that overcomes or at least partially solves the above problems.

[0004] In a first aspect, embodiments of the present invention provide a method for constructing a multi-source, multi-scale sample database for soda ash production, comprising: The data format involved was unified into the lightweight CSV format to obtain multi-source, multi-scale CSV files. Identify abnormal data that are outside the normal range of parameter values, and replace the abnormal data with the average value of the data within one hour before and after the abnormal data. The accumulated flow rate readings from the meter will be automatically adjusted to the hourly difference. Inspect and process cases where the instrument's cumulative flow value is remeasured, and correct the hourly difference in flow rate during the hour in which the remeasurement occurred; Based on the current integral coefficient, parameter values ​​updated at a frequency of 24 hours or 8 hours are divided into parameter values ​​updated at a frequency of 1 hour; The parameter description information and corresponding parameter values ​​are extracted separately. The parameter description information is then precisely matched with the CSV data columns to form a two-dimensional standardized data format, and a multi-source, multi-scale sample database is constructed.

[0005] In one embodiment, the verification and processing of the remetering of the instrument flow accumulation value includes: Compare the cumulative traffic value of the current hour with the difference of the previous hour. If the cumulative traffic value of the current hour is less than 1 / 2 of the difference of the previous hour, the cumulative traffic value of the next 1 / 2 hour will be remeasured by default, and the difference of the current hour will be corrected to twice the remeasured cumulative traffic value. Otherwise, the cumulative traffic value of the previous 1 / 2 hour will be remeasured by default, and the difference of the current hour will be corrected to once the remeasured cumulative traffic value.

[0006] In one embodiment, dividing parameter values ​​updated at a frequency of 24 hours or 8 hours into parameter values ​​updated at a frequency of 1 hour based on the current integral coefficient includes: Read the updated current of the involved devices at a minute-level frequency; Convert each current data file from xls format to xlsx format; The time column of each current data file is split into a date column and a time column, and the device running time is calculated in minutes based on the time. Delete negative current or low current fluctuation values ​​less than 1mA that exist during the downtime of each device and update them to 0; Interpolation of missing integer points and integer currents based on the CubicSpline method; Perform effective time integration on the equipment operating current within each hourly period; form a coefficient matrix W, and W matrix elements. The integral formula is shown in formula (1): (1) in This refers to the effective operating time of the equipment. It is the corresponding device.

[0007] Based on W matrix elements This forms a coefficient matrix M, and the elements of matrix M... The integral formula is shown in formula (2): (2) Based on M matrix elements Calculate the integral coefficient elements of the current for each hour. The calculation formula is shown in formula (3). (3) Based on the current integral coefficient, the parameter values ​​are divided with an update frequency of 24 hours or 8 hours, and the calculation formula is shown in formula (4): (4) in These are parameter values ​​that are updated every 24 hours or 8 hours. The parameter value is updated every hour.

[0008] In one embodiment, the extraction of parameter description information includes: Iterate through and read multi-source, multi-scale CSV files; Iterate through each row of the CSV file, and use the set() function to find the intersection of the row text set and the shift set, thereby determining and storing the row index of the parameter description information range; Iterate through each column of the CSV file, determine the starting position of the column index for parameter description information by the "time" marker, and determine the ending position of the column index by the text in the last column; Based on the row index of the parameter description information, determine the scope of the inclusion relationship between different rows of parameter description information, and fill in the missing parameter description information row by row within the scope of the inclusion relationship; Vertically merge and extract all description fields of the same parameter to obtain parameter description information.

[0009] In one embodiment, after establishing a multi-source, multi-scale sample database for soda ash production, the method further includes: Establish a data channel between the instrument data system and the multi-source, multi-scale sample database; when the data in the instrument data system is updated, update the multi-source, multi-scale sample database in real time.

[0010] Secondly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned method for constructing a multi-source, multi-scale sample database for soda ash production.

[0011] Thirdly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the aforementioned method for constructing a multi-source, multi-scale sample database for soda ash production.

[0012] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention provides a method and system for constructing a multi-source, multi-scale sample database for soda ash production. For the data involved, the data format related to soda ash production is unified into a lightweight CSV format, reducing storage space and facilitating program calls; abnormal parameter data is identified and corrected, improving the quality of data related to the soda ash production process, consumption, and products; the re-metering of instrument flow accumulation values ​​is checked and processed, and the difference in flow accumulation values ​​is converted into hourly differences to obtain hourly flow usage information; based on the current integral coefficient, parameter values ​​with a daily or shift (8-hour) update frequency are divided into parameter values ​​with an hourly update frequency, fully utilizing low-frequency data information. This approach helps construct a high-quality dataset for soda ash production, thereby improving the effectiveness of intelligent optimization of soda ash production. It extracts and matches parameter descriptions and values ​​to form a two-dimensional standardized data format, constructing a multi-source, multi-scale sample database containing soda ash production process parameters, raw material consumption, energy use, and product analysis data. This enables multi-source, multi-scale data association, laying the foundation for subsequent intelligent optimization of soda ash production operations, thereby reducing water, electricity, and steam consumption and increasing enterprise profits. Furthermore, a data transmission channel is established between the soda ash production sample database and the soda ash data system, enabling real-time updates of the soda ash production sample database and laying the foundation for real-time updates and optimization of the soda ash production data-driven model.

[0013] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a simplified diagram of the soda ash calcination workshop process in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the calculation of the current integration coefficient in an embodiment of the present invention; Figure 3 Here is the flowchart for step S6; Figure 4 This is a flowchart illustrating the parameter description information extraction process in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the extraction of parameter description information in an embodiment of the present invention; Figure 6 This is a schematic diagram of the sample database framework in an embodiment of the present invention. Detailed Implementation

[0016] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0017] This invention provides a method and system for constructing a multi-source, multi-scale sample database for soda ash production. The method includes the following steps: S1. Unify the data format involved in soda ash production into the lightweight CSV format to obtain multi-source, multi-scale CSV files; S2. Identify abnormal data that are outside the normal range of parameter values, and replace the abnormal data with the average value of the hour before and after the abnormal data. S3. Automatically differ the accumulated flow rate readings from the meter by the hourly difference value; S4. Inspect and process the re-metering of the instrument's cumulative flow value, and correct the hourly difference in flow rate during the hour in which the re-metering occurred; S5. Based on the current integral coefficient, the parameter values ​​updated at a frequency of 24 hours or 8 hours are divided into parameter values ​​updated at a frequency of 1 hour; S6. Extract the parameter description information and corresponding parameter values ​​respectively. Accurately match the parameter description information with the CSV data columns to form a two-dimensional standardized data format and construct a multi-source, multi-scale sample database.

[0018] In this embodiment, the calcination process of soda ash is used as an example for illustration. A simplified diagram of the soda ash calcination process is shown below. Figure 1 As shown, the data involved includes the following files: Daily Operation Report 1.xlsx, Daily Operation Report 2.xlsx, Daily Operation Report 3.xlsx, MES Consumption Data.xlsx, Calcination Electricity Consumption Data.xlsx, Daily Analysis Data.xlsx, Daily Analysis Report.xlsx, and Shift Score Data.xlsx. Data is stored daily, with each type of data stored in a separate folder. The data format has been standardized to the lightweight CSV format to reduce storage space and facilitate program access.

[0019] In step S2, the normal range of parameter values ​​is defined according to actual process requirements. If a parameter value is lower than 80% of the minimum normal range or higher than 120% of the maximum normal range, it is considered abnormal data. Replacing the abnormal data with the average value within one hour before and after the abnormal data is based on the short-term stability of soda ash production data. Using the "typical value" of nearby normal data to smooth out abnormal "spurts" can better preserve the local trend characteristics of the data, while avoiding data loss caused by deleting abnormal data, thus maintaining the continuity and integrity of the data sequence.

[0020] Soft water, circulating water, condensate, and flash steam are used throughout the production process of soda ash. Controlling their usage is crucial for the quality of soda ash products and for energy conservation and efficiency improvement. The cumulative values ​​obtained by instruments cannot measure the actual consumption of soft water, circulating water, condensate, and flash steam within a specific time period and cannot be used directly. Step S3 automatically calculates the hourly difference between the cumulative flow values ​​read from the instruments, which facilitates subsequent optimization of the process operation by combining it with other parameter data.

[0021] If the cumulative flow value of the instrument is remeasured, the hourly difference calculated in step S3 will have a large error at the remeasurement time point if no correction is made. Therefore, step S4 is required for correction.

[0022] The specific steps in S4 above will be explained below.

[0023] Furthermore, in step S4 above, the verification and processing of the re-metering of the instrument flow accumulation value is achieved through the following scheme: The system determines the relationship between the meter's cumulative flow value and the difference from the previous hour. If the meter's cumulative flow value is less than half of the difference from the previous hour, the meter's cumulative flow value for the latter half of the hour is remeasured by default, and the difference for that hour is twice the remeasured cumulative flow value. Otherwise, the meter's cumulative flow value for the first half of the hour is remeasured by default, and the difference for that hour is once the remeasured cumulative flow value.

[0024] In the existing technology, parameters for soda ash production are recorded in units of days or shifts (8 hours), which lacks flexibility and cannot be called up simultaneously with most parameters recorded in units of hours. Step S4 of the present invention fully mines the effective information of low-frequency data through the current integral coefficient, which helps to build a high-quality dataset for soda ash production, thereby facilitating the improvement of the effect of intelligent optimization of soda ash production.

[0025] The specific steps in S5 above are explained below.

[0026] Furthermore, such as Figure 2As shown, the following scheme converts parameter values ​​updated daily or per shift (8 hours) into parameter values ​​updated hourly, based on the current integral coefficient: S51. Read the minute-level frequency update current data file of the involved equipment; S52. If the current data file of the equipment involved is in xls format, convert it to xlsx format; S53. Split the time column of each current data file into a date column and a time column, and calculate the equipment running time within each hour based on the time. S54. Delete the negative current or low current fluctuation values ​​less than 1mA during the downtime of each device and replace them with 0. S55. Interpolation of missing integer points and integer currents based on the CubicSpline method; S56. Perform effective time integration on the equipment operating current within each hourly time period; S57. After processing through steps S51-S56, all current data files involving the equipment are converted into a coefficient matrix W, and the elements of the W matrix are... The integral formula is shown in formula (1): (1) in This refers to the effective operating time of the equipment within that hour, in minutes. It is the corresponding device.

[0027] S58, Based on W matrix elements This forms a coefficient matrix M, and the elements of matrix M... The integral formula is shown in formula (2): (2) S59, Based on M matrix elements Calculate the integral coefficient elements of the current for each hour. The calculation formula is shown in formula (3): (3) S510. Based on the current integral coefficient, the parameter values ​​are divided with an update frequency of 24 hours or 8 hours. The calculation formula is shown in formula (4): (4) in These are parameter values ​​that are updated every 24 hours or 8 hours. The parameter value is updated every hour.

[0028] In this embodiment, taking the output and electricity consumption parameters of the calcination process per shift as examples, the equipment involved in the hourly output current integral coefficient is shown in Table 1, and the equipment involved in the hourly electricity consumption current integral coefficient is shown in Table 2. The flowchart for calculating the current integral coefficient is as follows. Figure 2 As shown.

[0029] The current data of the equipment involved is updated every 2-3 minutes, and most of the hourly current data is missing. Therefore, the missing hourly and hourly current data are interpolated using the CubicSpline method to lay the foundation for subsequent effective time integration of the current.

[0030]

[0031] In the prior art, the descriptive information of the same parameter is distributed across multiple rows and columns, and the descriptive information of different parameters is distributed in a variety of ways. Conventional extraction methods are prone to missing descriptive information. This invention efficiently extracts parameter descriptive information through step S6, and accurately matches the parameter descriptive information with CSV data columns to form a two-dimensional standardized data format, which is convenient for constructing a multi-source, multi-scale sample database.

[0032] The specific steps in S6 above will be explained below.

[0033] Furthermore, in step S6 above, the parameter description information is extracted using the following scheme.

[0034] Iterate through and read the first CSV file in the data folders involved in the calcination process; Iterate through each row of the CSV file, and use the set() function to find the intersection of the row text set and the shift [shift A, shift B, shift C, shift D, day shift, afternoon shift, evening shift] set, thereby determining and storing the row index of the parameter description information range; Automatically traverse each column of the CSV file, determine the starting position of the column index for parameter description information by the "time" marker, and determine the ending position of the column index by judging the text in the last column; Based on the row index of the parameter description information, determine the scope of the inclusion relationship between different rows of parameter description information, and fill in the missing parameter description information row by row within the scope of the inclusion relationship; Vertically merge and extract all description fields of the same parameter to obtain parameter description information.

[0035] like Figure 3 , Figure 4As shown, the process iterates through the first CSV file in the data folder related to the calcination process. For each CSV file, an index check is performed first. Each row is traversed to search for a specific shift. The first row containing elements from the set [Shift A, Shift B, Shift C, Shift D, Day Shift, Afternoon Shift, Night Shift] is the first row of parameter values. This allows us to determine the index range of the parameter description field and the parameter value row. The column indices of the parameter description field and the parameter value are consistent. Using the specific text marker "Time" that appears in all source data files, the starting position of the column index for the parameter description field and parameter value in each source data file is determined. The position of the last column of text is determined as the ending position of the column index. For example... Figure 5 As shown, the parameter description fields of Excel files from different sources are stored in rows and merged. The relationships between the rows are varied and uncertain, making it difficult to directly and in batches obtain all description field information for the same parameter. After converting the Excel file format to CSV format, the original merged cells are canceled, resulting in blank cells. The first row is filled with blank cells according to the "forward fill" principle. The remaining rows are filled with blank cells in the range of cells with the same description field in the previous row. After filling, the parameter description fields of each row in the same column are merged and extracted, which is the parameter description information of the same parameter, and it corresponds one-to-one with the parameter value in that column.

[0036] Data from the same source is processed by sequentially traversing the CSV files for each day according to date, and the parameter values ​​are extracted based on the row and column indices of the parameter values ​​determined above.

[0037] like Figure 6 As shown, the extracted parameter descriptions from each CSV file are arranged sequentially as column names, with their source indicated. The leftmost two columns are date and time, with dates in days and times in hours, forming a two-dimensional standardized data framework. Parameter values ​​are filled into the corresponding parameter positions in the two-dimensional standardized data framework, with each row containing parameter values ​​updated hourly. It is important to note that this invention converts low-frequency parameter data into high-frequency data using the current integral coefficient, while retaining the original low-frequency data. The original low-frequency data, in units of days and shifts (8 hours), is stored in the first hour of the corresponding day or shift update. The parameters of the sample database are shown in Table 3. It is worth noting that the product analysis and team cost parameters in Table 3 retain their original frequencies and are not processed.

[0038] Table 3 Sample Database Parameters

[0039] Furthermore, the sample database is updated in real time through the following scheme: S7. Establish a data channel between the instrument data system and the multi-source, multi-scale sample database; S8. When the data in the instrument data system is updated, the multi-source, multi-scale sample database is updated in real time.

[0040] This invention establishes a data channel between the calcination workshop data system and the multi-source, multi-scale sample database based on the OPC-UA communication protocol. When the data system generates new or modified data, a data push mechanism is triggered; the sample database is updated in real time according to the processing rules of steps S1-S6 above.

[0041] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the aforementioned method and system for constructing a multi-source, multi-scale sample database for soda ash production.

[0042] This invention provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned method and system for constructing a multi-source, multi-scale sample database for soda ash production.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0044] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0046] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for constructing a multi-source, multi-scale sample database for soda ash production, characterized in that, Includes the following steps: The data format involved in soda ash production was unified into the lightweight CSV format, resulting in multi-source, multi-scale CSV files. Identify abnormal data that is outside the normal range of parameter values, and replace the abnormal data with the average value of the data within one hour before and after the abnormal data. The accumulated flow rate readings from the meter will be automatically adjusted to the hourly difference. Inspect and process cases of remeasurement of cumulative flow values, and correct hourly differences in flow rates during the hours in which remeasurement occurred; Based on the current integral coefficient, parameter values ​​updated at a frequency of 24 hours or 8 hours are divided into parameter values ​​updated at a frequency of 1 hour; Extract parameter description information and corresponding parameter values ​​respectively, accurately match parameter description information and parameter values ​​to form a two-dimensional standardized data format, and construct a multi-source multi-scale sample database; The method of dividing parameter values ​​updated at a frequency of 24 hours or 8 hours into parameter values ​​updated at a frequency of hours, based on the current integral coefficient, includes the following steps: Read the updated current of the involved devices at a minute-level frequency; Convert each current data file from xls format to xlsx format; The time column of each current data file is split into a date column and a time column, and the device running time is calculated in minutes based on the time. Delete negative current or low current fluctuation values ​​less than 1mA that exist during the downtime of each device and update them to 0; Interpolation of missing integer points and integer currents based on the CubicSpline method; Perform effective time integration on the equipment operating current within each hourly period; form a coefficient matrix W, and W matrix elements. The integral formula is shown in formula (1): (1) in This refers to the effective operating time of the equipment. It is the corresponding equipment; Based on W matrix elements This forms a coefficient matrix M, and the elements of matrix M... The integral formula is shown in formula (2): (2) Based on M matrix elements Calculate the integral coefficient elements of the current for each hour. The calculation formula is shown in formula (3). (3) Based on the current integral coefficient, the parameter values ​​are divided with an update frequency of 24 hours or 8 hours, and the calculation formula is shown in formula (4): (4) in These are parameter values ​​that are updated every 24 hours or 8 hours. The parameter value is updated every hour.

2. The method for constructing a multi-source, multi-scale sample database for soda ash production according to claim 1, characterized in that, The process of verifying and processing the re-metering of the instrument flow accumulation value includes: Compare the cumulative traffic value of the current hour with the difference of the previous hour. If the cumulative traffic value of the current hour is less than 1 / 2 of the difference of the previous hour, the cumulative traffic value of the next 1 / 2 hour will be remeasured by default, and the difference of the current hour will be corrected to twice the remeasured cumulative traffic value. Otherwise, the cumulative traffic value of the previous 1 / 2 hour will be remeasured by default, and the difference of the current hour will be corrected to once the remeasured cumulative traffic value.

3. The method for constructing a multi-source, multi-scale sample database for soda ash production according to claim 1, characterized in that, The extracted parameter description information includes: Iterate through and read multi-source, multi-scale CSV files; Iterate through each row of the CSV file, and use the set() function to find the intersection of the row text set and the shift set, thereby determining and storing the row index of the parameter description information range; Iterate through each column of the CSV file, determine the starting position of the column index for parameter description information by the "time" marker, and determine the ending position of the column index by the text in the last column; Based on the row index of the parameter description information, determine the scope of the inclusion relationship between different rows of parameter description information, and fill in the missing parameter description information row by row within the scope of the inclusion relationship; Vertically merge and extract all description fields of the same parameter to obtain parameter description information.

4. The method for constructing a multi-source, multi-scale sample database for soda ash production according to claim 1, characterized in that, After constructing a multi-source, multi-scale sample database, the following is also included: Establish a data channel between the instrument data system and the multi-source, multi-scale sample database; when the data in the instrument data system is updated, update the multi-source, multi-scale sample database in real time.

5. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for constructing a multi-source, multi-scale sample database for soda ash production as described in any one of claims 1 to 4.

6. A computer program product, characterized in that... The computer program product includes a computer program that, when executed by a processor, implements the method for constructing a multi-source, multi-scale sample database for soda ash production as described in any one of claims 1 to 4.

Images