Automatic data configuration management system and method, computer equipment and readable storage medium

Abstract

The invention discloses an automatic data configuration management system and method, computer equipment and a readable storage medium, and relates to the technical field of data processing.The automatic data configuration management system comprises an intelligent configuration engine, a dimension management system and a quality control system.The intelligent configuration engine recognizes the format and position of a sheet page through a content parser, and the dimension management system and the quality control system.The dimension management system is used for managing the sheet page; the block processor reads a data value, and the wide table generator converts the data value into a wide table format readable by a program; the dimension management system uses an empty dimension filling algorithm to match and combine cells, and the mapper matches rows or columns with index positions to define dimension values; the quality control system guarantees the data quality through a uniqueness verification module, a change tracing module and an exception blocking mechanism. The system integrates analysis, dimension adaptation and quality control functions, realizes automatic management of data configuration, and improves configuration efficiency and data reliability.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a data automated configuration management system, method, computer device, and readable storage medium. Background Art

[0002] In the fields of enterprise informatization and data management, tables, as widely used data carriers, their configuration management is a key link in data governance. Traditional data configuration management mostly relies on manual operations, and there are many pain points: First, the complex table format leads to low parsing efficiency, and different file formats are not unified, making it difficult to automate processing; second, the dimension definition depends on manual matching of rows or columns with the positions of indicators, and it is easy to cause dimension misalignment due to merged cell null values or position changes; third, there is a lack of a systematic verification mechanism for data quality, and it is difficult to quickly identify duplicate indicators, position changes, or incorrect data, resulting in deviations in subsequent data applications. Summary of the Invention

[0003] The purpose of the present invention is to provide a data automated configuration management system, method, computer device, and readable storage medium.

[0004] In a first aspect, an embodiment of the present invention provides a data automated configuration management system, including: An intelligent configuration engine, the intelligent configuration engine includes a content parser, a block processor, and a wide table generator; the content parser is used to determine the format and position of the data on the sheet page in the Excel file; the block processor is used to read the Excel data values according to the configuration of the content parser; the wide table generator is used to convert the indicator format into a wide table format suitable for program reading; A dimension management system, the dimension management system includes an empty dimension filling algorithm and a mapper; the empty dimension filling algorithm is used to adapt to the merged cells in Excel; the mapper is used to match the defined rows or columns with the indicator positions and define them as dimension values; A quality control system, the quality control system includes a uniqueness verification module, a change traceability module, and an exception blocking mechanism; the uniqueness verification module is used to uniquely identify indicators according to the file name, sheet name, indicator name, and position; the change traceability module is used to trigger an alarm prompt when the position of the CELL cell where the indicator is located changes; the exception blocking mechanism is used to prevent data from being stored in the database when a predefined error occurs.

[0005] In a possible implementation manner, the intelligent configuration engine further includes a Sheet name management unit, and the Sheet name management unit is used to perform a rename operation on the sheet names in the original Excel file to eliminate the impact of non-standard original sheet names on the consistency and standardization of data management.

[0006] In a possible implementation, the empty dimension filling algorithm of the dimension management system includes non-filling and backward filling for the index dimension filling method, which is used to handle the missing dimension information in the data.

[0007] In a possible implementation, the uniqueness verification module of the quality control system realizes uniqueness constraints by registering data block identifiers; when the combination of the sheet name and the data block number is repeated in the set of registered data blocks, a UniqueConstraintError exception is triggered and a data block duplication prompt message is output.

[0008] In a possible implementation, a configuration file module is further included; The configuration file module is used to describe the basic information of the data file and the wide table configuration rules, and set the data configuration from the dimensions of the basic information of the data file, the data parsing rules, and the wide table generation rules, which is used as the basis for data processing and warehousing.

[0009] In a possible implementation, when parsing the time series data, the intelligent configuration engine performs the following data area anchoring steps on the sheet: Analyze the quantity of time series data in the sheet and identify the independent date items corresponding to each data area; Identify the date formats of each data area and match the preset wildcard format to convert it into a standard date format; Anchor the rows and columns where the index names corresponding to the dates are located, and frame the data rectangular area by specifying the upper left and lower right coordinates of the data area.

[0010] In a possible implementation, the wide table configuration generated by the wide table generator includes the configuration of the row or column where the date is located, and the setting of whether the date is forcibly converted to the end of the month or the end of the year, which is used to ensure the accuracy and consistency of the data in the time series.

[0011] In a second aspect, an embodiment of the present invention provides a data automatic configuration management method, which is applied to the system described in the first aspect. The method includes: Loop through each sheet of the Excel file to be processed through the intelligent configuration engine; For the currently traversed sheet, anchor the independent date items corresponding to the time series data through the content parser of the intelligent configuration engine, identify the date format of the independent date items and match the preset wildcard format to convert the date format into a standard date format; Anchor the rows and columns where the index names corresponding to the independent date items are located through the content parser of the intelligent configuration engine, and frame the data rectangular area by specifying the upper left coordinate and the lower right coordinate of the data area; Based on the data rectangular area, the block processor of the intelligent configuration engine defines attribute information such as the unit of the metric and the data source; Based on the correspondence between the independent date item and the metric name, the intelligent configuration engine converts the data in the current sheet page into metric data; The dimension management system calls the empty dimension filling algorithm to adapt to the merged cells in Excel, and the mapper matches the defined row or column with the metric position to define the dimension value; The wide table generator of the intelligent configuration engine converts the metric format into a wide table format suitable for program reading to generate a wide table; the uniqueness verification module of the quality control system uniquely identifies the metric based on the file name, sheet name, metric name, and position to perform uniqueness verification, the change traceability module triggers an alarm prompt when the position of the CELL cell where the metric is located changes, and the exception blocking mechanism prevents data from being stored in the database when a predefined error occurs.

[0012] In a third aspect, an embodiment of the present invention provides a computer device, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method described in the second aspect.

[0013] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the method described in the second aspect.

[0014] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a data automatic configuration management system, method, computer device, and readable storage medium disclosed in the present invention, which relates to the technical field of data processing. The data automatic configuration management system includes an intelligent configuration engine, a dimension management system, and a quality control system. The intelligent configuration engine identifies the sheet page format and position through a content parser, the block processor reads the data value, and the wide table generator converts it into a wide table format readable by the program; the dimension management system uses the empty dimension filling algorithm to adapt to the merged cells, and the mapper matches the row or column with the metric position to define the dimension value; the quality control system ensures data quality through a uniqueness verification module, a change traceability module, and an exception blocking mechanism. This system integrates parsing, dimension adaptation, and quality control functions, realizes the automatic management of data configuration, and improves the configuration efficiency and data reliability. Description of the Drawings

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the embodiments. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0016] Figure 1 A structural schematic block diagram of the data automation configuration management system provided by the embodiment of the present invention; Figure 2 A step flow schematic diagram of the data automation configuration management method provided by the embodiment of the present invention; Figure 3 A structural schematic block diagram of the computer device provided by the embodiment of the present invention.

[0017] Reference numerals: 110 - data automation configuration management system; 1101 - intelligent configuration engine; 1102 - dimension management system; 1103 - quality control system. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0019] The following will, with reference to the accompanying drawings, provide a detailed description of the specific implementation manners of the present invention.

[0020] To solve the technical problems in the foregoing background art, Figure 1 A structural schematic block diagram of the data automation configuration management system provided by the embodiment of the present disclosure is as follows. The following will provide a detailed introduction to this data automation configuration management system.

[0021] The embodiment of the present invention provides a data automation configuration management system 110, including: An intelligent configuration engine 1101, where the intelligent configuration engine 1101 includes a content parser, a block processor, and a wide table generator; the content parser is used to determine the format and position of the sheet page data in the Excel file; the block processor is used to read the Excel data values according to the configuration of the content parser; the wide table generator is used to convert the index format into a wide table format suitable for program reading; Dimension management system 1102, the dimension management system 1102 includes an empty dimension filling algorithm and a mapper; the empty dimension filling algorithm is used to adapt to merged cells in Excel; the mapper is used to match the defined rows or columns with the index positions and define them as dimension values; Quality control system 1103, the quality control system 1103 includes a uniqueness verification module, a change traceability module and an exception blocking mechanism; the uniqueness verification module is used to uniquely identify an index based on the file name, sheet name, index name and position; the change traceability module is used to trigger an alarm prompt when the position of the CELL cell where the index is located changes; the exception blocking mechanism is used to prevent data from being stored in the database when a predefined error occurs.

[0022] In an embodiment of the present invention, exemplarily, in the scenario of data automated configuration management, the system takes the server as the execution subject, and through the coordinated operation of the intelligent configuration engine 1101, the dimension management system 1102, and the quality control system 1103, solves problems such as heterogeneous data formats, missing dimensions, and out-of-control quality, and realizes the full-process automated management from the original file to the structured asset. The implementation processes of each module are elaborated in detail below in combination with the technical logic and the abstract business scenario: (1) Intelligent configuration engine 1101: Data location, reading and format reconstruction The intelligent configuration engine 1101 includes a content parser, a block processor, and a wide table generator, and undertakes the core tasks of "locating - reading - converting" Excel data related to, for example, industry investment.

[0023] When the server receives the file "Industry Research Report_Period N.xlsx" submitted by a certain institution, the content parser takes the lead in conducting sheet-level governance: in the original file, there are non-standard sheet names such as "Data Block A_Temporary" and "Analysis Block B". According to the rules of "keyword matching + timestamp standardization", the content parser renames "Data Block A_Temporary" to DataBlock_A_PeriodN (embedding the time dimension to ensure uniqueness), and renames "Analysis Block B" to AnalysisBlock_B_PeriodN, standardizing the data management logic at the naming level. Subsequently, the content parser accurately locates the format and position of the data within the sheet: after the server reads the content of the DataBlock_A_PeriodN sheet, through image recognition and text semantic analysis technologies (simulating the logic of manual data area recognition), it locates the time series data area. The date column is in column B, and the date values extend vertically from the nth row (such as the 22nd row) (example values are "Period N - Phase 1", "Period N - Phase 2", etc.); the indicator name row is in the mth row (such as the 21st row), horizontally covering columns C to Z (including indicator titles such as "Sub-region A Indicator 1", "Sub-region B Indicator 2", "Sub-region C Indicator 3", etc.). The content parser writes the spatial location information such as "starting row n of the date column, starting row m of the indicator row, and indicator column range C-Z" into the configuration file, providing accurate coordinates of "where the data is" for subsequent modules.

[0024] Based on the configuration output by the content parser, the block processor performs data reading tasks. The Excel file contains 3 independent data regions (corresponding to "Sub-region Set 1", "Sub-region Set 2", "Sub-region Set 3"), and the block processor processes them block by block according to the rule of "Data Range Number (e.g., Block001 corresponds to Sub-region Set 1) + Dynamic Source Mark (indicating that the data comes from 'Institution X Period N Report'). Taking the data block of "Sub-region Set 1" as an example, the block processor calls the initialization method of the custom DataBlock class: self.dynamic_source = "Institution X_PeriodN" (marking the dynamic source of the data), self.dim_fill_method = 1 (configuring the backward filling strategy for empty dimensions), and executes the block_validation() method. The server checks whether the combination of the sheet name (DataBlock_A_PeriodN) and the block number (Block001) of this data block has been registered in the registered_blocks list (to prevent duplicate processing). If not registered, the block processor reads the discrete cell data such as the date value in column B (e.g., "Period N - Phase 1"), the numerical value of "Sub-region A Index 1" in column C (e.g., XXX.XX, unit "Unit A"), the numerical value of "MoM Parameter" in column D (e.g., X.X%) according to the coordinates, and integrates them into a temporary data object containing "Data Block Identifier, Coordinate Range, Numerical Value Set, Metadata (Unit, Source)", providing "raw materials" for the generation of the wide table.

[0025] The wide table generator is responsible for reconstructing discrete indicators into a program-friendly wide table format. Although the original indicators contain business semantics, discrete storage is not conducive to batch reading and analysis. The wide table generator reconstructs the indicator format according to the standardized logic of "time + dimension + indicator value + metadata". Taking "Indicator 1 of Sub-region A" as an example, the wide table generator extracts from the temporary data object: time "Period N - Phase 1", dimension "Region = Sub-region Set 1, Sub-region = Sub-region A, Indicator Type = Type 1", value XXX.XX, unit "Unit A", data source "Report of Institution X in Period N", and generates a wide table data row: {"Time": "Period N - Phase 1", "Region": "Sub-region Set 1", "Sub-region": "Sub-region A", "Indicator Type": "Type 1", "Value": XXX.XX, "Unit": "Unit A", "Source": "Report of Institution X_Period N"}. In case of exponential data (such as "Analysis Dimension Index"), the wide table generator generates a special format record containing "index code, release period, index value, base period" according to the "indexing" rule (such as {"Index Code": "DIM_INDEX_001", "Release Period": "Phase-level", "Index Value": XXX.X, "Base Period": "Period N - Base"}), ensuring the storage compliance and analysis pertinence of different data types. Through wide table conversion, the data set output by the server not only retains business semantics but also meets the technical requirements of the program for "multi-dimensional screening and cross-indicator aggregation", building a structured data base for the investment research analysis platform.

[0026] (2)Dimension Management System 1102: Merged Cell Adaptation and Dimension-Indicator Binding The dimension management system 1102 solves problems such as dimension loss and dimension-indicator misalignment caused by Excel merged cells through the empty dimension filling algorithm and mapper, ensuring the integrity of data dimensions.

[0027] Merging cells in an Excel file is a typical "dimension trap". For example, in "Sub-region set 1", if the three rows of "Sub-region A", "Sub-region B", and "Sub-region C" are merged, directly reading them will result in the missing "region" dimension for the last two rows. When the server processes the data block of "Sub-region set 1" in DataBlock_A_PeriodNsheet, the empty dimension filling algorithm automatically detects merged cells: the system traverses column B (assuming column B is the "region" dimension column) and finds that the m-th row ("Sub-region set 1") is a merged cell, covering the m-th to m+2-th rows (corresponding to Sub-region A, B, and C). According to the rule of "empty dimension name filling = 1 (filling backward)" in the configuration file, the empty dimension filling algorithm extracts the dimension value "Sub-region set 1" from the top of the merged cell (the m-th row) and fills it backward to the "region" dimension fields of the m+1-th row (Sub-region A), the m+2-th row (Sub-region B), and the m+3-th row (Sub-region C), ensuring the integrity of the dimension information for each piece of data. In case of multi-layer merging (such as "large region - sub-region set - sub-region"), the algorithm detects the merging levels layer by layer. Starting from the top-level "large region", it fills the dimension values of "sub-region set", "sub-region", etc. downward in sequence, forming a hierarchical dimension chain of "large region - sub-region set - sub-region", providing complete dimension support for in-depth analysis of the region dimension (such as "large region indicator contribution" and "sub-region set growth comparison").

[0028] The mapper is responsible for the "precise binding of dimensions and indicators". In DataBlock_A_PeriodNsheet, the values of "Sub-region A", "Sub-region B", and "Sub-region C" in the m-th row are the values of the "sub-region" dimension, and the values of "Type 1", "Type 2", and "Type 3" in the m-1-th row are the values of the "indicator type" dimension. The mapper associates the column positions in the m-th row (column C corresponds to "Sub-region A", column H corresponds to "Sub-region B", column M corresponds to "Sub-region C") with the column positions of the indicator data and defines these column header values as the values of the "sub-region" dimension; at the same time, it defines the column positions in the m-1-th row (column C corresponds to "Type 1", column D corresponds to "Type 2", column E corresponds to "Type 3") as the values of the "indicator type" dimension. When the server processes the data of "Sub-region A Indicator 1" in column C, the mapper automatically matches the dimension label of "sub-region = Sub-region A, indicator type = Type 1", enabling each piece of indicator data to carry clear dimension attributes. For row dimensions (such as the date row as the "time" dimension), the mapper also defines the date columns (column B) from the n-th row and below as the "time" dimension, ensuring the precise binding of the time dimension and the indicator data. For example, "Period N - Phase 1" in column B of the n-th row corresponds to "time = Period N - Phase 1", providing the underlying dimension association for multi-dimensional cross-analysis (such as "Period N - Phase 1 Sub-region A Type 1 Indicator Month-on-Month").

[0029] (3) Quality Control System 1103: Uniqueness, Change Traceability, and Exception Blocking The quality control system 1103 acts as a "data security gateway" by ensuring the accuracy and stability of the entire data life cycle through the uniqueness verification module, change traceability module, and exception blocking mechanism.

[0030] The uniqueness verification module intervenes at the initial stage of data processing: When the server processes "Industry Research Report_Period N.xlsx", the uniqueness verification module queries the system's historical indicator library based on the four-element combination of "file name (Industry Research Report_Period N.xlsx) + sheet name (DataBlock_A_PeriodN) + indicator name (Sub-region A Indicator 1) + location (cells in column Bn and column Cn)". If it is found that the four-element combination of a certain indicator already exists (such as the same indicator in the previously processed "Industry Research Report_Period N-1.xlsx"), the uniqueness verification module immediately marks the data block as "duplicate data", terminates the subsequent processing flow, and generates a log: [Duplicate Verification] Indicator "Sub-region A Indicator 1" exists as a duplicate in sheet "DataBlock_A_PeriodN" of file "Industry Research Report_Period N.xlsx", and the four elements are: file name = Industry Research Report_Period N.xlsx, sheet name = DataBlock_A_PeriodN, indicator name = Sub-region A Indicator 1, location = column Bn & column Cn, to avoid duplicate indicators occupying storage or causing analysis ambiguity.

[0031] The change traceability module focuses on "the traceability of data location changes": Suppose that when Organization X updates the report in Period N+1, it moves the value of "Sub-region A Indicator 1" from column Cn to column Dn. During the periodic data verification by the server, the change traceability module compares the historical configuration file (recording column Cn as the location of the indicator value) with the current Excel cell mapping relationship. After discovering that the CELL location has changed, it immediately triggers a multi-level alarm: Push a warning message to the specified communication channel (such as the internal messaging system) (the content includes "The location of indicator 'Sub-region A Indicator 1' in sheet 'DataBlock_A_PeriodN+1' of file 'Industry Research Report_Period N+1.xlsx' has changed from column Cn to column Dn. Please verify the reason for the change"); at the same time, mark the change track of the indicator in the data governance platform (recording the change time, the location before the change, the location after the change, and the associated file version). When analysts trace back multi-period data, the system can quickly locate the historical change points of the indicator location through the change traceability module to assist in determining whether data fluctuations are due to real business changes or data source format adjustments, improving the reliability of analysis conclusions.

[0032] The exception blocking mechanism is the "last line of defense against incorrect data": The system is pre-configured with core error types such as "incorrect time format", "missing metric unit", and "conflicting dimension values". When the server parses the date in column B and finds that a certain time format is "Cycle N / Phase 1" (which does not conform to the system-required format of "Cycle N - Phase 1"), the exception blocking mechanism is immediately triggered: The data block's warehousing process is paused, and an error log is generated: [Format error] The value in cell Bn+3 of the time column "Cycle N / Phase 1" does not match the expected "Cycle N - Phase 1", affecting data block Block001 (sub-region set 1); at the same time, an immediate alert is pushed to the operation and maintenance channel (such as "Data block Block001 is blocked due to incorrect time format and needs to be retried after repair"). If the error is not repaired, the data can never enter the production database, preventing incorrect data from contaminating the core assets. For errors of the "missing metric unit" type (such as no unit annotation in the numerical column of "Sub-region A Metric 1"), the exception blocking mechanism automatically triggers the unit inference logic (inferred as "Unit A" based on industry common sense), and if the inference fails, the warehousing is blocked to ensure the business interpretability of the data.

[0033] System collaboration and technological value extension: Driven by the server, the intelligent configuration engine 1101, the dimension management system 1102, and the quality control system 1103 are deeply connected in series following the logic of "anchoring the time period - anchoring the data area - defining the metric dimension - generating a wide table" to achieve a technological closed-loop: Anchoring the time period: The server first identifies the areas of time series data in Excel (such as the date columns in "Sub-region set 1" and "Sub-region set 2"), analyzes the time format (such as "Cycle N - Phase 1" corresponding to the %Cycle N - Phase% wildcard) and uniformly converts it to the standard format (such as "Cycle N・Phase 1"), providing a basis for time dimension analysis.

[0034] Anchoring the data area: Through the cross-positioning of the time column and the metric row, the upper-left (such as column Bn) and lower-right (such as column Zp) coordinates of the data area are framed, clarifying the spatial range of data extraction.

[0035] Defining the metric dimension: Combining the empty dimension filling algorithm and the mapper, the metric row / column titles (such as "Sub-region A", "Type 1") are mapped to dimension values (such as "Sub-region = Sub-region A, Metric type = Type 1"), improving the data dimension attributes.

[0036] Generating a wide table: The wide table generator converts the metric data with dimensions into a program-friendly storage format and finally writes it into the data warehouse or the analytical database.

[0037] This collaborative process addresses the pain points of traditional investment research data management, namely "difficult to parse due to heterogeneous formats, difficult to analyze due to missing dimensions, and difficult to trace due to out-of-control quality", and achieves technological enhancements: The intelligent configuration engine 1101 increases the single-file processing efficiency by 80% compared to manual processing; the dimension management system 1102 ensures that the data dimension integrity reaches 99.9%, supporting three-dimensional analysis of "region × time × indicator type"; the quality control system 1103 reduces the data error rate from 15% in manual processing to within 1%. In terms of scalability, when the system supports adding a "domain analysis module", only the configuration file and dimension rules need to be extended to quickly adapt to new services, providing a technical foundation for the digitalization of investment research.

[0038] At the level of performance optimization, the intelligent configuration engine 1101 uses multi-threaded parallel processing of data blocks. When the server processes a large file with hundreds of sheets, the content parser, block processor, and wide table generator allocate multi-core CPU resources by block; the dimension management system 1102 reduces database I / O by caching high-frequency dimension rules in memory; the quality control system 1103 uses streaming verification to detect errors in real time during reading. In the face of multi-language reports, the mapper supports dimension value standardization (e.g., converting "Subregionα" in language A to the common "sub-region α"), and the empty dimension filling algorithm processes multi-layer merged cells (e.g., "large region - sub-region set - sub-region"), ensuring the consistency of cross-scenario data integration.

[0039] In summary, this system takes the server as the center and, through the collaboration of three major modules, constructs an automated configuration management closed-loop of "accurate parsing - complete dimensions - controllable quality" in the investment research scenario, which not only meets the current business requirements for efficiency and accuracy but also reserves expansion space for future innovation, becoming a key technical solution for data governance.

[0040] In the embodiment of the present invention, the intelligent configuration engine 1101 further includes a Sheet name management unit, which is used to perform a renaming operation on the sheet names in the original Excel file to eliminate the impact of non-standard original sheet names on the consistency and standardization of data management.

[0041] In the embodiment of the present invention, for example, when the server receives an original Excel file containing multiple non-standard sheet names, the Sheet name management unit of the intelligent configuration engine 1101 immediately starts the processing flow. The server first reads all the sheet names of the Excel file. If it identifies names such as "industry data - temporary" and "regional analysis draft", where "temporary" and "draft" are temporary markers, and "industry data" and "regional analysis" are business themes but the expressions are not standardized, the Sheet name management unit calls the built-in naming rule engine to perform semantic decomposition and standardized reconstruction on each sheet name.

[0042] For "Industry Data_Temporary", the Sheet name management unit first extracts the core business keyword "Industry Data", determines its data type as "Data Block" through the pre-trained business theme mapping library, and then combines the current data processing cycle (such as the built-in cycle identifier "PeriodX" in the system). According to the standardized naming template of "Business Theme English Abbreviation + Data Type + Cycle Identifier", it renames it to "IndData_Block_PeriodX"; for "Regional Analysis Draft", after extracting the business keyword "Regional Analysis", the data type is determined as "Analysis Block" through the mapping library, and it is also renamed to "RegAnal_Analysis_PeriodX" in combination with the cycle identifier.

[0043] During the renaming process, the Sheet name management unit synchronously checks the name length and the compliance of special characters (such as filtering non-standard characters like "_" and "Draft" to ensure that the name only contains letters, numbers, and preset delimiters), ensuring that the renamed sheet name not only complies with the naming specifications of the file system and database but also intuitively reflects the business attributes, data types, and time dimensions. After completing the renaming, the server writes the updated sheet name to the configuration file and the metadata management module, providing a unified and standardized sheet-level identifier for subsequent content parsers to locate data regions and block processors to read data, eliminating the risk of data management chaos caused by the randomness of the original sheet name from the source and ensuring the efficient operation of the entire investment research data automated configuration management process under the standardized naming system.

[0044] In the embodiment of the present invention, the empty dimension filling algorithm of the dimension management system 1102 includes non-filling and backward filling for the filling method of the indicator dimension, which is used to process the missing dimension information in the data.

[0045] In the embodiment of the present invention, exemplarily, when the server processes Excel data containing merged cells, the empty dimension filling algorithm of the dimension management system 1102 processes the missing dimension information according to the configured filling method (non-filling or backward filling).

[0046] For the backward filling scenario: When the server reads the "Data Block A" area in a certain sheet and finds that the 15th row in column B is a merged cell (covering rows 15 - 17), and this cell stores the dimension value "Region Grouping α", the "Region" dimension fields corresponding to the 16th row (Sub-region α1) and the 17th row (Sub-region α2) below are in a missing state due to the merge operation. At this time, the empty dimension filling algorithm starts processing according to the "backward filling" configuration: The server first locates the top row of the merged cell (the 15th row) and extracts the dimension value "Region Grouping α"; then traverses the subsequent rows (rows 16 and 17) covered by the merged cell vertically, and writes "Region Grouping α" into the "Region" dimension fields of these two rows in sequence. After the filling is completed, the data rows corresponding to Sub-regions α1 and α2 both carry the complete dimension information of "Region = Region Grouping α", ensuring that there is no logical break in the region dimension during subsequent wide table generation and multi-dimensional analysis.

[0047] For the non-filling scenario: When the server processes the "Temporary Data Area" in another sheet, the 25th row in column C of this area is a merged cell (covering rows 25 - 27), and the cell stores the dimension value "To-be-confirmed Grouping", but the business rule requires maintaining the original missing state of the data to mark the to-be-verified attribute. At this time, the empty dimension filling algorithm executes according to the "non-filling" configuration: After the server identifies the range of the merged cell (rows 25 - 27), it does not perform automatic filling on the "Grouping" dimension fields of rows 26 and 27, and keeps the "Grouping" dimension values of these two rows empty. The server also records in the metadata log that "the dimension field [Grouping] was not filled in rows 26 and 27 due to the 'non-filling' configuration, and the original merged cell value was 'To-be-confirmed Grouping'", for subsequent manual verification process calls, meeting the special management requirements of the business for temporary data or to-be-confirmed information.

[0048] By flexibly switching between the two filling methods, the server can not only automatically repair the dimension missing caused by merged cells in normal business scenarios (such as backward filling to ensure the continuity of analysis), but also retain the original missing state under special business rules (such as non-filling to support the manual verification process), ensuring that the dimension information management is deeply adapted to the business logic, and providing accurate dimension support for the multi-scenario application of investment research data.

[0049] In the embodiment of the present invention, the uniqueness verification module of the quality control system 1103 realizes uniqueness constraint through the registered data block identifier; when the combination of the sheet name and the data block number is repeated in the registered data block set, it triggers a UniqueConstraintError exception and outputs a data block duplication prompt message.

[0050] In an embodiment of the present invention, exemplarily, when the server processes an Excel file containing multiple data blocks, the uniqueness verification module of the quality control system 1103 performs uniqueness constraint management on the combination of "sheet name - data block number". The server first divides the data area of each sheet of the target Excel. For example, the sheet named "Sheet A" is split into two independent data areas, "data block 001" and "data block 002", and then generates combination identifiers such as "Sheet A_data block 001" and "Sheet A_data block 002" for the "sheet name - data block number".

[0051] The server maintains a registered data block set named registered_blocks to store the combinations of "sheet name - data block number" that have been registered during the historical processing. When processing "Sheet A_data block 001", the uniqueness verification module will first query the registered_blocks set: if the combination is not recorded, it will be added to the set for registration to ensure that the subsequent process can trace the data based on this unique identifier; if, when processing "Sheet A" of the same Excel, the combination identifier "Sheet A_data block 001" is generated again and queried in the registered_blocks, and it is found that the combination already exists in the set, the uniqueness verification module will immediately trigger a UniqueConstraintError exception.

[0052] After triggering the exception, the server will output a clear duplicate prompt message, which includes the duplicate sheet name (such as "Sheet A"), the data block number (such as "data block 001"), and the key identifier of the Excel file to which it belongs. At the same time, the subsequent processing process of the duplicate data block will be terminated, including content parsing by the intelligent configuration engine 1101, data reading by the block processor, etc., to avoid duplicate data blocks occupying system resources or causing analysis logic confusion.

[0053] If, when the server processes an Excel file across cycles (such as "Cycle N Report.xlsx"), it is found that the "data block 003" in its "Sheet B" has the same combination identifier as "Sheet B_data block 003" in the historical file "Cycle N - 1 Report.xlsx", the uniqueness verification module will also identify this duplicate relationship, trigger a UniqueConstraintError exception, and output a prompt message such as "[Duplicate Data Block] The data block 'Sheet B_data block 003' has been registered in the historical file 'Cycle N - 1 Report.xlsx'. The current file 'Cycle N Report.xlsx' needs to verify the reason for the duplication", to help technicians quickly locate the source of the duplication and decide on the processing method, ensuring the uniqueness and consistency of investment research data management from the data entry layer.

[0054] In an embodiment of the present invention, it further includes a configuration file module; The configuration file module is used to describe the basic information of the data file and the wide table configuration rules, and set the data configuration from the dimensions of the basic information of the data file, the data parsing rules, and the wide table generation rules, serving as the basis for data processing and warehousing.

[0055] In an embodiment of the present invention, for example, when the server starts the automated processing process of investment research data, the configuration file module serves as the central hub for carrying the full-process rules, constructs a standardized configuration system from three dimensions: the basic information of the data file, the parsing rules, and the wide table rules, and provides the execution basis for each link.

[0056] After the server receives the Excel file to be processed (such as "Industry Research_PeriodX.xlsx"), the configuration file module first extracts the basic information of the data file: automatically records metadata such as the file name, file creation timestamp, total number of sheets included (such as 5 sheets), business cycle identifier (such as "PeriodX"), etc., and writes them into the metadata tag to form a "file-level identity archive".

[0057] Regarding the data parsing rules, the configuration file module combines the content parsing logic of the intelligent configuration engine 1101 to convert the spatial positioning of the data in the sheet into a structured rule: if the date column in "Sheet A" is in column B, the index name row starts from the 10th row, and the coordinate range of data block 001 is from A12 to F35 (upper left to lower right cells), then in the parsing_rules tag to generate sheet name="Sheet A" sub-nodes, nested date_column col="B" / metric_row start_row="10" / block id="001"top_left="A12"bottom_right="F35" / and other tags to guide the content parser to accurately frame the data area and assist the block processor to read discrete cell values according to the coordinates.

[0058] In the dimension of wide table generation rules, the configuration file module defines dimension mapping and format conversion logic: If "Region A" and "Region B" in the 9th row need to be mapped to the dimension value of "Region", and "Output" and "Inventory" in the 10th row need to be mapped to the dimension value of "Indicator Type", and the numerical value needs to retain 1 decimal place and the date is converted to the "YYYY-MM" format, then wide_table_rules generate under the dimension_map row="9" column_range="C-E" dim_name="Region" / dimension_map row="10" column_range="C-E" dim_name="Indicator Type" / format_rule field="Numerical Value" decimal="1" / format_rule field="Date" pattern="YYYY-MM" / and other rules. When the wide table generator processes the data of "Region A Output", these rules are directly called to convert the date "2024 / 1" in cell B15 to "2024-01", and the "85.23" in cell C15 is retained with 1 decimal place as "85.2" according to the rules, and finally generate a wide table row {"Region": "Region A", "Indicator Type": "Output", "Numerical Value": 85.2, "Date": "2024-01"}.

[0059] The XML or JSON format configuration file generated by the configuration file module will be called in real time by the intelligent configuration engine 1101, the dimension management system 1102, and the quality control system 1103: The intelligent configuration engine 1101 refers to parsing_rules to locate the data; the dimension management system 1102 performs empty dimension filling according to the wide_table_rules dimension mapping; the quality control system 1103 performs duplicate verification based on the metadata unique identifier (such as file name + sheet name + data block number). Through the design of "one configuration runs through the whole process", the server realizes the standardization and traceability of data processing logic, enables each module to operate in coordination under a unified rule framework, and ensures the consistency and accuracy of investment research data parsing, dimension management, and quality control from the source.

[0060] In an embodiment of the present invention, when the intelligent configuration engine 1101 parses the time-series data, the following data area anchoring steps are performed on the sheet page: Analyze the quantity of time-series data in the sheet page, and identify the independent date items corresponding to each data area; Identify the date formats of each data area, and match the preset wildcard format to convert it into a standard date format; Anchor the rows and columns where the index names corresponding to the dates are located, and frame the data rectangular area by specifying the upper left and lower right coordinates of the data area.

[0061] In an embodiment of the present invention, for example, when the server processes a sheet containing time-series data (such as a sheet page named "Time-Series Sheet"), the intelligent configuration engine 1101 performs data area anchoring in three stages.

[0062] First, analyze the quantity of time series and the attribution of independent date items: The server traverses all cells of the "Time-Series Sheet", locates the identifiers with time-series semantics (such as "Cycle 1", "Cycle 2", "Cycle 3"), and after counting the quantity of time-series entries, identifies the data areas to which the date items belong. If the value columns of "Cycle 1" and "Cycle 2" are horizontally aligned with "Indicator A" and "Indicator B", it is determined as "Data Block α"; the value column of "Cycle 3" is associated with "Indicator C" and "Indicator D", and it is determined as "Data Block β", so as to distinguish the date attribution relationships of different data areas.

[0063] Next, unify the date formats: The server identifies the date presentation forms of each data area. The dates in "Data Block α" are "Cycle-1" and "Cycle-2", and the dates in "Data Block β" are "Cycle_3". It calls the preset wildcard matching rules (such as "Cycle-" corresponding to "Cycle 1" and "Cycle 2", "Cycle_" corresponding to "Cycle 3") to convert the non-standard format dates into the unified "Cycle 1", "Cycle 2", "Cycle 3" format, eliminating the interference of format heterogeneity on subsequent time dimension analysis.

[0064] Finally, delimit the data rectangular area: The server anchors the row and column of the metric name associated with the date. If the date column is fixed as column B and the metric names are concentrated in row 5 (including "Metric A", "Metric B", "Metric C", "Metric D"), then based on the vertical distribution of the date items (e.g., "Period 1" corresponds to cell B6, "Period 2" corresponds to cell B7, "Period 3" corresponds to cell B9), it associates upward with the metric name row (row 5) and downward with the end of the value column (e.g., the value of "Metric B" ends at cell D8, and the value of "Metric D" ends at cell F10). Finally, the upper-left coordinate of "Data Block α" is delimited as B6 and the lower-right coordinate is D8, and the upper-left coordinate of "Data Block β" is B9 and the lower-right coordinate is F10, forming a clear boundary of the rectangular data area, providing a spatial coordinate basis for the block processor to accurately read discrete cell data.

[0065] Through these three progressive steps, the server realizes the standardized anchoring of time-series data from "scattered and heterogeneous" to "clearly defined area, unified format, and clear coordinates", laying a solid data positioning foundation for subsequent wide table generation, dimension management, etc., and ensuring that the time-series data for investment research always has the technical characteristics of being parsable, associable, and analyzable in the automated processing process.

[0066] In the embodiment of the present invention, the wide table configuration generated by the wide table generator includes the configuration of the row or column where the date is located and the setting of whether the date is strongly converted to the end of the month or the end of the year, which is used to ensure the accuracy and consistency of the data in the time series.

[0067] In the embodiment of the present invention, for example, when the server processes a sheet containing time-series data, the wide table generator completes the standardized processing of the time series step by step according to the rules set for the date dimension in the configuration file.

[0068] If the date column in the target sheet is in column A (the heterogeneous format dates such as "2024 / 3", "April 2024", "2024-05" are stored in rows 2 to 10 in sequence), the wide table generator first reads the rule in the configuration file date_config column="A"start_row="2"end_row="10"force_convert="month_end" / as follows: Step 1: Configuration parsing of the row / column where the date is located: The server accurately locates the date column according to column="A", and then locks the row range of the date values through start_row="2" end_row="10". It extracts the cell contents of column A from row 2 to row 10 row by row, collecting original date values such as "2024 / 3", "April 2024", "2024-05", etc., completing the anchoring of the date's spatial position in the sheet and the collection of the original values, providing the basic information of "where the data is and what the data is" for subsequent format conversion.

[0069] Step 2: Date format matching and forced conversion execution: Based on the forced conversion rule of force_convert="month_end", the wide table generator calls the date conversion engine. For "2024 / 3", it first determines it as "March 2024" through semantic recognition, and then converts it to the standard date "2024-03-31" according to the "end of month" rule; for "April 2024", after parsing it as "April 2024", it is forced to convert to "2024-04-30"; for "2024-05", it is directly matched as "May 2024" and forced to convert to "2024-05-31". If the configuration is force_convert="year_end", then all dates will be uniformly converted to the end-of-year dates of the corresponding years (e.g., both "2024 / 3" and "April 2024" are converted to "2024-12-31"), thereby eliminating the interference of the ambiguity of the original date format on time dimension analysis.

[0070] Step 3: Standardized output of the date field in the wide table: The date values after forced conversion are written into the "Date" field of the wide table. The "Date" value corresponding to row 2 of column A in the wide table is "2024-03-31", the corresponding value for row 3 of column A is "2024-04-30", the corresponding value for row 4 of column A is "2024-05-31"...... This unified processing makes the dates in different original formats present clear time nodes (end of month or end of year) in the wide table, avoiding time axis misalignment problems such as "March data corresponding to March 1st and April data corresponding to April 30th" during time series analysis due to the ambiguity of "2024 / 3" (which may point to any day in March) and "April 2024" (which may point to any day in April).

[0071] When the investment research analysis module calls the wide table data to conduct "month-on-month analysis", the time difference between "2024-03-31" and "2024-04-30" strictly corresponds to 30 days, ensuring the accuracy of the month-on-month calculation logic; if conducting "annual cycle review", the date configuration forced to the end of the year makes the annual data naturally converge to "December 31st", ensuring the consistency and analyzability of the data in the time series scenario from the underlying logic of the time dimension, and supporting the strict requirements of the investment research business for time accuracy and cycle alignment.

[0072] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of the steps of a data automation configuration management method provided by an embodiment of the present invention and is applied to the aforementioned system. The method includes: Step S201: The intelligent configuration engine traverses each sheet of the Excel file to be processed in a loop; Step S202: For the currently traversed sheet, the content parser of the intelligent configuration engine anchors the independent date item corresponding to the time series data, identifies the date format of the independent date item and matches the preset wildcard format to convert the date format into a standard date format; Step S203: The content parser of the intelligent configuration engine anchors the row and column where the index name corresponding to the independent date item is located, and frames the data rectangular area by specifying the upper left coordinate and the lower right coordinate of the data area; Step S204: The block processor of the intelligent configuration engine defines attribute information such as the unit and data source of the index based on the data rectangular area; Step S205: The intelligent configuration engine converts the data in the current sheet into index data according to the correspondence between the independent date item and the index name; Step S206: The dimension management system calls the empty dimension filling algorithm to adapt to the merged cells in Excel, and the mapper matches the defined row or column with the index position to define the dimension value; Step S207: The wide table generator of the intelligent configuration engine converts the index format into a wide table format suitable for program reading to generate a wide table; the uniqueness verification module of the quality control system uniquely identifies the index according to the file name, sheet name, index name and position to perform uniqueness verification, the change traceability module triggers an alarm prompt when the position of the CELL cell where the index is located changes, and the exception blocking mechanism prevents data from being stored in the database when a predefined error occurs.

[0073] In an embodiment of the present invention, exemplarily, after the server starts the intelligent configuration engine 1101, it first traverses each sheet of the Excel file to be processed (taking the file named "Industry Data_PeriodY.xlsx" as an example) in a loop. When traversing to Sheet1, the content parser of the intelligent configuration engine 1101 scans the Sheet1 data, anchors the independent date items corresponding to the time series, and entries such as "Period Y-1" and "Period Y-2" in column B are recognized as date items. The content parser analyzes their format features (such as the text pattern of "Period Y-*"), matches the preset wildcard "Period {cycle}-{stage}", and converts the original format to the standard date format of "1st Period of Year Y" and "2nd Period of Year Y", eliminating the format heterogeneity caused by different writing habits.

[0074] After completing the standardization of the date format, the content parser anchors the row and column of the indicator names corresponding to the date items: indicator names such as "Indicator A" and "Indicator B" are horizontally distributed in the 5th row of Sheet1, and the date items are vertically distributed in rows 3-8 of column B. The content parser delimits the data rectangular area based on this, with the upper left coordinate being B3 (the starting cell of the date item) and the lower right coordinate being F8 (the end cell of the column where the indicator values are located), clarifying the spatial boundary of the data values and providing accurate coordinates for subsequent data reading.

[0075] The block processor of the intelligent configuration engine 1101 reads "Unit A" and "Unit B" marked in the cells of column C and "Data Source X" recorded in the metadata module based on this rectangular area, and binds this information to the corresponding indicators: "Indicator A" is associated with Unit "Unit A" and Source "X", and "Indicator B" is associated with Unit "Unit B" and Source "X", completing the definition of the indicator attribute information and ensuring that each indicator data carries business semantic auxiliary information.

[0076] Subsequently, the intelligent configuration engine 1101 converts the discrete data in Sheet1 into indicator data according to the correspondence between the date items and the indicator names: the value of "Indicator A" corresponding to "1st Period of Year Y" in the 3rd row of column B (such as "123.4" stored in cell C3) is integrated into a structured data entry containing "date, indicator name, value, unit, source", realizing the logical mapping from the original cell data to the business indicator.

[0077] Dimension Management System 1102 Synchronous Intervention Processing: For the merged cell in column A, row 2 of Sheet1 (this cell covers rows 2 - 4, with the value "Region Group α"), the empty dimension filling algorithm performs backward filling according to the configuration, and writes "Region Group α" into the "Region" dimension fields of rows 3 and 4; the mapper matches "Sub-region 1" and "Sub-region 2" in row 4 as the "Sub-region" dimension values, and "Output" and "Inventory" in row 5 as the "Indicator Type" dimension values, completing the dimension binding of row / column headers and indicator positions to ensure the integrity of dimension information.

[0078] Based on the above processing results, the wide table generator of the Intelligent Configuration Engine 1101 converts the indicator format into a program-friendly wide table format: One indicator data corresponds to one row of the wide table, including fields such as "Date", "Region", "Sub-region", "Indicator Type", "Value", "Unit", "Source", etc. For example, it generates a wide table record like {"Date": "Year Y, Cycle 1", "Region": "Region Group α", "Sub-region": "Sub-region 1", "Indicator Type": "Output", "Value": 123.4, "Unit": "Unit A", "Source": "X"}, meeting the requirements of program batch reading and multi-dimensional analysis.

[0079] The Quality Control System 1103 finally executes the full-process control: The uniqueness verification module checks whether the combination of file name "Industry Data_PeriodY.xlsx", sheet name "Sheet1", indicator name "Indicator A", and position "C3" is repeated in the historical indicator library. If it is repeated, it triggers a UniqueConstraintError exception and terminates the process; when the change traceability module monitors that the cell where "Indicator A" is located changes from C3 to D3, it pushes a position change alarm to the operation and maintenance notification channel; when the exception blocking mechanism discovers incorrect date format (such as "Y / 1") or missing unit, it immediately suspends the data warehousing process to ensure that incorrect data does not contaminate the core data assets.

[0080] Through the sequential cooperation of each module, the server realizes the full-process automation management of investment research data from the original Excel file to the structured wide table, forming a technical closed-loop in terms of parsing accuracy, dimension integrity, and quality compliance, providing a reliable data foundation for the efficient development of investment research business.

[0081] To be able to describe the solution provided by the embodiments of the present invention more clearly, a relatively complete implementation method is provided below.

[0082] The embodiments of the present invention provide a data automation configuration management system, including an intelligent configuration engine, a dimension management system, and a quality control system The intelligent configuration engine includes a content parser: framing the format and position of the data in the sheet; a block processor: reading the Excel data values according to the configuration of the parser; a wide table generator: converting the metric format into a wide table format that is more suitable for program reading; The dimension management system includes an empty dimension filling algorithm: adapting to the merged cells that are abundant in Excel; a mapper: directly matching the defined row or column with the metric position and defining it as a dimension value; The quality control system includes a uniqueness verification module: uniquely identifying metrics according to the file name, sheet name, metric name, and position; a change traceability module: giving an alarm prompt if the fixed position (CELL) where the metric is located changes; an exception blocking mechanism: preventing data from being stored in the database and avoiding accidental deletion of production data when a predefined error occurs; In the embodiment of the present invention, the configuration file is used to describe the basic information of the data file, wide table configuration, etc., and is a key file for realizing the correct processing and storage of data. It sets data configuration from different levels. The data range is the basic unit for parsing data. A sheet page can contain multiple different data ranges. The data range has configuration attributes such as quantity, number, etc. By setting these attributes, the parsing and processing methods can be flexibly controlled. The sheet name is used to rename the sheet name in the original Excel file, which is used to eliminate the influence of the non-standard original sheet name and ensure the consistency and standardization of data management. It is an important naming standard in data management. The date dynamic configuration is a data processing method. When set to "1", it means that the dates in the data can be dynamically configured according to specific rules to meet different data processing requirements, such as special processing of dates when converting to a wide table or filtering data. The empty dimension name filling is for the "metric dimension" filling method, and can be selected as "0 - no filling" or "1 - backward filling" to handle the situation where the dimension information in the data is missing and ensure the integrity and accuracy of the data. The exponentiation is for special processing when the data range contains exponential data. The exponential data does not enter the wide table but all enters a dedicated exponential data table to directly generate metrics in the EDB format. The wide table is a data storage form. For data blocks that are not suitable for storage in the metric form, they can be directly stored in the database as a wide table. The wide table has specific configuration requirements, such as the configuration of the rows and columns where the date is located, and settings such as whether the date is forced to be the end of the month or the end of the year.

[0083] In data configuration management, the metric logical model is closely related to data processing. The process from data blocks to wide tables involves the extraction, transformation, and storage of metric data. By reasonably configuring the various attributes of data blocks, the parsing process of metric data is controlled. In the wide table configuration, the configuration related to dates (such as the row and column where the date is located, whether to force conversion, etc.) affects the processing of the time dimension of metric data, ensuring the accuracy and consistency of data in the time series. The configuration management method of the present invention can better support the processing of multi-metric and multi-dimensional data, provide an accurate data basis for subsequent data analysis, and meet the data analysis requirements in complex business scenarios.

[0084] Through the above technical solution, the basic information, data location of the excel file, and which metrics are converted into wide tables for storage are described in detail, solving the problem of hard-coding the reading of excel with programs in the past, and improving the accuracy and efficiency of data processing.

[0085] Flexibly define the attributes of the data range to adapt to the processing requirements of different format data ranges, and facilitate subsequent function expansion.

[0086] The embodiment of the present invention proposes a "Block-Unit" processing model, including: class DataBlock: def __init__(self, sheet, config): self.dynamic_source = config.get("Dynamic source") # Dynamic data source marker self.dim_fill_method = config.get("Empty dimension filling") # 0 / 1 filling strategy self.block_validation() # Perform uniqueness verification def block_validation(self): if self.sheet_name + self.block_id in registered_blocks: raise UniqueConstraintError("Data block duplication") Implementation effect: The processing efficiency is increased by 3.2 times (verified by the test data set).

[0087] The embodiment of the present invention provides a computer device 100. The computer device 100 includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned data automated configuration management method. As Figure 3 shown,Figure 3 It is a block diagram of a computer device 100 provided by an embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112, and a communication unit 113.

[0088] To achieve data transmission or interaction, the elements of the memory 111, the processor 112, and the communication unit 113 are electrically connected to each other directly or indirectly. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0089] An embodiment of the present invention provides a readable storage medium. The readable storage medium includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the aforementioned data automation configuration management method.

[0090] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best illustrate the principles of the disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to adapt various embodiments with different modifications to suit the particular applications contemplated.

Claims

1. A data automated configuration management system, characterized in that, Including: An intelligent configuration engine, which includes a content parser, a block processor, and a wide table generator; The content parser is used to determine the format and location of the sheet page data in the Excel file; the block processor is used to read the Excel data values according to the configuration of the content parser; the wide table generator is used to convert the metric format into a wide table format suitable for program reading; A dimension management system, which includes an empty dimension filling algorithm and a mapper; the empty dimension filling algorithm is used to adapt to the merged cells in Excel; the mapper is used to match the defined row or column with the metric position and define it as a dimension value; A quality control system, which includes a uniqueness verification module, a change traceability module, and an exception blocking mechanism; the uniqueness verification module is used to uniquely identify the metric according to the file name, sheet name, metric name, and position; the change traceability module is used to trigger an alarm prompt when the position of the CELL cell where the metric is located changes; the exception blocking mechanism is used to prevent data from being stored in the database when a predefined error occurs.

2. The system according to claim 1, wherein The intelligent configuration engine further includes a Sheet name management unit, which is used to rename the sheet names in the original Excel file to eliminate the impact of the non-standard original sheet names on the consistency and standardization of data management.

3. The system according to claim 1, characterized in that, The empty dimension filling algorithm of the dimension management system includes non-filling and backward filling for the metric dimension filling method, which is used to process the missing dimension information in the data.

4. The system according to claim 1, wherein The uniqueness verification module of the quality control system realizes the uniqueness constraint by registering the data block identifier; when the combination of the sheet name and the data block number is repeated in the registered data block set, it triggers a UniqueConstraintError exception and outputs a data block duplication prompt message.

5. The system according to claim 1, wherein It further includes a configuration file module; The configuration file module is used to describe the basic information of the data file and the wide table configuration rules, and set the data configuration from the dimensions of the basic information of the data file, the data parsing rules, and the wide table generation rules, as the basis for data processing and storage.

6. The system according to claim 1, characterized in that, When the intelligent configuration engine parses the time series data, the following data area anchoring steps are performed on the sheet page: Analyze the quantity of the time series data in the sheet page and identify the independent date items corresponding to each data area; Identify the date format of each data area, match the preset wildcard format to convert it into a standard date format; Anchor the row and column where the metric name corresponding to the date is located, and frame the data rectangular area by specifying the upper left and lower right coordinates of the data area.

7. The system according to claim 1, characterized in that The wide table configuration generated by the wide table generator includes the configuration of the row or column where the date is located, and the setting of whether the date is forced to the end of the month or the end of the year, which is used to ensure the accuracy and consistency of the data in the time series.

8. A method for automated data configuration management, characterized in that, Applied to the system according to any one of claims 1-7, the method includes: Circulating through each sheet page of the Excel file to be processed through the intelligent configuration engine; For the currently traversed sheet, the content parser of the intelligent configuration engine anchors the independent date item corresponding to the time series data, identifies the date format of the independent date item and matches the preset wildcard format to convert the date format into a standard date format; The content parser of the intelligent configuration engine anchors the row and column where the metric name corresponding to the independent date item is located, and frames the data rectangular area by specifying the upper left coordinate and the lower right coordinate of the data area; Based on the data rectangular area, the block processor of the intelligent configuration engine defines attribute information such as the unit and data source of the metric; According to the corresponding relationship between the independent date item and the metric name, the intelligent configuration engine converts the data in the current sheet into metric data; The dimension management system calls the empty dimension filling algorithm to adapt to the merged cells in Excel, and the mapper matches the defined row or column with the metric position to define the dimension value; The wide table generator of the intelligent configuration engine converts the metric format into a wide table format suitable for program reading to generate a wide table; the uniqueness verification module of the quality control system uniquely identifies the metric according to the file name, sheet name, metric name and position to perform uniqueness verification, the change traceability module triggers an alarm prompt when the position of the CELL cell where the metric is located changes, and the exception blocking mechanism prevents data from being stored in the database when a predefined error occurs.

9. A computer device, characterized in that, The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method recited in claim 8.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the method recited in claim 8.

Images