Method for automatically generating library table resources based on data acquisition

Abstract

The invention discloses a method for automatically generating library table resources based on data collection, and relates to the technical field of big data. Comprising the steps of 1, configuring basic information of library table resources through a front end, 2, uploading an Excel data file, 3, creating a library table: automatically detecting whether the library table exists or not, updating a table structure if the library table exists, and automatically constructing a data table structure according to the configured basic information of the library table resources if the library table does not exist, 4, importing data: sending a request to a file server, the method comprises the following steps: step 1, acquiring an Excel file by introducing a unique identifier of a file, analyzing file content, and generating and executing SQL (Structured Query Language) statements to import data in batches, and step 2, creating library table resources: associating the imported data with a corresponding catalog according to a configured information resource catalog, and setting resource attributes: setting a corresponding attribute for each library table resource, the method comprises the following steps of: storing basic information of library table resources in a metadata system, and generating and managing the library table resources: generating a library table resource management system which comprises all imported data tables and catalogs and metadata information associated with the data tables, data resources are managed and queried through a system.

Description

Technical Field

[0001] The present invention discloses a method for automatically generating library table resources based on data collection, and relates to the technical field of big data. Background Art

[0002] In the field of data governance, data collection, processing and management is a complex and important task. Whether it is the source of data or the type of data, there are various differences and characteristics. These differences are mainly reflected in the diverse sources of data, the various data types, and the different methods of data collection, which increases the complexity and difficulty of data collection. At the same time, file-type data usually has a complex structure and format, and requires special parsing and processing to be effectively used. Therefore, the existing resource type registration methods, such as library table registration, file registration, interface registration, etc., can no longer meet the parsing and processing needs of file-type data. Summary of the invention

[0003] In response to the problems of the prior art, the present invention provides a method for automatically generating library and table resources based on data collection, which is applied to data governance and government service scenarios. It mainly focuses on improving the automation and standardization of data collection to storage processes, and aims to simplify the conversion process from raw data files to structured database resources.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for automatically generating library table resources based on data collection, comprising:

[0006] Step 1: Configure the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition, and the resource location.

[0007] Step 2: Upload Excel data files: upload Excel data files in various formats to the file server, parse the data files, obtain the metadata information of the corresponding table names and column names, encapsulate the data list according to the metadata information, and feed back the data list and operation status to the front end in JSON format.

[0008] Step 3: Create a library table: Automatically detect whether the library table exists. If it exists, update the table structure. If it does not exist, automatically build the data table structure according to the configured basic information of the library table resources.

[0009] Step 4: Import data: Send a request to the file server, obtain the Excel file by passing in the unique file identifier, parse the file content, generate and execute SQL statements to import data in batches,

[0010] Step 5: Create library table resources:

[0011] According to the configured information resource directory, the imported data is associated with the corresponding directory.

[0012] Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table.

[0013] Storing metadata information: storing basic information of library and table resources in the metadata system.

[0014] Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

[0015] Furthermore, the method for automatically generating library table resources based on data collection, step 1: configuring basic information of library table resources through the front end, includes:

[0016] On the basic information page of the library and table resources, define a request mapping named / register through the @RequestMapping annotation to configure the registration request for the basic information of the resource.

[0017] Get dictionary data: Use the platformService service to get dictionary data and store the dictionary data in the model so that it can be displayed on the page.

[0018] Set the sharing type: call catalogService to get the list of sharing types for the catalog, and add the list of sharing types to the model.

[0019] Set the open type: Get the open type list of the catalog through catalogService and store the open type list in the model.

[0020] Set page attributes: Set page-related attributes to the model.

[0021] Return view: Specify and return the view named resource / filetable / register through the this.getView() method, and pass the model to the view for page rendering.

[0022] Further, in step 2 of the method for automatically generating library table resources based on data collection, the specific steps include:

[0023] Get remote file: Get the Excel data file uploaded to the file server and save it to a local temporary folder.

[0024] Parse Excel data files: Use parsing tools to parse Excel data files and extract metadata information from the files.

[0025] Convert and organize data: Convert the metadata information obtained by parsing, including converting the pinyin of file names to table names and column names to pinyin.

[0026] Display data: Display the processed metadata information in the form of a list, and set the number of rows to display as needed.

[0027] Error handling and logging: During the process of file acquisition, parsing and processing, any abnormal situations that occur are captured and handled, and the relevant error information is recorded in the log.

[0028] Output result: Return the data to the front-end page in JSON format.

[0029] Further, the step 3 of creating a library table in the method for automatically generating library table resources based on data collection specifically includes:

[0030] Get parameters: Get parameters from the creation request, including file ID, file name, directory ID, whether to rebuild the table, table name, data source ID, and column information.

[0031] Preprocessing: Add creation time and update time as additional columns to the column information list.

[0032] Directory and organization verification: query directory information based on directory ID, obtain directory title and organization code, check whether the logged-in user is bound to the organization, and return an error message if not bound.

[0033] Database operation: Get database connection information based on the data source ID, determine whether the table exists, and decide whether to create a new table, rebuild the table, or update the data based on the isdrop parameter. If the table already exists and needs to be rebuilt, delete the original table first and then create a new table.

[0034] If the table needs to be updated, check and adjust the primary key to ensure that the table structure is consistent with the column information passed in.

[0035] Create or update the table structure, use the database.createTableSql method to generate the table creation SQL and execute it.

[0036] Metadata management: After successfully creating a table, call the service method to record the metadata information of the table.

[0037] Further, the step 4 of importing data in the method for automatically generating library table resources based on data collection includes:

[0038] Use the HTTP client to send a request to the file server and obtain the Excel file by passing in the unique file identifier.

[0039] Parse the downloaded Excel file and convert the data into a data structure that can be directly stored in the database (List <Map<String,Object> >),

[0040] Identify column names: automatically remove the first row of Excel as column names, adapt to Excel data headers in different formats,

[0041] Verify data: Perform basic verification on data before importing, remove invalid data rows or data rows that do not meet database constraints, and ensure data quality.

[0042] Dynamic column matching: query the database table structure based on the parsed column information to achieve intelligent matching between data columns and database fields.

[0043] Batch insert: Use the database batch insert API to write data into the database at one time or in multiple batches.

[0044] Status feedback: Provides detailed import result feedback, including the number of successfully imported data rows, error codes, and prompt information, so that users can monitor the import progress.

[0045] The present invention also provides a device for automatically generating library table resources based on data collection, including a library table resource basic information configuration module, a data file upload module, a library table creation module, a data import module and a library table resource creation module.

[0046] The basic information configuration module of the library and table resources configures the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition and the resource location.

[0047] The data file upload module uploads Excel data files: uploads Excel data files in various formats to the file server, parses the data files, obtains the metadata information of the corresponding table names and column names, encapsulates the data list according to the metadata information, and feeds back the data list and operation status to the front end in JSON format.

[0048] Create a library table module to create a library table: automatically detect whether the library table exists, if it exists, update the table structure, if not, automatically build the data table structure according to the configured basic information of the library table resources,

[0049] Import data module imports data: sends a request to the file server, obtains the Excel file by passing in the unique identifier of the file, parses the file content, generates and executes SQL statements to import data in batches,

[0050] Create library table resource module to create library table resources:

[0051] According to the configured information resource directory, the imported data is associated with the corresponding directory.

[0052] Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table.

[0053] Storing metadata information: storing basic information of library and table resources in the metadata system.

[0054] Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

[0055] Furthermore, the basic information configuration module of the library table resource of the device for automatically generating library table resources based on data collection configures the basic information of the library table resource through the front end, including:

[0056] On the basic information page of the library and table resources, define a request mapping named / register through the @RequestMapping annotation to configure the registration request for the basic information of the resource.

[0057] Get dictionary data: Use the platformService service to get dictionary data and store the dictionary data in the model so that it can be displayed on the page.

[0058] Set the sharing type: call catalogService to get the list of sharing types for the catalog, and add the list of sharing types to the model.

[0059] Set the open type: Get the open type list of the catalog through catalogService and store the open type list in the model.

[0060] Set page attributes: Set page-related attributes to the model.

[0061] Return view: Specify and return the view named resource / filetable / register through the this.getView() method, and pass the model to the view for page rendering.

[0062] Furthermore, the data file uploading module of the device for automatically generating library table resources based on data collection comprises the following steps:

[0063] Get remote file: Get the Excel data file uploaded to the file server and save it to a local temporary folder.

[0064] Parse Excel data files: Use parsing tools to parse Excel data files and extract metadata information from the files.

[0065] Convert and organize data: Convert the metadata information obtained by parsing, including converting the pinyin of file names to table names and column names to pinyin.

[0066] Display data: Display the processed metadata information in the form of a list, and set the number of rows to display as needed.

[0067] Error handling and logging: During the process of file acquisition, parsing and processing, any abnormal situations that occur are captured and handled, and the relevant error information is recorded in the log.

[0068] Output result: Return the data to the front-end page in JSON format.

[0069] Further, the library table creation module of the device for automatically generating library table resources based on data collection creates a library table, specifically including:

[0070] Get parameters: Get parameters from the creation request, including file ID, file name, directory ID, whether to rebuild the table, table name, data source ID, and column information.

[0071] Preprocessing: Add creation time and update time as additional columns to the column information list.

[0072] Directory and organization verification: query directory information based on directory ID, obtain directory title and organization code, check whether the logged-in user is bound to the organization, and return an error message if not bound.

[0073] Database operation: Get database connection information based on the data source ID, determine whether the table exists, and decide whether to create a new table, rebuild the table, or update the data based on the isdrop parameter. If the table already exists and needs to be rebuilt, delete the original table first and then create a new table.

[0074] If the table needs to be updated, check and adjust the primary key to ensure that the table structure is consistent with the column information passed in.

[0075] Create or update the table structure, use the database.createTableSql method to generate the table creation SQL and execute it.

[0076] Metadata management: After successfully creating a table, call the service method to record the metadata information of the table.

[0077] Furthermore, the data importing module of the device for automatically generating library table resources based on data acquisition imports data, including:

[0078] Use the HTTP client to send a request to the file server and obtain the Excel file by passing in the unique file identifier.

[0079] Parse the downloaded Excel file and convert the data into a data structure that can be directly stored in the database (List <Map<String,Object> >),

[0080] Identify column names: automatically remove the first row of Excel as column names, adapt to Excel data headers in different formats,

[0081] Verify data: Perform basic verification on data before importing, remove invalid data rows or data rows that do not meet database constraints, and ensure data quality.

[0082] Dynamic column matching: query the database table structure based on the parsed column information to achieve intelligent matching between data columns and database fields.

[0083] Batch insert: Use the database batch insert API to write data into the database at one time or in multiple batches.

[0084] Status feedback: Provides detailed import result feedback, including the number of successfully imported data rows, error codes, and prompt information, so that users can monitor the import progress.

[0085] The benefits of the method of the present invention are:

[0086] Improved efficiency: Automated processing of file-based data avoids the tedious process of traditional manual conversion of library and table resources, greatly improving data processing speed and the work efficiency of government agencies.

[0087] Data consistency and accuracy: Through scientific and reasonable data format design and automated data verification, the consistency and accuracy of the stored data are guaranteed, and the quality of data governance is improved.

[0088] Flexibility and compatibility: Supports multiple document formats and database types, including domestic databases, which enhances the flexibility and compatibility of the system and meets the needs of different government systems.

[0089] Resource management optimization: The newly added resource registration method entrance and multi-dimensional management functions provide government officials with more convenient and efficient resource management methods, and improve the standardization and transparency of resource management.

[0090] Domestication support: Support for domestication of databases has promoted independent control of information technology and enhanced data security.

[0091] Dynamic data processing capability: supports full coverage and incremental addition of data, ensuring the real-time and integrity of data and meeting data update requirements in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0093] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0094] Example 1

[0095] The present invention provides a method for automatically generating library table resources based on data collection, comprising:

[0096] Step 1: Configure the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition, and the resource location.

[0097] Step 2: Upload Excel data files: Upload Excel data files in various formats to the file server, parse the data files, obtain the metadata information of the corresponding table names and column names, encapsulate the data list according to the metadata information, and feed back the data list and operation status to the front end in JSON format.

[0098] Step 3: Create a library table: Automatically detect whether the library table exists. If it exists, update the table structure. If it does not exist, automatically build the data table structure according to the configured basic information of the library table resources.

[0099] Step 4: Import data: Send a request to the file server, obtain the Excel file by passing in the unique file identifier, parse the file content, generate and execute SQL statements to import data in batches.

[0100] Step 5: Create library table resources:

[0101] According to the configured information resource directory, the imported data is associated with the corresponding directory, so that users can find and manage related library table resources through the directory.

[0102] Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table.

[0103] Storing metadata information: The basic information of library and table resources is stored in the metadata system. The metadata system is a system used to store and manage data resource metadata. It can provide users with a unified data resource view to help users quickly find the data resources they need.

[0104] Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

[0105] The present invention can also adapt to the parsing function of various document formats, mainly including Excel2003, Excel2007, csv and other file formats. At the same time, it supports the creation of library tables in various databases, not only supporting traditional databases such as mysql and oracle, but also supporting domestic databases, including domestic databases such as DAMO and HANGO.

[0106] Example 2

[0107] Based on Example 1, in step 1, the user can configure the basic information of the resource, including the resource name, the associated information resource directory, the shared open attribute definition, and the resource location. The configuration of this information is intended to ensure the compliance and traceability of the resource. For example, the resource name should be unique so that it can be identified and managed in the system; the associated information resource directory helps users quickly find related resources; the shared open attribute definition can control resource access rights and ensure data security; the resource location setting helps users quickly locate resources.

[0108] The basic information of the library and table resources configured by the front end can further include:

[0109] On the basic information page of the library and table resources, define a request mapping named / register through the @RequestMapping annotation to configure the registration request for the basic information of the resource.

[0110] Get dictionary data: Use the platformService service to get dictionary data of a specific type (catalog_data_format) and store the dictionary data in the model so that it can be displayed on the page.

[0111] Set the sharing type: call catalogService to get the list of sharing types for the catalog, and add the list of sharing types to the model.

[0112] Set the open type: Get the open type list of the catalog through catalogService and store the open type list in the model.

[0113] Set page properties: Set page-related properties, such as canDelColumn, auditValidation, innerShare, showFrontMsg, and rcserviceOutterUrl, to the model.

[0114] Return view: Specify and return the view named resource / filetable / register through the this.getView() method, and pass the model to the view for page rendering.

[0115] Example 3

[0116] Based on Example 1, when uploading the data file in step 2, after obtaining the metadata information, check whether there is an empty value in the first row and exclude the corresponding column, retain the first 10 rows of data and remove the empty value column and encapsulate it into a list, and finally feedback the table name, the processed data list and the operation status, such as success or error information, to the front end in JSON format to facilitate user confirmation and further operation. It can further include:

[0117] Get remote file: Get the Excel data file uploaded to the file server and save it to a local temporary folder.

[0118] Parse Excel data files: Use parsing tools such as EasyExcelUtil to parse Excel data files and extract metadata information from the files.

[0119] Convert and organize data: Convert the metadata information obtained by parsing, including converting the pinyin of file names to table names and column names to pinyin.

[0120] Display data: Display the processed metadata information in the form of a list, and set the number of rows to display as needed.

[0121] Error handling and logging: During the process of file acquisition, parsing and processing, any abnormal situations that occur are captured and handled, and the relevant error information is recorded in the log.

[0122] Output result: Return the data to the front-end page in JSON format.

[0123] Example 4

[0124] Based on Example 1, when creating the library table in step 3, the metadata can be synchronized to the metadata system. For example, if the user selects the MySQL database, the system will automatically create the corresponding data table according to the user configuration and store the table structure information in the metadata system.

[0125] Furthermore, step 3 may further specifically include:

[0126] Get parameters: Get parameters from the creation request, including file ID, file name, directory ID, whether to rebuild the table, table name, data source ID, and column information.

[0127] Preprocessing: Add creation time and update time as additional columns to the column information list.

[0128] Directory and organization verification: query directory information based on directory ID, obtain directory title and organization code, check whether the logged-in user is bound to the organization, and return an error message if not bound.

[0129] Database operation: Get database connection information based on the data source ID, determine whether the table exists, and decide whether to create a new table, rebuild the table, or update the data based on the isdrop parameter. If the table already exists and needs to be rebuilt, delete the original table first and then create a new table.

[0130] If the table needs to be updated, check and adjust the primary key to ensure that the table structure is consistent with the column information passed in.

[0131] Create or update the table structure, use the database.createTableSql method to generate the table creation SQL and execute it.

[0132] Metadata management: After successfully creating a table, call the service method to record the metadata information of the table.

[0133] Example 5

[0134] Based on Example 1, data is imported in step 4 to optimize import efficiency and support full and incremental data processing. Whether it is full coverage or incremental appending, the integrity and accuracy of the data can be guaranteed.

[0135] Step 4 may further include:

[0136] Use the HTTP client to send a request to the file server and obtain the Excel file by passing in the unique file identifier (doc_id).

[0137] Use custom tools such as EasyExcelUtil.getData to parse the downloaded Excel file and convert the data into a data structure that can be directly stored in the database (List <Map<String,Object> >),

[0138] Identify column names: automatically remove the first row of Excel as column names, adapt to Excel data headers in different formats,

[0139] Verify data: Perform basic verification on data before importing, remove invalid data rows or data rows that do not meet database constraints, and ensure data quality.

[0140] Dynamic column matching: query the database table structure based on the parsed column information to achieve intelligent matching between data columns and database fields.

[0141] Batch insert: Use the database batch insert API to write data into the database at one time or in multiple batches.

[0142] Status feedback: Provides detailed import result feedback, including the number of successfully imported data rows, error codes, and prompt information, so that users can monitor the import progress.

[0143] Example 6

[0144] The present invention also provides a device for automatically generating library table resources based on data collection, including a library table resource basic information configuration module, a data file upload module, a library table creation module, a data import module and a library table resource creation module.

[0145] The basic information configuration module of the library and table resources configures the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition and the resource location.

[0146] The data file upload module uploads Excel data files: uploads Excel data files in various formats to the file server, parses the data files, obtains the metadata information of the corresponding table names and column names, encapsulates the data list according to the metadata information, and feeds back the data list and operation status to the front end in JSON format.

[0147] Create a library table module to create a library table: automatically detect whether the library table exists, if it exists, update the table structure, if not, automatically build the data table structure according to the configured basic information of the library table resources,

[0148] Import data module imports data: sends a request to the file server, obtains the Excel file by passing in the unique identifier of the file, parses the file content, generates and executes SQL statements to import data in batches,

[0149] Create library table resource module to create library table resources:

[0150] According to the configured information resource directory, the imported data is associated with the corresponding directory.

[0151] Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table.

[0152] Storing metadata information: storing basic information of library and table resources in the metadata system.

[0153] Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

[0154] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0155] Likewise, the benefits of the device of the present invention are:

[0156] Improved efficiency: Automated processing of file-based data avoids the tedious process of traditional manual conversion of library and table resources, greatly improving data processing speed and the work efficiency of government agencies.

[0157] Data consistency and accuracy: Through scientific and reasonable data format design and automated data verification, the consistency and accuracy of the stored data are guaranteed, and the quality of data governance is improved.

[0158] Flexibility and compatibility: Supports multiple document formats and database types, including domestic databases, which enhances the flexibility and compatibility of the system and meets the needs of different government systems.

[0159] Resource management optimization: The newly added resource registration method entrance and multi-dimensional management functions provide government officials with more convenient and efficient resource management methods, and improve the standardization and transparency of resource management.

[0160] Domestication support: Support for domestication of databases has promoted independent control of information technology and enhanced data security.

[0161] Dynamic data processing capability: supports full coverage and incremental addition of data, ensuring the real-time and integrity of data and meeting data update requirements in different scenarios.

[0162] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0163] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for automatically generating library table resources based on data collection, characterized in that include: Step 1: Configure the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition, and the resource location. Step 2: Upload Excel data files: upload Excel data files in various formats to the file server, parse the data files, obtain the metadata information of the corresponding table names and column names, encapsulate the data list according to the metadata information, and feed back the data list and operation status to the front end in JSON format. Step 3: Create a library table: Automatically detect whether the library table exists. If it exists, update the table structure. If it does not exist, automatically build the data table structure according to the configured basic information of the library table resources. Step 4: Import data: Send a request to the file server, obtain the Excel file by passing in the unique file identifier, parse the file content, generate and execute SQL statements to import data in batches, Step 5: Create library table resources: According to the configured information resource directory, the imported data is associated with the corresponding directory. Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table. Storing metadata information: storing basic information of library and table resources in the metadata system. Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

2. The method for automatically generating library table resources based on data collection according to claim 1 is characterized in that step 1: Configure the basic information of the library and table resources through the front end, including: On the basic information page of the library and table resources, define a request mapping named / register through the @RequestMapping annotation to configure the registration request for the basic information of the resource. Get dictionary data: Use the platformService service to get dictionary data and store the dictionary data in the model so that it can be displayed on the page. Set the sharing type: call catalogService to get the list of sharing types for the catalog, and add the list of sharing types to the model. Set the open type: Get the open type list of the catalog through catalogService and store the open type list in the model. Set page attributes: Set page-related attributes to the model. Return view: Specify and return the view named resource / filetable / register through the this.getView() method, and pass the model to the view for page rendering.

3. The method for automatically generating library table resources based on data collection according to claim 1 is characterized in that In step 2, the specific steps include: Get remote file: Get the Excel data file uploaded to the file server and save it to a local temporary folder. Parse Excel data files: Use parsing tools to parse Excel data files and extract metadata information from the files. Convert and organize data: Convert the metadata information obtained by parsing, including converting the pinyin of file names to table names and column names to pinyin. Display data: Display the processed metadata information in the form of a list, and set the number of rows to display as needed. Error handling and logging: During the process of file acquisition, parsing and processing, any abnormal situations that occur are captured and handled, and the relevant error information is recorded in the log. Output result: Return the data to the front-end page in JSON format.

4. The method for automatically generating library table resources based on data collection according to claim 1 is characterized in that In step 3, create a library table, including: Get parameters: Get parameters from the creation request, including file ID, file name, directory ID, whether to rebuild the table, table name, data source ID, and column information. Preprocessing: Add creation time and update time as additional columns to the column information list. Directory and organization verification: query directory information based on directory ID, obtain directory title and organization code, check whether the logged-in user is bound to the organization, and return an error message if not bound. Database operation: Get database connection information based on the data source ID, determine whether the table exists, and decide whether to create a new table, rebuild the table, or update the data based on the isdrop parameter. If the table already exists and needs to be rebuilt, delete the original table first and then create a new table. If the table needs to be updated, check and adjust the primary key to ensure that the table structure is consistent with the column information passed in. Create or update the table structure, use the database.createTableSql method to generate the table creation SQL and execute it. Metadata management: After successfully creating a table, call the service method to record the metadata information of the table.

5. The method for automatically generating library table resources based on data collection according to claim 1 is characterized in that the step of importing data in step 4 comprises: Use the HTTP client to send a request to the file server and obtain the Excel file by passing in the unique file identifier. Parse the downloaded Excel file and convert the data into a data structure that can be directly stored in the database (List <Map<String,Object> >), Identify column names: automatically remove the first row of Excel as column names, adapt to Excel data headers in different formats, Verify data: Perform basic verification on data before importing, remove invalid data rows or data rows that do not meet database constraints, and ensure data quality. Dynamic column matching: Based on the parsed column information, query the database table structure to achieve intelligent matching between data columns and database fields. Batch insert: Use the database batch insert API to write data into the database at one time or in multiple batches. Status feedback: Provides detailed import result feedback, including the number of successfully imported data rows, error codes, and prompt information, so that users can monitor the import progress.

6. A device for automatically generating library table resources based on data collection, characterized in that It includes basic information configuration module of library and table resources, data file upload module, library and table creation module, data import module and library and table resource creation module. The basic information configuration module of the library and table resources configures the basic information of the library and table resources through the front end. The basic information of the library and table resources includes the resource name, the associated information resource directory, the shared open attribute definition and the resource location. The data file upload module uploads Excel data files: uploads Excel data files in various formats to the file server, parses the data files, obtains the metadata information of the corresponding table names and column names, encapsulates the data list according to the metadata information, and feeds back the data list and operation status to the front end in JSON format. Create a library table module to create a library table: automatically detect whether the library table exists, if it exists, update the table structure, if not, automatically build the data table structure according to the configured basic information of the library table resources, Import data module imports data: sends a request to the file server, obtains the Excel file by passing in the unique identifier of the file, parses the file content, generates and executes SQL statements to import data in batches, Create library table resource module to create library table resources: According to the configured information resource directory, the imported data is associated with the corresponding directory. Set resource properties: Set corresponding properties for each library table resource, including the name, description, and creation time of the data table. Storing metadata information: storing basic information of library and table resources in the metadata system. Generate and manage library and table resources: Generate a library and table resource management system, which includes all imported data tables and their associated directories and metadata information. Data resources are managed and queried through the system.

7. The device for automatically generating library table resources based on data collection according to claim 6 is characterized in that the library table The resource basic information configuration module configures the basic information of the library table resources through the front-end, including: On the basic information page of the library and table resources, define a request mapping named / register through the @RequestMapping annotation to configure the registration request for the basic information of the resource. Get dictionary data: Use the platformService service to get dictionary data and store the dictionary data in the model so that it can be displayed on the page. Set the sharing type: call catalogService to get the list of sharing types for the catalog, and add the list of sharing types to the model. Set the open type: Get the open type list of the catalog through catalogService and store the open type list in the model. Set page attributes: Set page-related attributes to the model. Return view: Specify and return the view named resource / filetable / register through the this.getView() method, and pass the model to the view for page rendering.

8. The device for automatically generating library table resources based on data collection according to claim 6 is characterized in that Upload data file module, the execution steps include: Get remote file: Get the Excel data file uploaded to the file server and save it to a local temporary folder. Parse Excel data files: Use parsing tools to parse Excel data files and extract metadata information from the files. Convert and organize data: Convert the metadata information obtained by parsing, including converting the pinyin of file names to table names and column names to pinyin. Display data: Display the processed metadata information in the form of a list, and set the number of rows to display as needed. Error handling and logging: During the process of file acquisition, parsing and processing, any abnormal situations that occur are captured and handled, and the relevant error information is recorded in the log. Output result: Return the data to the front-end page in JSON format.

9. The device for automatically generating library table resources based on data collection according to claim 1 is characterized in that Create a library table module to create a library table, including: Get parameters: Get parameters from the creation request, including file ID, file name, directory ID, whether to rebuild the table, table name, data source ID, and column information. Preprocessing: Add creation time and update time as additional columns to the column information list. Directory and organization verification: query directory information based on directory ID, obtain directory title and organization code, check whether the logged-in user is bound to the organization, and return an error message if not bound. Database operation: Get database connection information based on the data source ID, determine whether the table exists, and decide whether to create a new table, rebuild the table, or update the data based on the isdrop parameter. If the table already exists and needs to be rebuilt, delete the original table first and then create a new table. If the table needs to be updated, check and adjust the primary key to ensure that the table structure is consistent with the column information passed in. Create or update the table structure, use the database.createTableSql method to generate the table creation SQL and execute it. Metadata management: After successfully creating a table, call the service method to record the metadata information of the table.

10. The device for automatically generating library table resources based on data collection according to claim 1 is characterized in that: The data module imports data, including: Use the HTTP client to send a request to the file server and obtain the Excel file by passing in the unique file identifier. Parse the downloaded Excel file and convert the data into a data structure that can be directly stored in the database (List <Map<String,Object> >), Identify column names: automatically remove the first row of Excel as column names, adapt to Excel data headers in different formats, Verify data: Perform basic verification on data before importing, remove invalid data rows or data rows that do not meet database constraints, and ensure data quality. Dynamic column matching: Based on the parsed column information, query the database table structure to achieve intelligent matching between data columns and database fields. Batch insert: Use the database batch insert API to write data into the database at one time or in multiple batches. Status feedback: Provides detailed import result feedback, including the number of successfully imported data rows, error codes, and prompt information, so that users can monitor the import progress.

Images