File data access method and device based on metadata, equipment and storage medium

Abstract

The invention discloses a file data access method and device based on metadata, equipment and a storage medium, and the method comprises the steps: analyzing file access task configuration information, and obtaining data source connection information, data storage configuration information and personalized rule configuration information required in a file access task execution process; constructing a file connection channel according to the data source connection information; reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data; and establishing connection with a database according to the storage rule configuration information and preset metadata configuration information, and updating the first file data into the database. According to the method, the common content and the personalized content of the file data access class task are separated through the template thought, personalized data processing is supported based on simple configuration, the development period is shortened, and the development cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a metadata-based file data access method, device, equipment and storage medium. Background Art

[0002] With the rapid development of information technology, the processing requirements of business data are becoming increasingly complex and diverse. As business data increases, the demand for data processing in the form of files and output to different databases is increasing. However, the processing logic of different file access tasks is different, and the same file may also have different processing logic and output database types due to different requirements.

[0003] Current file data access technologies generally use hard-coding to develop specific file access tasks. When there are new file access task requirements, developers need to rewrite the code, which not only increases development costs and maintenance difficulties, but also prolongs the development cycle and cannot quickly respond to changes in business needs. Secondly, once business requirements change, or new file types or databases need to be accessed, the original code needs to be extensively modified and debugged, which undoubtedly increases the difficulty of system upgrades and maintenance, and limits the flexibility and scalability of the system.

[0004] In addition, traditional file data access solutions usually require developers or operation and maintenance personnel to manually re-execute when file access fails, and the file task access status also requires manual confirmation, which greatly relies on manual operations and reduces work efficiency.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a metadata-based file data access method, device, equipment and storage medium, aiming to solve the technical problems of the traditional file data access solution using hard coding, resulting in repeated coding and difficulty in adapting to the ever-changing data processing needs.

[0007] To achieve the above object, the present application provides a metadata-based file data access method, the metadata-based file data access method comprising the following steps:

[0008] Parse the file access task configuration information to obtain the data source connection information, data storage configuration information, and personalized rule configuration information required during the execution of the file access task;

[0009] Building a file connection channel according to the data source connection information;

[0010] Reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0011] According to the storage rule configuration information and the preset metadata configuration information, a connection with a database is established, and the first file data is updated in the database.

[0012] In one embodiment, the step of parsing the file access task configuration information and obtaining the data source connection information, data storage configuration information and personalized rule configuration information required in the process of executing the file access task includes:

[0013] Read and parse file access task configuration information from the storage location;

[0014] Obtain relevant metadata according to the file access task configuration information;

[0015] extracting and verifying data source connection information from the relevant metadata;

[0016] Extract data storage configuration information from file access task configuration;

[0017] The personalized rule configuration information is obtained by dragging and dropping the personalized rule configuration on the front end.

[0018] In one embodiment, before the step of parsing the file access task configuration information and obtaining the data source connection information, data storage configuration information and personalized rule configuration information required in the process of executing the file access task, the following steps are further included:

[0019] Perform metadata configuration in a preset metadata management module to obtain metadata configuration information;

[0020] The file access task configuration is performed based on the metadata configuration information to obtain the file access task configuration information.

[0021] In one embodiment, after the step of constructing a file connection channel according to the data source connection information, the step further includes:

[0022] Checking whether the target file exists through the file connection channel;

[0023] If the target file exists, the following steps are executed: reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0024] If the target file does not exist, the current execution process is terminated, and a determination is made based on a preset maximum number of retries whether to retry the file access task.

[0025] In one embodiment, before the step of reading the file data through the file connection channel and performing personalized processing on the file data based on the personalized rule configuration information to obtain the first file data, the step further includes:

[0026] The file data is personalized pre-processed based on the personalized rule configuration information.

[0027] In one embodiment, before or after the step of building a file connection channel according to the data source connection information, the step further includes:

[0028] Get the alarm rules corresponding to the file access task;

[0029] The execution log of the file access task is obtained according to the preset monitoring interval, and if the execution log hits the alarm rule, an alarm is triggered.

[0030] In one embodiment, the step of establishing a connection with a database according to the storage rule configuration information and the preset metadata configuration information, and updating the first file data into the database further includes:

[0031] Get file execution logs and store them in the database.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a metadata-based file data access device, the metadata-based file data access device comprising:

[0033] The acquisition module is used to parse the file access task configuration information and obtain the data source connection information, data storage configuration information and personalized rule configuration information required in the execution of the file access task;

[0034] A construction module, used to construct a file connection channel according to the data source connection information;

[0035] A data processing module, configured to read file data through the file connection channel, and perform personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0036] An updating module is used to establish a connection with a database according to the storage rule configuration information and the preset metadata configuration information, and to update the first file data into the database.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal device, which includes a memory, a processor, and a metadata-based file data access program stored on the memory and executable on the processor, and the metadata-based file data access program, when executed by the processor, implements the steps of the metadata-based file data access method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a metadata-based file data access program is stored. When the metadata-based file data access program is executed by a processor, the steps of the metadata-based file data access method as described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] This application separates the common content and personalized content of file data access tasks through the template concept, and automatically performs file data access processing, handling and output based on simple configuration.

[0041] First, parse the file access task configuration information. By dynamically reading and parsing the configuration information, the data source connection information, data storage configuration information, and personalized rule configuration information required during the execution of the file access task are obtained and cached. This avoids the traditional hard-coding method and improves the flexibility and scalability of the system. Build a file connection channel based on the data source connection information, then read the file data through the file connection channel, and personalize the file data based on the personalized rule configuration information. This process allows users to customize the data processing logic according to actual needs, enhancing the flexibility and accuracy of data processing. Finally, establish a connection with the database based on the storage rule configuration information and the preset metadata configuration information, and update the processed first file data to the database. Based on the metadata configuration information and the storage rule configuration information, it supports multiple data source output after file data processing.

[0042] In summary, compared with the existing technology, this technical solution achieves the following effects: First, it reduces development costs and maintenance difficulties, and there is no need to hard-code each file access task; second, it automatically performs file task access logic processing according to metadata configuration, and supports personalized data processing, thereby improving data processing efficiency and quality; third, based on metadata configuration information and storage rule configuration information, it supports multiple data source output after file data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flowchart of a first exemplary embodiment of a metadata-based file data access method of the present application;

[0044] Figure 2 This is a flow chart of a second exemplary embodiment of the metadata-based file data access method of the present application;

[0045] Figure 3 This is a flowchart of a third exemplary embodiment of the metadata-based file data access method of the present application;

[0046] Figure 4 This is a flowchart of a fourth exemplary embodiment of the metadata-based file data access method of the present application;

[0047] Figure 5 This is a flowchart of a fifth exemplary embodiment of the metadata-based file data access method of the present application;

[0048] Figure 6 This is a flowchart of a sixth exemplary embodiment of the metadata-based file data access method of the present application;

[0049] Figure 7 This is a schematic diagram of the module structure of a file data access device based on metadata according to an embodiment of the present application;

[0050] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the metadata-based file data access method in the embodiment of the present application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The main solution of the embodiment of the present application is: parsing the file access task configuration information, obtaining the data source connection information, data storage configuration information and personalized rule configuration information required in the execution of the file access task; constructing a file connection channel according to the data source connection information; reading the file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain the first file data; establishing a connection with the database according to the storage rule configuration information and the preset metadata configuration information, and updating the first file data to the database. Compared with the hard-coded method, which leads to many repeated coding, high development costs and maintenance difficulties, and limited system flexibility and scalability, the common content and personalized content of file data access tasks are separated through the template idea, and file data access processing, processing and output are automatically performed based on simple configuration, supporting personalized data processing, greatly shortening the development cycle and reducing development costs.

[0054] The executor of the method of this embodiment can be a metadata-based file data access device, or a metadata-based file data access terminal device or server. This embodiment takes a metadata-based file data access device as an example. The metadata-based file data access device can be integrated in terminal devices such as smart phones and tablet computers with data processing functions.

[0055] Specifically, the following are detailed steps of an exemplary embodiment of the metadata-based file data access method:

[0056] Reference Figure 1 , Figure 1 This is a flowchart of a first exemplary embodiment of a metadata-based file data access method of the present application. In this embodiment, the metadata-based file data access method includes steps S10 to S40:

[0057] Step S10, parsing the file access task configuration information to obtain the data source connection information, data storage configuration information and personalized rule configuration information required during the execution of the file access task;

[0058] In a feasible implementation, step S10 may include steps S11 to S15:

[0059] Step S11, reading and parsing file access task configuration information from a storage location;

[0060] Specifically, first determine the storage location of the configuration information, which may be a database, configuration file, or remote server. Once the storage location is determined, the file access task configuration information can be read through corresponding technical means (such as database query statements, file reading operations, etc.). File access task configuration information is usually stored in formats such as JSON, XML, or YAML, and contains key content such as basic task information, data source connection details, data entry rules, and personalized processing rules. After reading the file access task configuration information, it needs to be parsed and converted into a data structure or object that can be recognized within the system for use in subsequent steps. For example, if the file access task configuration information is stored in JSON format, a JSON parser can be used to convert it into a corresponding object or data structure.

[0061] Step S12, acquiring relevant metadata according to the file access task configuration information;

[0062] After parsing the file access task configuration information, relevant metadata is obtained and cached based on this information. Metadata plays a very important role in the file data access solution. It contains key content such as data asset information, data source connection information, and data asset field information. In order to obtain the metadata related to this file access task, that is, the relevant metadata in step S12, first retrieve the corresponding metadata items from the metadata warehouse according to the metadata identifier or path in the configuration information. Then, these metadata items are cached in the memory or intermediate database for quick access in subsequent steps. The purpose of caching metadata is to improve the efficiency of data processing and avoid repeated queries to the metadata warehouse during the processing process. For example, if the data asset information of a data source is specified in the configuration information, all field information of the data source can be retrieved from the metadata warehouse and cached in the memory or intermediate database for subsequent data processing operations.

[0063] Step S13, extracting and verifying data source connection information from the relevant metadata;

[0064] Specifically, after the relevant metadata is cached, the data source connection information is extracted and verified from the relevant metadata. First, the data source connection information is extracted from the cached relevant metadata. This information usually includes key contents such as the address, port, user name and password of the data source. Then, the connection information is verified to ensure its integrity and correctness. The verification process may include checking whether the format of the connection information meets the requirements, trying to use this information to establish a connection with the data source, etc. If the connection information is valid and the connection can be successfully established, the connection information can be cached for subsequent data reading operations. If the connection information is invalid or the connection cannot be established, the corresponding error information can be recorded, and corresponding measures (such as re-obtaining the connection information, notifying the administrator, etc.) can be taken to handle these problems.

[0065] Step S14, extracting data storage configuration information from the file access task configuration;

[0066] Specifically, the data entry configuration information defines how to write the processed data into the target database. In order for subsequent data entry operations to proceed smoothly, it is necessary to extract the data entry configuration information from the file access task configuration information and cache it in memory. The process of extracting data entry configuration information is similar to the process of extracting data source connection information, which is achieved by parsing the configuration information. The purpose of caching data entry configuration information is to improve the efficiency of data entry and avoid re-reading and parsing the configuration information each time data is entered.

[0067] Step S15, configuring personalized rules by dragging and dropping on the front end to obtain personalized rule configuration information.

[0068] It should be noted that the personalized rule configuration allows users to configure personalized processing rules by dragging and dropping on the front-end interface according to actual needs. These personalized processing rules define the personalized processing operations that need to be performed during the access process of file data.

[0069] Specifically, in order to implement the personalized rule configuration function, the system will first provide a friendly front-end interface, which contains various available personalized processing rule components (such as data cleaning components, data conversion components, etc.). Users can add these components to the configuration interface by dragging and dropping, and configure the parameters of each component according to actual needs. After the configuration is completed, the system will convert these personalized rule configuration information into corresponding drools rule codes and store them in the database. In the subsequent file access task execution process, these personalized rule configuration information can be read from the database, and the file data can be personalized accordingly based on this information. For example, if the user configures a data cleaning rule by dragging and dropping to remove null value data, then in the subsequent file data access process, the read data will be cleaned according to this rule to remove the null value data.

[0070] Step S20, constructing a file connection channel according to the data source connection information;

[0071] For example, if the data source connection information points to an Excel file located on a remote server, then it is necessary to access the server through a network request. Specifically, the remote server is located using information such as the server address and port number, and then the provided username and password are used for authentication to ensure that the target file has access. Once the authentication is passed, the specific Excel file can be further located according to the file path, thereby building a stable and secure file connection channel.

[0072] This file connection channel is not only used to read file data, but will also remain open during subsequent processing so that the system can access and operate the file again at any time as needed.

[0073] Step S30, reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0074] Specifically, after successfully building the file connection channel, the file data is read through the file connection channel. Select an appropriate reading method according to the type and format of the file. For example, for Excel files, a special library function can be used to parse and read the data in the table; for text files, a line-by-line reading method can be used for processing.

[0075] After reading the file data, the file data is personalized according to the personalized rule configuration information obtained in the above steps. Personalized rules can include multiple operations such as data filtering, format conversion, and field mapping. For example, a user may want to extract only the rows that meet specific conditions in an Excel file, or convert the data type of certain fields from strings to numbers. Taking an Excel file as an example, suppose that the user wants to extract only the records where "sales" is greater than 10,000, and convert the "date" field format of these records from "YYYY-MM-DD" to "YYYYMMDD". In this case, first filter out the records where "sales" is greater than 10,000, and then perform format conversion on the "date" field of these records. After such personalized processing, a new data set that meets the user's needs can be obtained, namely the first file data. This data set not only contains the information that the user cares about, but also presents it in the format expected by the user, thereby greatly improving the availability and ease of use of the data.

[0076] It should be noted that after the file data is personalized based on the personalized rule configuration information to obtain the first file data, personalized post-processing may also be performed on the first file data.

[0077] Specifically, the first file data may be subjected to a quality check. This check process includes verifying the integrity and consistency of the data and whether it complies with the expected format and standards. For example, rules may be set to check for null values, duplicate values, or abnormal values ​​in the data, and process or mark them accordingly.

[0078] You can also generate a processing report. The processing report contains key information about the processing process, such as how many data were processed, which data were successfully or failed to be processed, and errors or warnings that occurred during the processing. The report can be presented in the form of text, table or chart, so that users can intuitively understand the processing results and existing problems.

[0079] Data archiving or backup operations can also be performed to ensure that the processed data can be safely stored and accessed. Specifically, key information such as the first file data and processing reports can be archived in a designated storage area, and corresponding access rights and backup policies can be set. In this way, even if data is lost or damaged in the future, data can be quickly restored and processing can continue.

[0080] Step S40: establishing a connection with a database according to the storage rule configuration information and the preset metadata configuration information, and updating the first file data into the database.

[0081] Specifically, first clarify the contents of the storage rule configuration information and the preset metadata configuration information. The storage rule configuration information usually includes data field mapping rules, data type conversion rules, data verification rules, etc. These rules ensure that the data can meet the storage requirements of the database before storage. The preset metadata configuration information includes database connection information, table structure information, field attribute information, etc., which is the basis for establishing database connection.

[0082] Next, a connection to the target database is established based on this information. This usually involves steps such as selecting a database driver, configuring a connection string, and authenticating a database user. Once the connection is established successfully, data operations can be performed through the API interface provided by the database.

[0083] During the data update process, the consistency and integrity of the data need to be ensured. Therefore, before writing the data into the database, the data is pre-processed according to the storage rule configuration information, including data cleaning, format conversion, data verification and other steps to ensure that the data meets the storage requirements of the database.

[0084] In addition, when implementing data updates, it is also possible to support the output of multiple data sources. This usually involves steps such as data synchronization and data distribution, ensuring that data can be transmitted to each target data source in a timely and accurate manner.

[0085] Take a company's customer information file as an example. This file contains fields such as the customer's name, contact information, and address. When updating this file data to the database, the table structure and field attributes of the target database will first be determined based on the preset metadata configuration information. Then, the file data will be preprocessed according to the warehousing rule configuration information, such as converting the format of the contact phone field from text format to digital format, and verifying it to ensure its validity. Next, establish a connection with the target database, and write the preprocessed data into the database through the INSERT or UPDATE statement provided by the database.

[0086] Further, refer to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the metadata-based file data access method of this application. Figure 1 In the embodiment shown, in this embodiment, before step S10, steps S51 to S52 may be included:

[0087] Step S51, performing metadata configuration in a preset metadata management module to obtain metadata configuration information;

[0088] Specifically, firstly, a metadata management module is designed and developed according to business requirements and technical specifications. The main function of this module is to store, manage and maintain various metadata, including but not limited to file structure information, field type, data format, etc.

[0089] After the metadata management module is created, metadata configuration is performed. According to the actual data source and file type, the corresponding metadata is configured for each file or data set. For example, if you are processing an Excel file containing employee information, you need to configure the corresponding metadata for each field in the file (such as employee ID, name, position, etc.), including field name, data type (such as string, number, etc.), field length, etc. These metadata configuration information will be stored in the metadata management module to provide necessary reference and basis for subsequent file access tasks.

[0090] Step S52: Perform file access task configuration based on the metadata configuration information to obtain file access task configuration information.

[0091] Specifically, first read the corresponding metadata configuration information from the metadata management module. Then, according to the metadata configuration information, configure the corresponding data source connection information, data storage configuration information and personalized rule configuration information for each file access task. For example, for the above-mentioned employee information table file access task, configure the data source connection information to specify the storage location and format of the file; configure the data storage configuration information to specify which database and table the data should be stored in and the corresponding relationship between each field; configure the personalized rule configuration information to specify any special processing rules or logic (such as data cleaning, conversion, etc.).

[0092] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0093] Further, refer to Figure 3 , Figure 3 This is a flowchart of the third embodiment of the metadata-based file data access method of this application. Figure 1 In the embodiment shown, in this embodiment, step S20 may include steps S61 to S63:

[0094] Step S61, checking whether the target file exists through the file connection channel;

[0095] Specifically, the specified target file location is accessed through the previously constructed file connection channel. In this process, the connection information stored in the file connection channel (such as file path, access rights, etc.) is used to locate and try to open the target file. If the target file exists and is accessible, a success status is recorded and the next step of file data reading and processing is ready to proceed.

[0096] Step S62, if the target file exists, then executing the steps of: reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0097] Specifically, once the target file is confirmed to exist and accessible, the file data will be read through the file connection channel. In this process, the system will select the appropriate reading method based on the file type and format to ensure the integrity and accuracy of the data.

[0098] After reading the file data, the system will process the data based on the previously acquired personalized rule configuration information. These personalized rules may include operations such as data screening, conversion, and formatting, aiming to convert the original file data into the first file data that meets specific requirements.

[0099] Step S63: If the target file does not exist, the current execution process is terminated, and it is determined whether to retry the file access task according to the preset maximum number of retries.

[0100] Specifically, if the target file does not exist or cannot be accessed when trying to access it, the system will record a failure status and immediately terminate the current execution process. However, this does not mean that the entire file access task is over.

[0101] Based on the preset maximum number of retries, the system will determine whether it is necessary to retry the file access task. If the number of retries has not reached the upper limit, the system will try to access the file again after a certain time interval. During this process, the system may check the network connection status, the operation status of the file server, etc. to ensure the success rate of the retry.

[0102] If the number of retries reaches the upper limit and the file cannot be successfully accessed, the system will record a final failure status and may need to send a notification to the administrator or user to inform them of the file access failure.

[0103] Further, refer to Figure 4 , Figure 4 This is a flowchart of the fourth embodiment of the metadata-based file data access method of this application. Figure 1In the embodiment shown, in this embodiment, step S30 may include step S70:

[0104] Step S70: performing personalized pre-processing on the file data based on the personalized rule configuration information.

[0105] It should be noted that the purpose of the above-mentioned personalized pre-processing is to perform pre-processing according to the personalized rule configuration information to meet the needs of subsequent personalized processing.

[0106] Specifically, the data in the target file can be converted to a specified format according to the rule configuration information, for example, converting a date field from the "year-month-day" format to the "year-month-day" format, or converting a numeric field from a string type to a numeric type.

[0107] You can also filter the fields in the target file according to the field list specified in the personalized rule configuration information. Only the fields specified in the list are retained and other unnecessary fields are deleted. This helps to reduce the amount of data to be processed later and improve processing efficiency.

[0108] You can also clean the data in the target file to remove duplicate data, invalid data, or abnormal data. For example, for a phone number field, the system will check whether each phone number conforms to the specified format and delete phone numbers that do not conform to the format. At the same time, the system will also check for missing values ​​or empty values ​​in the data and fill or delete them according to the instructions in the rule configuration information.

[0109] Further, refer to Figure 5 , Figure 5 This is a flowchart of the fifth embodiment of the metadata-based file data access method of this application. Figure 1 In the embodiment shown, in this embodiment, steps S81 to S82 may be included before or after step S20:

[0110] Step S81, obtaining an alarm rule corresponding to the file access task;

[0111] Specifically, the database is first queried for an alarm rule set that matches the current task type, source, priority, and other attributes. These rule sets are usually developed by professional operation and maintenance personnel or data analysts based on historical data and business experience, and are designed to cover all possible abnormal situations.

[0112] After obtaining the alarm rule set, the system will parse and load these rules for real-time monitoring in the subsequent file processing process. These rules may include various situations such as abnormal file size, mismatched file format, and file content containing sensitive information.

[0113] Let's take a specific example to illustrate: suppose the system is processing a financial statement file access task from a financial institution. In step S81, the system will first query the alarm rule set related to the financial institution, which may include rules such as "trigger an alarm when the file size exceeds 10MB" and "trigger an alarm when the file execution time is greater than 120 minutes". The system will load these rules into memory to provide real-time monitoring capabilities for subsequent file processing.

[0114] Step S82: acquiring the execution log of the file access task according to the preset monitoring interval, and triggering an alarm if the execution log matches the alarm rule.

[0115] Specifically, first set a reasonable monitoring interval. This interval is usually determined based on the expected completion time of the file processing task and business experience, to ensure that the system can obtain the execution log at the appropriate time point and make effective alarm judgments.

[0116] In the monitoring interval, the execution logs generated during the file processing are collected in real time. These logs may include timestamps of each stage of file processing, processing results, exception information, etc. The system will parse and organize these logs to match them with the alarm rules obtained previously.

[0117] If the system finds that an execution log hits an alarm rule, it will immediately trigger the alarm mechanism. This mechanism may include sending email notifications, triggering SMS alarms, generating alarm logs, etc. The specific method used depends on the system configuration and business requirements.

[0118] Further, refer to Figure 6 , Figure 6 This is a flowchart of the sixth embodiment of the metadata-based file data access method of this application. Figure 1 In the embodiment shown, in this embodiment, step S40 may include step S90:

[0119] Step S90, obtaining the file execution log and storing it in the database.

[0120] Specifically, it comprehensively retrieves and records all operations during the file data update process, such as file reading, data conversion, library table mapping and writing, and generates detailed file execution logs. Subsequently, these logs will be organized into a structured data format and stored in the specified log data table through the previously established database connection.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the metadata-based file data access method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] In addition, the present application also proposes a metadata-based file data access device, the metadata-based file data access device comprising:

[0123] The acquisition module 10 is used to parse the file access task configuration information and obtain the data source connection information, data storage configuration information and personalized rule configuration information required in the execution process of the file access task;

[0124] A construction module 20, configured to construct a file connection channel according to the data source connection information;

[0125] A data processing module 30, configured to read file data through the file connection channel, and perform personalized processing on the file data based on the personalized rule configuration information to obtain first file data;

[0126] The updating module 40 is used to establish a connection with the database according to the storage rule configuration information and the preset metadata configuration information, and update the first file data into the database.

[0127] The metadata-based file data access device provided by the present application adopts the metadata-based file data access method in the above-mentioned embodiment, aiming to solve the technical problem that the traditional file data access solution adopts a hard-coded method, resulting in repeated coding and difficulty in adapting to the ever-changing data processing requirements. Compared with the prior art, the beneficial effects of the metadata-based file data access device provided by the present application are the same as the beneficial effects of the metadata-based file data access method provided by the above-mentioned embodiment, and the other technical features of the metadata-based file data access device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0128] The present application provides a metadata-based file data access device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the metadata-based file data access method in the above-mentioned embodiment 1.

[0129] Reference below Figure 8, which shows a schematic diagram of the structure of a metadata-based file data access device for implementing an embodiment of the present application. The metadata-based file data access device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The metadata-based file data access device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 8 As shown, the metadata-based file data access device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the metadata-based file data access device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the metadata-based file data access device to communicate wirelessly or wired with other devices to exchange data. Although the metadata-based file data access device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0131] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0132] The metadata-based file data access device provided by the present application adopts the metadata-based file data access method in the above-mentioned embodiment, aiming to solve the technical problem that the traditional file data access solution adopts a hard-coded method, resulting in repeated coding and difficulty in adapting to the ever-changing data processing requirements. Compared with the prior art, the beneficial effects of the metadata-based file data access device provided by the present application are the same as the beneficial effects of the metadata-based file data access method provided by the above-mentioned embodiment, and the other technical features of the metadata-based file data access device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0133] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0135] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the metadata-based file data access method in the above-mentioned embodiment.

[0136] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0137] The computer-readable storage medium may be included in the metadata-based file data access device; or may exist independently without being assembled into the metadata-based file data access device.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0141] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned metadata-based file data access method, aiming to solve the technical problems that the traditional file data access solution adopts a hard-coded method, resulting in repeated coding and difficulty in adapting to the ever-changing data processing requirements. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the metadata-based file data access method provided by the above-mentioned embodiment, and will not be repeated here.

[0142] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned metadata-based file data access method when executed by a processor.

[0143] The computer program product provided by the present application aims to solve the technical problem that the traditional file data access solution adopts a hard-coded method, which leads to repeated coding and is difficult to adapt to the ever-changing data processing requirements. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the metadata-based file data access method provided by the above embodiment, and will not be repeated here.

[0144] Compared with the prior art, the metadata-based file data access method, device, equipment, medium and computer product proposed in the embodiment of the present application extracts the business feature information of the target business, performs data standardization processing on the business feature information to obtain standard feature data, performs hash processing on the standard feature data to obtain unique feature data, performs numerical processing and splicing processing on the unique feature data to obtain the first business feature value, accumulates the first business feature value of the target business to obtain the target business feature value, and finally compares the target business feature value with the feature value set to obtain the metadata-based file data access result. Compared with the traditional method of identifying duplicate businesses by generating a unique key value or a continuous serial number for each business, it is more efficient, flexible and reliable. Based on the solution of the present application, the business of complex scenarios is finally converted into a comparison of two numbers through a series of simple transformations, so that the comparison process is very intuitive and efficient. The system only needs to simply compare whether the two values ​​are equal to quickly determine whether the two businesses are exactly the same.

[0145] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0146] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0148] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A file data access method based on metadata, characterized in that: The metadata-based file data access method includes: Parse the file access task configuration information to obtain the data source connection information, data storage configuration information, and personalized rule configuration information required during the execution of the file access task; Building a file connection channel according to the data source connection information; Reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data; According to the storage rule configuration information and the preset metadata configuration information, a connection with a database is established, and the first file data is updated in the database.

2. The metadata-based file data access method according to claim 1, characterized in that: The step of parsing the file access task configuration information to obtain the data source connection information, data storage configuration information and personalized rule configuration information required in the process of executing the file access task includes: Read and parse file access task configuration information from the storage location; Obtain relevant metadata according to the file access task configuration information; extracting and verifying data source connection information from the relevant metadata; Extract data storage configuration information from file access task configuration; The personalized rule configuration information is obtained by dragging and dropping the personalized rule configuration on the front end.

3. The metadata-based file data access method according to claim 2, characterized in that: Before the step of parsing the file access task configuration information and obtaining the data source connection information, data storage configuration information and personalized rule configuration information required in the process of executing the file access task, the following steps are also included: Perform metadata configuration in a preset metadata management module to obtain metadata configuration information; The file access task configuration is performed based on the metadata configuration information to obtain the file access task configuration information.

4. The file data access method based on metadata according to claim 1, characterized in that: After the step of constructing a file connection channel according to the data source connection information, the following step further includes: Checking whether the target file exists through the file connection channel; If the target file exists, the following steps are executed: reading file data through the file connection channel, and performing personalized processing on the file data based on the personalized rule configuration information to obtain first file data; If the target file does not exist, the current execution process is terminated, and a determination is made based on a preset maximum number of retries whether to retry the file access task.

5. The file data access method based on metadata according to claim 1, characterized in that: Before the step of reading the file data through the file connection channel and performing personalized processing on the file data based on the personalized rule configuration information to obtain the first file data, the step further includes: The file data is personalized pre-processed based on the personalized rule configuration information.

6. The metadata-based file data access method according to claim 1, characterized in that: Before or after the step of building a file connection channel according to the data source connection information, the following steps may also be included: Get the alarm rules corresponding to the file access task; The execution log of the file access task is obtained according to the preset monitoring interval, and if the execution log hits the alarm rule, an alarm is triggered.

7. The metadata-based file data access method according to claim 1, characterized in that: After the step of establishing a connection with a database according to the storage rule configuration information and the preset metadata configuration information, and updating the first file data into the database, the step further includes: Get file execution logs and store them in the database.

8. A file data access device based on metadata, characterized in that: The device comprises: The acquisition module is used to parse the file access task configuration information and obtain the data source connection information, data storage configuration information and personalized rule configuration information required in the execution of the file access task; A construction module, used to construct a file connection channel according to the data source connection information; A data processing module, configured to read file data through the file connection channel, and perform personalized processing on the file data based on the personalized rule configuration information to obtain first file data; An updating module is used to establish a connection with a database according to the storage rule configuration information and the preset metadata configuration information, and to update the first file data into the database.

9. A file data access device based on metadata, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the metadata-based file data access method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the metadata-based file data access method according to any one of claims 1 to 7 are implemented.

Images