File storage method and device, equipment and medium

Abstract

The invention discloses a file storage method and device, equipment and a medium. The method comprises the following steps: acquiring a to-be-processed file, and determining a processing rule corresponding to the to-be-processed file; processing the to-be-processed file based on the processing rule, and detecting the processing state of the to-be-processed file in real time; if it is detected that the processing state is an abnormal state, data corresponding to the abnormal state is captured and separated from a to-be-processed file, and abnormal data and a normal file are obtained; and respectively processing the abnormal data and the normal file and storing the abnormal data and the normal file in a database. According to the technical scheme, corresponding automatic processing can be performed when abnormal data occurs in file storage, so that normal file storage is realized, the quality of file data is ensured, the continuity and reliability of file data storage are improved, and the cost and time of manual maintenance are reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for file storage. Background Technology

[0002] With the development of information technology, the storage and management of file data has become an indispensable key link in all walks of life. Especially in big data-intensive industries such as finance, e-commerce, and logistics, the processing of file data into the database is a core component of daily operations. For example, in the financial industry, the industry quotation data at the end of each day needs to be accurately entered into the system for subsequent analysis and decision-making.

[0003] However, in practical applications, the file import process often faces numerous challenges. On the one hand, due to the large volume and diverse sources of data, the data in the imported files may contain various anomalies, such as primary key conflicts, format errors, and missing data. In traditional file import processes, once anomalies such as primary key conflicts or data format errors are encountered, the entire import process is often interrupted, causing the system to suspend operation and requiring manual intervention to find and resolve the issues. This not only affects the quality of the file data and the stability and continuity of file import, but also significantly increases the cost and time of manual maintenance. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for file entry into a database. It can automatically process abnormal data when it occurs during file entry, thereby ensuring normal file entry, guaranteeing the quality of file data, improving the continuity and reliability of file data entry, and reducing the cost and time of manual maintenance.

[0005] According to one aspect of the present invention, a method for importing files into a database is provided, comprising:

[0006] Obtain the file to be processed and determine the processing rules corresponding to the file to be processed;

[0007] The file to be processed is processed based on the processing rules, and the processing status of the file to be processed is detected in real time.

[0008] If the processing state is detected to be abnormal, the data corresponding to the abnormal state is captured and separated from the file to be processed to obtain abnormal data and normal file;

[0009] The abnormal data and the normal files are processed separately and stored in the database.

[0010] According to another aspect of the present invention, a document storage device is provided, comprising:

[0011] The data acquisition module is used to acquire the file to be processed and determine the processing rules corresponding to the file to be processed.

[0012] The file processing module is used to process the file to be processed based on the processing rules and to detect the processing status of the file to be processed in real time.

[0013] The data separation module is used to capture the data corresponding to the abnormal state and separate it from the file to be processed if the processing state is detected to be abnormal, so as to obtain abnormal data and normal file.

[0014] The file storage module is used to process the abnormal data and the normal files respectively and store them in the database.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the file import method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the file loading method according to any embodiment of the present invention.

[0020] The technical solution of this invention involves acquiring a file to be processed and determining the processing rules corresponding to that file; processing the file based on the processing rules and monitoring its processing status in real time; if an abnormal processing status is detected, capturing the data corresponding to the abnormal status and separating it from the file to be processed to obtain abnormal data and normal files; processing the abnormal data and normal files separately and storing them in a database. This technical solution enables automated processing when abnormal data occurs during file entry into the database, ensuring normal file entry, guaranteeing file data quality, improving the continuity and reliability of file data entry, and reducing the cost and time of manual maintenance.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a file import method according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a file import method according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of a document entry device according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," and "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1This is a flowchart of a file entry method according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of entering file data into a database. The method can be executed by a file entry device, which can be implemented in hardware and / or software. This file entry device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] The technical solution of this embodiment can be executed by a file import system. This file import system may include modules such as a configuration module, a file processing module, and an exception information storage module. The file import system of this embodiment can efficiently handle exceptions during the file import process, has flexible configuration capabilities, and can customize exception handling strategies according to the needs of different scenarios. Simultaneously, it should also have powerful exception capture and separation capabilities, enabling it to quickly and accurately identify and process abnormal data without affecting normal file import, thus solving many problems existing in current file import systems and improving system stability and efficiency.

[0032] S110. Obtain the file to be processed and determine the processing rules corresponding to the file to be processed.

[0033] In this embodiment, the file to be processed refers to a file that needs to be processed. In this embodiment, the file to be processed refers to a file that needs to be added to the database. The processing rules can be understood as the specific processing flow for the file to be processed. In this embodiment, the processing rules can be pre-configured. The processing rules in this embodiment can include file processing methods, exception handling flows, and specific database insertion methods. The exception handling flow refers to the exception handling strategies corresponding to different types of abnormal data.

[0034] Specifically, in this embodiment, the file to be processed can be obtained, the file name can be obtained, and the corresponding pre-configured file processing flow can be found in the configuration module based on the file name, thereby determining the file processing method, exception handling flow and specific database entry method corresponding to the file to be processed.

[0035] S120. Process the files to be processed based on the processing rules, and monitor the processing status of the files to be processed in real time.

[0036] In this context, "processing status" can be understood as the real-time processing status of the file to be processed. In this embodiment, the processing status can reflect the current stage, result, or problem type of data processing. For example, the processing status in this embodiment can include normal and abnormal states. The "in progress" state can specifically include "pending processing," "processing," and "completed." "Pending processing" can mean that file data has been received but processing has not yet begun. "Processing" can mean that data is being parsed, converted, or written to the target database. "Completed" can mean that data processing was successful without errors or conflicts; for example, all records have been successfully inserted into the database. "Processing failure" can mean that file data processing was terminated due to an error. An abnormal state can refer to the state corresponding to an abnormal situation that occurs during the file insertion process. For example, in this embodiment, abnormal states can include primary key conflicts or data format errors. In this embodiment, corresponding abnormal identifiers can be set based on different abnormal states to identify the corresponding abnormal states.

[0037] In this embodiment, the file to be processed can be processed according to the specific file processing flow contained in the processing rules, and the current processing status of the file to be processed can be detected in real time.

[0038] S130. If an abnormal processing state is detected, the data corresponding to the abnormal state is captured and separated from the file to be processed to obtain abnormal data and normal files.

[0039] Here, abnormal data refers to data corresponding to an abnormal state in the file to be processed. In this embodiment, abnormal data may include key data such as an abnormal description of the abnormal state, the time of occurrence, and related file identifiers. For example, abnormal data may include abnormal data corresponding to primary key conflicts or data format errors. Normal files may refer to the files obtained after separating the abnormal data from the file to be processed.

[0040] In this embodiment, each file to be processed contains many data entries. If an abnormal processing status is detected, an abnormal data entry can be detected. This data entry is then extracted from the file to be processed, and key data such as the abnormal status description, occurrence time, and related file identifiers are captured to obtain the abnormal data and the normal file.

[0041] S140. Process the abnormal data and normal files respectively and store them in the database.

[0042] In this embodiment, the exception handling strategy corresponding to the abnormal data and the file entry method of the normal file can be determined according to the specific processing flow of the file to be processed. Then, the abnormal data is processed according to the exception handling strategy and the normal file is entered into the database according to the file entry method, so that the normal file is stored normally in the specified database.

[0043] Furthermore, in this embodiment, after the exception handling strategy completes the processing of the exception data, if the processing result can be normally entered into the database, it will also be stored in the designated database. If there is no corresponding exception handling strategy in the specific processing flow for the exception data, it means that the exception data situation has not been pre-configured with a corresponding processing strategy, and the exception data can be stored in the corresponding exception database.

[0044] The technical solution of this invention involves acquiring a file to be processed and determining the corresponding processing rules; processing the file based on the processing rules and monitoring its processing status in real time; if an abnormal processing status is detected, capturing the data corresponding to the abnormal status and separating it from the file to be processed to obtain abnormal data and normal files; processing the abnormal data and normal files separately and storing them in a database. This technical solution enables automated processing when abnormal data occurs during file entry into the database, ensuring normal file entry, guaranteeing file data quality, and improving the continuity and reliability of file data entry.

[0045] Example 2

[0046] Figure 2 This is a flowchart of a file import method according to Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, the optimization includes: obtaining the file to be processed and determining the processing rules corresponding to the file to be processed, including: obtaining the file name of the file to be processed; determining the corresponding regular expression based on the file name, and then reading the processing rules of the file to be processed according to the regular expression. For example... Figure 2 As shown, the method includes:

[0047] S210. Obtain the file name of the file to be processed.

[0048] Here, the file name can be considered as the name of the file to be processed. In this embodiment, the file name of the file to be processed can be extracted from the file path.

[0049] S220. Determine the corresponding regular expression based on the file name, and then read the processing rules of the file to be processed according to the regular expression.

[0050] Regular expressions (regex or regexp) are patterns used to match combinations of characters in a string. They consist of a series of characters and special symbols and can be used to search, edit, or manage text.

[0051] In this embodiment, the regular expressions corresponding to different file names and the specific file processing flow corresponding to the regular expressions can be pre-configured during the configuration phase. It is understood that in this embodiment, the file name and regular expression of the file to be processed can be one-to-one, with each file to be processed having its corresponding regular expression.

[0052] In this embodiment, during the file import process, a predefined regular expression for the file name is searched and matched in the configuration module based on the file name of the file to be processed. For example, the file name "00-FISS-NH1016 0033-1G-1991-07-03.gz" can be successfully matched with the regular expression "^00-FISS-NH1016 0033-1G-(\d+)-(\d+).gz$", thereby reading the corresponding file processing flow configuration, i.e., the processing rule, from the configuration module based on this regular expression. After a successful match, the system obtains the corresponding file import processing flow information. If the configuration matching fails or key configurations are missing, the file import process is immediately terminated.

[0053] In this embodiment, the configuration module is responsible for recording file processing descriptions and using regular expressions of file names to match files to be processed. The configuration module can obtain configuration information for handling unforeseen exceptions by associating exception configurations. Furthermore, by describing the file entry method and process, this module enables dynamic configuration of the file entry process.

[0054] In this embodiment, the file processing module is the component that actually performs file processing operations. The file processing module can differentiate the file processing flow based on the file's configuration model and exception configuration model. Specifically, this can be achieved through differentiated file processing interfaces. This module can retrieve class methods from the BeanFactory in the configuration based on the class names to obtain the processing rules corresponding to different files, thereby achieving differentiated processing of the file processing flow based on the configuration.

[0055] S230. Process the files to be processed based on the processing rules and monitor the processing status of the files to be processed in real time.

[0056] S240. If an abnormal processing state is detected, the data corresponding to the abnormal state is captured and separated from the file to be processed to obtain the abnormal data and the normal file.

[0057] S250: Process abnormal data and normal files separately and store them in the database.

[0058] In this embodiment, optionally, the processing rules also include an exception handling strategy and a file processing method; correspondingly, the exception data and normal files are processed and stored in the database respectively, including: processing the normal file according to the file processing method to obtain first data; determining whether the processing rules contain an exception handling strategy corresponding to the exception data; if the processing rules contain an exception handling strategy corresponding to the exception data, then processing the exception data according to the exception handling strategy to obtain second data; and storing the first data and the second data in the first database.

[0059] In this context, the first data can refer to the data obtained after processing a normal file through specific preprocessing steps included in the file processing method. The second data can be considered as the data obtained after processing abnormal data through an exception handling strategy. The exception handling strategy can be considered as performing different processing operations for different exception situations. In this embodiment, the exception handling strategy and the exception situation are corresponding and can be pre-configured according to actual needs. The file processing method can refer to the preprocessing steps that need to be performed during the file entry process. In this embodiment, different files to be processed may require different preprocessing steps during the entry process. For example, the preprocessing steps may include historical data clearing, data file decompression, file field dictionary translation, file field mapping, and file field preprocessing. The first database can be understood as the specified database into which the files to be processed need to be stored.

[0060] In this embodiment, when processing abnormal data and normal files separately, during the file processing flow, the system determines the preprocessing steps required for file entry based on the configuration information of the files to be processed in the configuration module. Then, it performs the corresponding preprocessing steps on normal files according to the configured file processing method to obtain preprocessed data, i.e., the first data. The processed data is then entered into the database. The database entry operation in this embodiment may include: ① directly adding file information in batches; ② adding file information in batches, possibly placing it in a message queue and then entering it into different databases through multiple threads. If the entry is successful, the operation is complete. This embodiment can determine whether the processing rules contain an exception handling strategy corresponding to the abnormal data. If the processing rules contain such a strategy, the abnormal data can be processed according to the strategy to obtain the processed data, i.e., the second data, which is then stored in the corresponding database.

[0061] In this embodiment, the file to be processed may include multiple data entries. When the file data is entered into the database, it is processed in the form of data blocks. For example, the file to be processed includes 500 data entries. If one data record is found to be in an abnormal state, the abnormal data is identified and separated from the other data records. The remaining records are processed as normal files for entry into the database.

[0062] The exception capture and separation module in this embodiment is responsible for handling exception scenarios in file processing. It can determine whether to directly handle exceptions based on the configuration information. If no relevant exception scenarios are configured, it will roll back all possible error information according to the default handling process and send this information to the exception information storage module for manual processing.

[0063] In this embodiment, through this configuration, the system can automatically match and obtain the file processing flow using predefined filename regular expressions, thereby performing corresponding processing operations on the files to be processed and storing them in the corresponding database. This provides high flexibility and scalability, improving processing efficiency. Furthermore, this application can execute different preprocessing steps and database insertion operations based on different predefined configurations in the configuration module, adapting to different types of file and data processing needs.

[0064] In this embodiment, optionally, the method further includes: if the processing rules do not include an exception handling strategy corresponding to the exception data, then the exception data is stored in the second database.

[0065] The second database can refer to the exception database, which is used to store exception data corresponding to undefined exception states.

[0066] In this embodiment, if the data is successfully stored in the database according to the predetermined configuration during the process termination and exception handling, the processing flow of the file to be processed ends normally. If an unexpected error occurs, i.e., the processing rules do not include an exception handling strategy corresponding to the abnormal data, then the individual data entries in the file to be processed involving the abnormal situation will be analyzed in detail to determine the cause of the problem, and the problem and related records will be stored in the exception information database, i.e., the second database. At the same time, the system will continue to process the separated ordinary file information normally.

[0067] In this embodiment, the exception information storage module is responsible for storing the exception data corresponding to unconfigured exception information scenarios and submitting it to manual processing. This embodiment also provides a human-computer interaction interface to handle the current exception information processing and prompt maintenance personnel to improve the configuration of the exception handling process.

[0068] This embodiment allows for a detailed exception handling process in the configuration file, accurately recording exception scenarios and submitting them to operations and maintenance personnel for processing. Operations and maintenance personnel can optimize the file entry configuration based on the exception information, forming a closed-loop handling mechanism, which helps to continuously improve and optimize the file processing flow.

[0069] In this embodiment, optionally, the abnormal state includes a primary key conflict state, and the abnormal data includes the data corresponding to the primary key conflict state; accordingly, the abnormal data is processed according to the abnormal handling strategy to obtain the second data, including: obtaining the primary key index set of the file to be processed; reading the data records contained in the abnormal data one by one to obtain the primary key information of each data record; determining whether the primary key information exists in the primary key index set; if the primary key information exists in the primary key index set, then the data record corresponding to the primary key information is processed according to the abnormal handling strategy to obtain the target data, and the target data is used as the second data.

[0070] The primary key index set can refer to a dataset composed of the primary key information of all data during the processing of the file to be processed. In this embodiment, to prevent primary key conflicts, the system can create a primary key index result set corresponding to the file to be processed as soon as it receives the file to be processed. This set records the primary key index information of all data during the processing of the file to be processed. Therefore, in this embodiment, the primary key index set of the file to be processed can be directly obtained. The target data can refer to the data obtained after processing the primary key information according to the primary key conflict processing strategy, or it can be considered as the final data record determined under the abnormal state of primary key conflict.

[0071] In this embodiment, when the abnormal data is obtained as the corresponding data under the primary key conflict anomaly, the primary key index set of the file to be processed can be directly obtained. Then, the data of the file to be processed is read. The system reads the data records contained in the abnormal data one by one, obtains the primary key information of each data record, and checks whether the primary key of each record already exists in the primary key index result set. Specifically, in this embodiment, the system can determine whether the primary key information exists in the primary key index set. If the primary key information exists in the primary key index set, the primary key information is processed according to the anomaly handling strategy to obtain the target data, and the target data is used as the second data. If the primary key does not exist, it indicates that there is no risk of primary key conflict in the current record entry, and the data is then stored in the entry dataset. This step continues until the predetermined number of record rows in the file configuration is read or the end of the file is reached. In this embodiment, the data entry operation can be to store the data in the entry dataset into the database. If the end of the file has been reached, the current data entry operation ends.

[0072] In addition, if the primary key already exists in this embodiment, conflicting data can be rolled back, and all conflicting data can be sent to the exception information processing module for storage.

[0073] In this embodiment, this configuration creates a primary key index result set during file processing. This set records the primary key information of all data processed in the current file. By reading file data line by line and checking for the existence of the primary key, the uniqueness and integrity of each record are ensured. Furthermore, this embodiment allows the storage of anomaly and conflicting data in an anomaly storage module, which helps protect data security and traceability.

[0074] In this embodiment, optionally, the data records corresponding to the primary key information are processed according to the exception handling strategy to obtain the target data, including: taking the first data record corresponding to the same primary key information as the target data, or taking the last data record corresponding to the same primary key information as the target data.

[0075] In this embodiment, various exception handling strategies can be implemented through configuration. In the case of a primary key conflict, the first data record corresponding to the same primary key information can be used as the target data, or the last data record corresponding to the same primary key information can be used as the target data. That is, the first or last record of the primary key conflict can be used as the final data record. Alternatively, in this embodiment, the final data entry result can be selected based on the specific characteristics of a certain field (such as the maximum value). Different processing strategies can be set according to actual needs.

[0076] This embodiment may also include an exception configuration module, which is used to store exception scenarios and can handle exception scenarios according to manual configuration.

[0077] This embodiment allows for different exception handling strategies to be set for different abnormal states, effectively preventing primary key conflicts and improving the accuracy and reliability of data entry.

[0078] In this embodiment, optionally, before obtaining the file to be processed, the method further includes: in response to the configuration operation, configuring the exception handling strategies corresponding to various exception scenarios through a visual interface.

[0079] The configuration operation can be a triggered configuration operation performed by technicians or users. The visual interface refers to a user-interactive visual interface. Multiple abnormal scenarios refer to different pre-defined abnormal states corresponding to different abnormal scenarios. In this embodiment, by responding to the user's configuration operation, the user is allowed to configure different abnormal scenarios and their corresponding handling strategies in a graphical manner.

[0080] In this embodiment, the configuration module can also provide a visual user interface, allowing system administrators or relevant operators to flexibly configure exception handling strategies when configuring data entry into the database according to actual needs. This includes, but is not limited to, methods for handling primary key conflicts (such as ignoring conflicts, automatic renaming, logging, etc.), and the storage format and location of exception information. Through this setup, users can intuitively view and modify configurations via a graphical interface, greatly simplifying the configuration process and improving its accuracy and efficiency.

[0081] This embodiment provides a closed-loop, configuration-based anomaly handling process, achieving full automation of anomaly detection, configuration, and processing. The file processing workflow is designed to handle anomalies and accurately record them, subsequently submitting the anomaly information to operations and maintenance personnel. These personnel can then optimize the file storage configuration based on this information. Notably, the file processing procedure retrieves the latest file configuration before each file processing step, ensuring continuous optimization of the file storage scenario.

[0082] The technical solution of this embodiment enables efficient capture, recording, and separation of abnormal situations that occur during the file entry process, ensuring that the entry operation of other non-abnormal files can continue even when abnormal data is encountered. At the same time, it provides a visual configuration interface, allowing administrators to intuitively configure and adjust the abnormal handling strategy, thereby significantly improving the efficiency of file entry and the reliability of the system.

[0083] The technical solution of this invention involves: obtaining the filename of the file to be processed; determining the corresponding regular expression based on the filename, thereby reading the processing rules for the file to be processed according to the regular expression; processing the file to be processed based on the processing rules and monitoring the processing status of the file to be processed in real time; if an abnormal status is detected, capturing the data corresponding to the abnormal status and separating it from the file to be processed to obtain abnormal data and normal files; processing the abnormal data and normal files separately and storing them in a database. This technical solution enables automated processing when abnormal data occurs during file entry into the database, thereby ensuring normal file entry, guaranteeing the quality of file data, improving the continuity and reliability of file data entry, and reducing the cost and time of manual maintenance.

[0084] Example 3

[0085] Figure 3 This is a schematic diagram of a document entry device according to Embodiment 3 of the present invention.

[0086] like Figure 3 As shown, the device includes:

[0087] The data acquisition module 310 is used to acquire the file to be processed and determine the processing rules corresponding to the file to be processed;

[0088] The file processing module 320 is used to process the files to be processed based on processing rules and to detect the processing status of the files to be processed in real time.

[0089] The data separation module 330 is used to capture the data corresponding to the abnormal state and separate it from the file to be processed if the processing state is detected to be abnormal, so as to obtain abnormal data and normal file.

[0090] The file storage module 340 is used to process abnormal data and normal files respectively and store them in the database.

[0091] Optionally, the data acquisition module 310 is specifically used for:

[0092] Get the filename of the file to be processed;

[0093] The corresponding regular expression is determined based on the file name, and then the processing rules of the file to be processed are read according to the regular expression.

[0094] Optionally, the processing rules may also include exception handling strategies and file handling methods;

[0095] Correspondingly, the file storage module 340 includes:

[0096] The first processing unit is used to process normal files according to the file processing method to obtain the first data;

[0097] The judgment unit is used to determine whether the processing rules contain an exception handling strategy corresponding to the abnormal data.

[0098] The second processing unit is used to process the abnormal data according to the abnormal data if the processing rules contain an abnormal handling strategy corresponding to the abnormal data, and obtain the second data.

[0099] The first storage unit is used to store the first data and the second data into the first database.

[0100] Optionally, the file storage module 340 also includes:

[0101] The second storage unit is used to store the abnormal data in the second database if the processing rules do not contain an exception handling strategy corresponding to the abnormal data.

[0102] Optionally, abnormal states include primary key conflict states, and abnormal data includes the data corresponding to primary key conflict states;

[0103] Accordingly, the second processing unit is specifically used for:

[0104] Get the primary key index set of the file to be processed;

[0105] Read each data record contained in the abnormal data one by one to obtain the primary key information of each data record;

[0106] Determine if the primary key information exists in the primary key index set;

[0107] If the primary key information exists in the primary key index set, the data record corresponding to the primary key information is processed according to the exception handling strategy to obtain the target data, and the target data is used as the second data.

[0108] Optionally, the second processing unit is specifically used for:

[0109] Use the first data record with the same primary key information as the target data, or use the last data record with the same primary key information as the target data.

[0110] Optionally, the device may also include:

[0111] The configuration module is used to configure exception handling strategies for various abnormal scenarios through a visual interface before obtaining the file to be processed, in response to the configuration operation.

[0112] The file entry device provided in this embodiment of the invention can execute a file entry method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0113] Example 4

[0114] Figure 4 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0115] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the file loading method.

[0118] In some embodiments, the file loading method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file loading method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the file loading method by any other suitable means (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0123] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0124] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0125] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for storing files in a database, characterized in that, include: Obtain the file to be processed and determine the processing rules corresponding to the file to be processed; The file to be processed is processed based on the processing rules, and the processing status of the file to be processed is detected in real time. If the processing state is detected to be abnormal, the data corresponding to the abnormal state is captured and separated from the file to be processed to obtain abnormal data and normal file; The abnormal data and the normal files are processed separately and stored in the database.

2. The method according to claim 1, characterized in that, Obtain the file to be processed and determine the processing rules corresponding to the file to be processed, including: Get the filename of the file to be processed; The corresponding regular expression is determined based on the file name, and the processing rules of the file to be processed are read according to the regular expression.

3. The method according to claim 1, characterized in that, The processing rules also include exception handling strategies and file processing methods; Accordingly, the abnormal data and the normal files are processed and stored in the database respectively, including: The normal file is processed according to the file processing method to obtain the first data; Determine whether the processing rules include an exception handling strategy corresponding to the abnormal data; If the processing rules include an exception handling strategy corresponding to the abnormal data, then the abnormal data is processed according to the exception handling strategy to obtain the second data; The first data and the second data are stored in the first database.

4. The method according to claim 3, characterized in that, Also includes: If the processing rules do not include an exception handling strategy corresponding to the abnormal data, the abnormal data will be stored in the second database.

5. The method according to claim 3, characterized in that, The abnormal state includes the primary key conflict state, and the abnormal data includes the data corresponding to the primary key conflict state; Accordingly, the abnormal data is processed according to the aforementioned anomaly handling strategy to obtain second data, including: Obtain the primary key index set of the file to be processed; Read each data record contained in the abnormal data one by one to obtain the primary key information of each data record; Determine whether the primary key information exists in the primary key index set; If the primary key information exists in the primary key index set, the data record corresponding to the primary key information is processed according to the exception handling strategy to obtain the target data, and the target data is used as the second data.

6. The method according to claim 5, characterized in that, The target data is obtained by processing the data records corresponding to the primary key information according to the exception handling strategy, including: Use the first data record with the same primary key information as the target data, or use the last data record with the same primary key information as the target data.

7. The method according to claim 1, characterized in that, Before obtaining the file to be processed, the following steps are also included: In response to configuration operations, exception handling strategies for various abnormal scenarios can be configured through a visual interface.

8. A document entry device, characterized in that, include: The data acquisition module is used to acquire the file to be processed and determine the processing rules corresponding to the file to be processed. The file processing module is used to process the file to be processed based on the processing rules and to detect the processing status of the file to be processed in real time. The data separation module is used to capture the data corresponding to the abnormal state and separate it from the file to be processed if the processing state is detected to be abnormal, so as to obtain abnormal data and normal file. The file storage module is used to process the abnormal data and the normal files respectively and store them in the database.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the file storage method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the file loading method according to any one of claims 1-7.

Images