File storage method and device and electronic equipment

Abstract

The invention discloses a file storage method and device and electronic equipment. The method relates to the field of big data, and comprises the steps of obtaining target file information of a to-be-stored file indicated by a file storage instruction under the condition that the file storage instruction is received; judging whether the to-be-stored file is abnormal or not according to the target file information; under the condition that the to-be-stored file is not abnormal, database information of a target database is obtained, the to-be-stored file is processed according to the database information, and an updated to-be-stored file is obtained; and storing the updated to-be-stored file into the target database. Through the data importing method and device, the problem that the accuracy of data importing through a preset importing method is low in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a file storage method, device, and electronic device. Background Art

[0002] In the era of big data and cloud computing, the efficient data import capability of databases is crucial to the data processing efficiency of financial institutions. Currently, when importing data into a database, the usual method is to use the data import method preset by the database. However, when processing complex file data, especially when faced with file format errors, data type mismatches, and scenarios that require complex data pre-processing and post-processing, existing import tools and methods often seem inadequate.

[0003] Specifically, although the data import methods preset by the database provide relatively high import efficiency when the data volume is large, their fault tolerance mechanisms are relatively basic. For example, when the file data format is seriously inconsistent with the expected format of the database, these tools are often unable to intelligently handle errors and adjust data, and need to rely on manual intervention or additional program logic to correct erroneous data, which not only increases development and maintenance costs, but also reduces data processing efficiency and overall system stability.

[0004] With regard to the problem of low accuracy of data importing through preset import methods in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of the present application is to provide a file storage method, device and electronic device to solve the problem of low accuracy of data importing through a preset import method in the related art.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a file storage method is provided. The method comprises: when receiving a file storage instruction, obtaining the target file information of the file to be stored indicated by the file storage instruction; judging whether there is an abnormality in the file to be stored according to the target file information; when there is no abnormality in the file to be stored, obtaining the database information of the target database, and processing the file to be stored according to the database information to obtain an updated file to be stored; and storing the updated file to be stored in the target database.

[0007] Optionally, judging whether there is an abnormality in the file to be stored according to the target file information includes: obtaining preset file information of the file to be stored in the local memory according to the file name of the file to be stored; or obtaining preset file information of the file to be stored from the file storage instruction; comparing the preset file information with the target file information to obtain a comparison result, and when the comparison result indicates that the preset file information and the target file information are the same, determining that there is no abnormality in the file to be stored; when the comparison result indicates that the preset file information and the target file information are different, determining that there is an abnormality in the file to be stored.

[0008] Optionally, processing the file to be stored according to database information to obtain an updated file to be stored includes: determining a preset data structure and a preset data type of the data stored in the target database; processing the data in the file to be stored according to the preset data structure and the preset data type, as well as the initial data structure and the initial data type of the data contained in the file to be stored to obtain an updated file to be stored.

[0009] Optionally, storing the updated file to be stored in the target database includes: determining performance information of the target database, and determining the maximum number of records for obtaining data from the target database based on the performance information; obtaining the number of rows of data in the updated file to be stored, and comparing the number of rows with the maximum number of records; when the number of rows is greater than the maximum number of records, dividing the updated file to be stored according to the maximum number of records to obtain multiple sub-files, and storing them in the target database in sequence according to the arrangement order of the multiple sub-files in the updated file to be stored; when the number of rows is less than or equal to the maximum number of records, storing the updated file to be stored in the target database.

[0010] Optionally, before storing the updated file to be stored in the target database, the method also includes: determining whether it is necessary to process the target data in the updated file to be stored; if it is necessary to process the target data, obtaining the row data where the target data is located, and processing the row data according to the processing flow to obtain the processed row data; and replacing the row data where the target data is located in the updated file to be stored with the processed row data.

[0011] Optionally, before storing the updated file to be stored in the target database, the method also includes: determining whether there is an initial file associated with the file to be stored in the target database; if the initial file exists in the target database, adjusting the data in the initial file according to the association relationship between the file to be stored and the initial file; after completing the adjustment operation of the initial file, storing the updated file to be stored in the target database; or, obtaining storage conditions required for storing the file to be stored to obtain target storage conditions; updating the target database according to the target storage conditions to obtain an updated target database, and storing the updated file to be stored in the updated target database.

[0012] Optionally, after storing the updated files to be stored in the target database, the method also includes: receiving the storage results fed back by the target database; if there is an abnormality in the storage results, displaying an alarm message, wherein the alarm message is used to prompt the operation and maintenance personnel of the file storage abnormality; if there is no abnormality in the storage results, determining whether there are post-storage operation instructions for the files to be stored; if there are post-storage operation instructions, sending the post-storage operation instructions to the target database, wherein the target database processes the updated files to be stored according to the post-storage operation instructions.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a file storage device is provided. The device comprises: a first acquisition unit, which is used to acquire the target file information of the file to be stored indicated by the file storage instruction when receiving the file storage instruction; a first judgment unit, which is used to judge whether there is an abnormality in the file to be stored according to the target file information; a second acquisition unit, which is used to acquire the database information of the target database when there is no abnormality in the file to be stored, and process the file to be stored according to the database information to obtain an updated file to be stored; and a first storage unit, which is used to store the updated file to be stored in the target database.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, the electronic device comprising a memory storing an executable program; and a processor for running the program, wherein the above-mentioned file storage method is executed when the program is running.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is provided, including computer instructions, which implement the steps of the above-mentioned file storage method when executed by a processor.

[0016] In an embodiment of the present application, when a file entry instruction is received, target file information of the file to be entered indicated by the file entry instruction is obtained; whether there is an abnormality in the file to be entered is determined based on the target file information; when there is no abnormality in the file to be entered, database information of the target database is obtained, and the file to be entered is processed based on the database information to obtain an updated file to be entered; the updated file to be entered is stored in the target database, and the target file information is identified to ensure the accuracy of the file to be entered, and the file to be entered is processed based on the database information to ensure that the file to be entered can be stored in the target database, thereby achieving the purpose of checking the file to be entered, thereby achieving the technical effect of ensuring that the file to be entered can be accurately stored in the target database, and further solving the technical problem of low accuracy of data import through a preset import method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a file storage method is shown;

[0019] Figure 2 is a flowchart of a file storage method provided according to Embodiment 1 of the present application;

[0020] Figure 3 is a flowchart of an optional file storage method provided according to Embodiment 1 of the present application;

[0021] Figure 4 is a schematic diagram of a file storage device provided according to Embodiment 2 of the present application;

[0022] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the file storage method, device and electronic device determined in the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the file storage method, device and electronic device determined in the present disclosure is not limited.

[0027] It should be noted that the collected information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) used in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated. Corresponding operation entrances are provided for users to choose to authorize use or refuse use. If the user chooses to refuse, the expert decision-making process is entered. For example, an interface is set between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0028] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0029] Example 1

[0030] According to an embodiment of the present application, an embodiment of a file storage method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a file storage method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the file storage method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned file storage method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0036] Under the above operating environment, this application provides Figure 2 The file storage method shown. Figure 2 is a flowchart of a file storage method provided in Example 1 of the present application. Figure 2 As shown, the method includes:

[0037] Step S201, when a file storage instruction is received, the target file information of the file to be stored indicated by the file storage instruction is obtained.

[0038] Specifically, the execution subject of this embodiment is a file processing system, and the file storage instruction is an instruction issued by the user to instruct the import of the file to be stored into the target database. When the system receives the storage instruction, it needs to identify whether the file to be stored is an abnormal file, that is, whether there are some defects in it, so that the file to be stored cannot be stored in the target database.

[0039] When identifying files to be stored, it is necessary to obtain the target file information of the files to be stored, that is, detailed information of the files to be stored, including but not limited to the file path, file name, file size, encoding format, column attributes and data type, etc., and then determine whether the files to be stored can be normally imported into the target database based on the target file information.

[0040] Step S202, judging whether there is any abnormality in the file to be stored according to the target file information.

[0041] Specifically, after obtaining the target file information of the file to be stored, the target file information can be used to check whether the integrity, format and content of the file meet expectations. For example, the system can check whether the size of the file exceeds the limit, whether the encoding format of the file is correct, and whether the number of columns and types of the file match the database table, etc., to determine whether there are any abnormalities in the file to be stored. If the file has any of the above abnormalities, the system will prevent further processing of the file and feedback error information, thereby avoiding storing the abnormal file to be stored in the target database, resulting in database operation failure or data corruption.

[0042] Step S203, when there is no abnormality in the file to be stored, the database information of the target database is obtained, and the file to be stored is processed according to the database information to obtain an updated file to be stored.

[0043] It should be noted that the database information is relevant information of the target database, and may include structural information of the target database, such as table name, column name, data type, etc., and database connection information, such as server address, port, etc.

[0044] Specifically, when it is determined that there is no abnormality in the file to be stored, it is characterized that the file to be stored can be stored in the target database. However, it is necessary to first obtain the database information of the target database and further process the file to be stored according to the database information to ensure that the file to be stored meets the storage requirements of the target database and ensure that the file to be stored can be normally stored in the target database.

[0045] For example, when the system confirms that there are no abnormalities in the files to be stored, it can further obtain detailed information of the target database and preprocess the file data based on this. Preprocessing may include data cleaning (such as removing blank lines and spaces, converting data formats), data slicing (splitting large files into multiple small files for concurrent processing), and data mapping (aligning file data with database table structures). After preprocessing, the updated files to be stored are obtained, thereby ensuring the accuracy and consistency of the files to be stored before being imported into the database, improving the import efficiency, and providing preparation for subsequent data storage operations, thereby ensuring that the file data can be smoothly imported into the database.

[0046] Step S204, storing the updated files to be stored in the target database.

[0047] Specifically, after completing the identification and update of the files to be stored, the database import command (such as the \copy command) can be used to import the preprocessed file data into the database table, thereby completing the storage operation of the files to be stored and ensuring the accuracy of the files stored in the target database.

[0048] The file storage method provided by the embodiment of the present application adopts the method of obtaining the target file information of the file to be stored indicated by the file storage instruction when a file storage instruction is received; judging whether there is an abnormality in the file to be stored according to the target file information; obtaining the database information of the target database when there is no abnormality in the file to be stored, and processing the file to be stored according to the database information to obtain an updated file to be stored; storing the updated file to be stored in the target database, by identifying the target file information, thereby ensuring the accuracy of the file to be stored, and processing the file to be stored according to the database information, thereby ensuring that the file to be stored can be stored in the target database, achieving the purpose of checking the file to be stored, thereby achieving the technical effect of ensuring that the file to be stored can be accurately stored in the target database, and further solving the technical problem of low accuracy of data import through a preset import method in the related art.

[0049] Optionally, in the file storage method provided in the embodiment of the present application, judging whether there is an abnormality in the file to be stored according to the target file information includes: obtaining preset file information of the file to be stored in the local memory according to the file name of the file to be stored; or, obtaining preset file information of the file to be stored from the file storage instruction; comparing the preset file information with the target file information to obtain a comparison result, and when the comparison result indicates that the preset file information and the target file information are the same, determining that there is no abnormality in the file to be stored; when the comparison result indicates that the preset file information and the target file information are different, determining that there is an abnormality in the file to be stored.

[0050] Specifically, when determining whether there is an anomaly in the file to be stored, it is first necessary to read the name of the file to be stored, and then query in the local memory whether there is preset file information corresponding to the file to be stored, thereby utilizing the high-speed characteristics of memory retrieval to quickly locate and obtain the preset file information, wherein the preset file information may be a set of standardized file description information defined by the system or user, which may include file format, column attributes, data type, encoding, etc., for comparison with the actual target file information.

[0051] It should be noted that if the preset information of the file to be stored is not stored in the local memory, or a different information acquisition strategy is adopted, the system can also directly extract the preset file information from the file storage instruction, thereby ensuring that even if the preset information is not in the local memory, it can be obtained through the instruction, maintaining the flexibility and integrity of the process, and thus ensuring the accuracy of the judgment of the target file information.

[0052] After obtaining the preset file information, it is necessary to compare the preset file information with the target file information to obtain a comparison result. For example, the system can compare the key attributes in the preset information and the target file information one by one, such as file size, encoding format, number of columns and data type, etc., to obtain comparison results of multiple attributes, and then accurately identify whether the file meets the preset standards based on the comparison results.

[0053] If the comparison result shows that all key attributes are consistent, that is, the comparison result indicates that the preset file information and the target file information are the same, then the system will determine that there is no abnormality in the file to be stored, and the subsequent steps of data import can be safely carried out, thereby ensuring the accuracy of the file content and format of the file to be stored in the subsequent process.

[0054] If the comparison finds that the target file information does not match any key attribute in the preset file information, that is, the preset file information and the target file information are different, then the system will determine that there is an abnormality in the file to be stored and terminate the subsequent process, thereby ensuring the accurate identification and interception of the abnormal file to be stored, and ensuring the accuracy of the files stored in the target database.

[0055] This embodiment effectively avoids importing non-compliant data into the target database by accurately identifying whether the file meets the preset import standards, thereby improving the accuracy and efficiency of data entry and reducing the risk of database operation failure or data corruption due to file anomalies.

[0056] Optionally, in the file storage method provided in the embodiment of the present application, the file to be stored is processed according to the database information to obtain an updated file to be stored, including: determining a preset data structure and a preset data type of the data stored in the target database; processing the data in the file to be stored according to the preset data structure and the preset data type, as well as the initial data structure and the initial data type of the data contained in the file to be stored, to obtain an updated file to be stored.

[0057] It should be noted that the preset data structure can be the column layout and relationship of the table in the target database, including the name, order and association method of the column. The preset data type can be the type of each column of data in the database table, such as integer, floating point, string, etc. The initial data structure can be the original layout and relationship of the data in the file to be stored, including the name and order of the column. The initial data type can be the original type of each column of data in the file to be stored.

[0058] Specifically, after obtaining the database information, the files to be stored need to be processed according to the database information. First, the preset data structure and preset data type of the target database table are read from the database information, and the initial data structure and initial data type of the files to be stored are obtained, thereby comparing the initial data structure and initial data type of the files to be stored with the preset data structure and preset data type of the database table.

[0059] If inconsistencies are found, the system will automatically or according to preset rules convert and adjust the data in the file to match the preset standards of the database. For example, if the data type of a column in the file is a string, and the preset type of the database is an integer, the system will try to convert the string to an integer, thereby adjusting and optimizing the data in the file to be imported to ensure that the data can smoothly match the structure and type of the database table when imported into the database, thereby improving the accuracy and success rate of data import.

[0060] After the data structure and data type of the file to be stored are adjusted, an updated file to be stored will be obtained. At this time, the updated file to be stored can be stored in the target database, thereby ensuring the smooth progress of the data import process and improving the efficiency and success rate of data storage.

[0061] This embodiment compares the data structure and data type of the file to be stored with the data structure and data type required by the target database, thereby ensuring that the data in the file to be stored matches the structure and type of the database table, avoiding data type conversion errors or inconsistent data structures that lead to failure in importing the file to be stored, and ensuring normal storage of the file to be stored.

[0062] Optionally, in the file storage method provided in the embodiment of the present application, storing the updated file to be stored in the target database includes: determining the performance information of the target database, and determining the maximum number of records for obtaining data from the target database based on the performance information; obtaining the number of rows of data in the updated file to be stored, and comparing the number of rows with the maximum number of records; when the number of rows is greater than the maximum number of records, dividing the updated file to be stored according to the maximum number of records to obtain multiple sub-files, and storing them in the target database in sequence according to the arrangement order of the multiple sub-files in the updated file to be stored; when the number of rows is less than or equal to the maximum number of records, storing the updated file to be stored in the target database.

[0063] It should be noted that when storing the updated files to be stored in the target database, due to the different performance of the target database, there are different requirements for the storage method of the data when storing data. Therefore, when storing data, it is necessary to determine the data storage method based on the performance information of the target database to ensure the normal operation of the target database while ensuring the normal storage of the data in the files to be stored.

[0064] Specifically, the performance information may be a performance indicator of the target database when processing data, including but not limited to the database's read and write speed, processing capacity, memory limit, etc. The maximum number of records may be an upper limit of the number of data rows that can be processed in a single data import operation, determined based on the database performance information. When storing data, the system first obtains the performance information of the target database at the current moment, and calculates the maximum number of records that can ensure stable operation of the database based on the performance information, thereby determining a reasonable amount of data to be imported by analyzing the database's performance information, which can fully utilize the database's data processing capabilities, avoid system burden or data operation failure due to excessive data volume, and ensure the efficiency and security of data import.

[0065] Furthermore, when the maximum number of records is obtained, it is necessary to read the updated file to be stored in the database and count the number of data rows in the file, and then compare the number of rows with the previously determined maximum number of records to determine whether the file needs to be divided into multiple sub-files, wherein the operation of splitting the file can be to divide the updated file to be stored in the database into multiple smaller sub-files according to the maximum number of records, and the number of data rows contained in each sub-file does not exceed the maximum number of records.

[0066] When the number of rows is greater than the maximum number of records, the system will split the file into multiple sub-files according to the maximum number of records, and the number of data records contained in each sub-file is equal to or less than the maximum number of records. The system will maintain the arrangement order of the sub-files in the original file and import the data of these sub-files into the target database in sequence, thereby ensuring the order and consistency of data import while making full use of the concurrent processing capabilities of the database, avoiding data confusion or loss, and improving the efficiency and accuracy of data import.

[0067] When the number of rows is less than or equal to the maximum number of records, the system can directly import the entire updated file to be imported into the target database without additional file segmentation, thereby simplifying the data processing process and improving the efficiency of data import.

[0068] For example, when the number of data rows in the file to be stored is 10,000 rows and the maximum number of records is 100,000 rows, the file to be stored can be directly stored; when the number of data rows in the file to be stored is 120,000 rows and the maximum number of records is 100,000 rows, the number of data rows in the file to be stored can be divided into two sub-files with 60,000 rows each, and the sub-files can be stored in sequence, thereby ensuring that the target database can operate normally while the data in the file to be stored is stored normally.

[0069] This embodiment improves the efficiency and concurrency of data import by reasonably determining the maximum number of records for data import based on the performance information of the target database, and then decides whether to perform file segmentation and how to perform the segmentation and import based on the number of rows of the file to be stored, thereby ensuring the security and accuracy of data import.

[0070] Optionally, in the file storage method provided in the embodiment of the present application, before storing the updated file to be stored in the target database, the method also includes: determining whether it is necessary to process the target data in the updated file to be stored; if it is necessary to process the target data, obtaining the row data where the target data is located, and processing the row data according to the processing flow to obtain the processed row data; and replacing the row data where the target data is located in the updated file to be stored with the processed row data.

[0071] Specifically, the target data may be data in the updated files to be stored that require special processing. These data may require format modification, calculation conversion, data cleaning, and other operations to meet the specific requirements of the target database. Before the data is officially stored, the system needs to evaluate the updated files to be stored to determine whether there is target data that requires additional processing, such as target data that needs to be modified, added, deleted, etc., so as to ensure that all data meets the preset storage requirements before storage.

[0072] When it is determined that the target data needs to be processed, the row containing the data, i.e., the row data, can be located first, and the entire row data containing the target data can be extracted, so as to further operate the extracted row data according to the predefined processing flow, such as converting the data format, executing calculation logic, removing invalid data, or filling missing values. The processing flow may include steps such as data type conversion, data cleaning, and data verification to ensure that the target data in the row data meets the standards for storage.

[0073] After the row data processing is completed, the system will use the processed row data to replace the original corresponding row data in the updated file to be stored, thereby ensuring that the data stored in the file is the latest and processed data, and ensuring the accuracy of the data in the file to be stored.

[0074] This embodiment significantly improves the accuracy and efficiency of data import into the target database by adding a data processing step before storing the files to be stored. This operation can identify and process specific target data, ensure the correctness of the data format and content, and maintain the consistency and integrity of the file data by performing corresponding processing on the data, thereby improving the accuracy of the data files stored in the database.

[0075] Optionally, in the file storage method provided in the embodiment of the present application, before storing the updated file to be stored in the target database, the method also includes: determining whether there is an initial file associated with the file to be stored in the target database; if the initial file exists in the target database, adjusting the data in the initial file according to the association relationship between the file to be stored and the initial file; after completing the adjustment operation of the initial file, storing the updated file to be stored in the target database; or, obtaining the storage conditions required for storing the file to be stored to obtain the target storage conditions; updating the target database according to the target storage conditions to obtain the updated target database, and storing the updated file to be stored in the updated target database.

[0076] Specifically, the initial file can be a file that already exists in the target database and is associated with the file to be imported in terms of data or business logic. Before storing the file in the target database, the target database needs to be checked to confirm whether there is an initial file associated with the updated file to be imported. The checking process can be based on the file name, data table name or other identifiers, so that any initial file that may be affected can be preprocessed before importing new data to prevent data conflicts or business logic errors.

[0077] If an initial file exists in the target database, the data in the initial file needs to be adjusted according to the association between the file to be stored and the initial file. This may include operations such as data cleaning, data conversion, data merging, and data deletion, so as to ensure that after the file to be stored is stored in the target database, it will not affect the data in the file to be stored or the initial file, thereby ensuring the accuracy of the data in the file.

[0078] For example, when a primary key conflict exists between a certain data in the initial file and the file to be stored, the conflicting primary key in the initial file needs to be adjusted to ensure that no abnormal problems will occur in the file to be stored after it is stored.

[0079] After the adjustment of the initial file is completed, the updated file to be stored in the target database can be stored in the target database, thereby ensuring the smooth storage of the file to be stored.

[0080] It should be noted that before the updated files to be stored are stored, it is necessary to analyze the storage requirements of the files to be stored, such as data size, data type, index requirements, etc., to determine the target storage conditions. By understanding and optimizing the storage conditions in advance, it is ensured that the target database has sufficient resources and appropriate configuration to efficiently store new data, avoiding storage failures due to insufficient resources or improper configuration.

[0081] After determining the target storage conditions, the system can update the target database, including but not limited to table structure adjustment, index creation, storage space optimization, etc., so that after completing the database update, it can ensure that the updated files to be stored can be correctly stored in the updated target database, and then by updating the database configuration, ensure the efficiency and security of data storage.

[0082] This embodiment not only identifies and resolves data conflicts related to the files to be stored and the initial files by adding steps such as pre-data storage inspection, initial file adjustment, and database configuration optimization, but also ensures that the database has sufficient resources and correct configuration to store new data, thereby achieving the technical effect of enhancing the accuracy and efficiency of importing data into the target database.

[0083] Optionally, in the file storage method provided in the embodiment of the present application, after storing the updated file to be stored in the target database, the method also includes: receiving the storage result fed back by the target database; if there is an abnormality in the storage result, displaying an alarm message, wherein the alarm message is used to prompt the operation and maintenance personnel of the file storage abnormality; if there is no abnormality in the storage result, determining whether there is a post-storage operation instruction for the file to be stored; if there is a post-storage operation instruction, sending the post-storage operation instruction to the target database, wherein the target database processes the updated file to be stored according to the post-storage operation instruction.

[0084] Specifically, after storing the file in the target database, it is necessary to receive the storage result feedback from the target database, so as to determine whether the file is completely and correctly stored in the database according to the storage result. If there is an abnormality in the storage result feedback, the system will generate and display an alarm message, thereby informing the operation and maintenance personnel of the abnormal storage operation through the alarm message, and the need to process the file to be stored, thereby ensuring the normal storage and operation of the file to be stored and the target database.

[0085] If the data storage operation is successful, the system needs to check whether there are post-storage operation instructions for this data storage to determine whether additional operations need to be performed after the data is successfully stored, ensuring the integrity of the data processing process and the consistency of the business logic.

[0086] If there are post-storage operation instructions, the system will send the operation instructions to the target database, so that after receiving the operation instructions, the database will perform corresponding processing according to the operation instructions, such as updating data, creating backups, or triggering business logic, etc., thereby ensuring the complete processing operation flow for the updated files to be stored and ensuring the accuracy of the files in the target database.

[0087] This embodiment ensures the transparency of the data storage process and the timely response to abnormal situations by introducing a storage result feedback mechanism, abnormal alarm processing, and the execution of post-storage operation instructions, reducing the risk of business interruption caused by data storage failure or abnormalities. At the same time, the execution of post-storage operation instructions ensures the continuation of business logic after data storage, meets the data processing requirements in specific scenarios, improves the automation of data processing and business flexibility, and thus improves the accuracy of the data storage process to the target database and the consistency of business processes.

[0088] Figure 3 is a flowchart of an optional file storage method provided according to Example 1 of the present application, such as Figure 3 As shown, first, the file needs to be verified, and the file size, number of lines, file name, encoding format and other information need to be checked to determine whether there are any abnormalities in the file. If there are no abnormalities, determine whether the file and the target database need to be preprocessed. If preprocessing is required, perform the preprocessing operation, and split the processed file according to the performance of the target database to obtain multiple sub-files, and store the files through stream processing.

[0089] Furthermore, when storing a file, it is necessary to determine whether the data in the file needs to be intercepted and processed. If interception is required, the sub-file containing the data to be intercepted is intercepted and processed, and the processed sub-file is stored in the target database. If interception is not required, the sub-file is directly stored in the target database, and after storing all the file contents in the target database, it is determined whether there is a post-warehousing processing flow. If so, the post-warehousing processing flow is executed for the file, and if not, the data warehousing operation is completed, thereby ensuring that the files to be stored in the warehouse can be stored in the target database, achieving the purpose of checking the files to be stored, and thus achieving the technical effect of ensuring that the files to be stored in the warehouse can be accurately stored in the target database, thereby solving the technical problem of low accuracy of data import through preset import methods in related technologies.

[0090] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0091] Example 2

[0092] The present application also provides a file storage device. It should be noted that the file storage device of the present application can be used to execute the file storage method provided in the above embodiment. The file storage device provided in the present application is introduced below.

[0093] According to an embodiment of the present application, a device for implementing the above-mentioned file storage method is also provided. Figure 4 is a schematic diagram of a file storage device provided according to Embodiment 2 of the present application, such as Figure 4 As shown, the device comprises:

[0094] The first acquisition unit 41 is used for acquiring target file information of the to-be-stored file indicated by the file-stored instruction when receiving the file-stored instruction.

[0095] The first judgment unit 42 is used to judge whether there is any abnormality in the file to be stored according to the target file information.

[0096] The second acquisition unit 43 is used to acquire database information of the target database when there is no abnormality in the file to be stored, and process the file to be stored according to the database information to obtain an updated file to be stored.

[0097] The first storage unit 44 is used to store the updated files to be stored in the target database.

[0098] The file storage device provided by the embodiment of the present application, when receiving the file storage instruction, obtains the target file information of the file to be stored indicated by the file storage instruction through the first acquisition unit 41; the first judgment unit 42 judges whether there is an abnormality in the file to be stored according to the target file information; the second acquisition unit 43 obtains the database information of the target database when there is no abnormality in the file to be stored, and processes the file to be stored according to the database information to obtain an updated file to be stored; the first storage unit 44 stores the updated file to be stored in the target database. By identifying the target file information, the accuracy of the file to be stored is guaranteed, and the file to be stored is processed according to the database information, so as to ensure that the file to be stored can be stored in the target database, so as to achieve the purpose of checking the file to be stored, thereby achieving the technical effect of ensuring that the file to be stored can be accurately stored in the target database, thereby solving the technical problem of low accuracy of data import through the preset import method in the related technology.

[0099] Optionally, in the file storage device provided in the embodiment of the present application, the first judgment unit 42 includes: a first acquisition module, used to obtain preset file information of the file to be stored in the local memory according to the file name of the file to be stored; or, a second acquisition module, used to obtain preset file information of the file to be stored from the file storage instruction; a first comparison module, used to compare the preset file information with the target file information to obtain a comparison result, and when the comparison result indicates that the preset file information and the target file information are the same, determine that there is no abnormality in the file to be stored; a first determination module, used to determine that there is an abnormality in the file to be stored when the comparison result indicates that the preset file information and the target file information are different.

[0100] Optionally, in the file storage device provided in the embodiment of the present application, the second acquisition unit 43 includes: a second determination module, used to determine the preset data structure and preset data type of the data stored in the target database; a processing module, used to process the data in the file to be stored according to the preset data structure and preset data type, and the initial data structure and initial data type of the data contained in the file to be stored, to obtain an updated file to be stored.

[0101] Optionally, in the file storage device provided in the embodiment of the present application, the first storage unit 44 includes: a third determination module, used to determine the performance information of the target database, and determine the maximum number of records for obtaining data from the target database based on the performance information; a second comparison module, used to obtain the number of rows of data in the updated file to be stored, and compare the number of rows with the maximum number of records; a segmentation module, used to segment the updated file to be stored according to the maximum number of records when the number of rows is greater than the maximum number of records, to obtain multiple sub-files, and store the sub-files in sequence in the target database according to the arrangement order of the multiple sub-files in the updated file to be stored; a storage module, used to store the updated file to be stored in the target database when the number of rows is less than or equal to the maximum number of records.

[0102] Optionally, in the file storage device provided in the embodiment of the present application, before storing the updated file to be stored in the target database, the device also includes: a second judgment unit, used to judge whether it is necessary to process the target data in the updated file to be stored; a third acquisition unit, used to obtain the row data where the target data is located when it is necessary to process the target data, and process the row data according to the processing flow to obtain the processed row data; a replacement unit, used to replace the row data where the target data is located in the updated file to be stored with the processed row data.

[0103] Optionally, in the file storage device provided in the embodiment of the present application, before storing the updated file to be stored in the target database, the device also includes: a third judgment unit, used to judge whether there is an initial file associated with the file to be stored in the target database; an adjustment unit, used to adjust the data in the initial file according to the association relationship between the file to be stored and the initial file when the initial file exists in the target database; a second storage unit, used to store the updated file to be stored in the target database after completing the adjustment operation of the initial file; or, a fourth acquisition unit, used to acquire the storage conditions required for storing the file to be stored and obtain the target storage conditions; an update unit, used to update the target database according to the target storage conditions, obtain the updated target database, and store the updated file to be stored in the updated target database.

[0104] Optionally, in the file storage device provided in the embodiment of the present application, after the updated file to be stored in the target database is stored, the device also includes: a receiving unit, used to receive the storage result fed back by the target database; a display unit, used to display an alarm message when there is an abnormality in the storage result, wherein the alarm message is used to prompt the operation and maintenance personnel of the file storage abnormality; a fourth judgment unit, used to judge whether there is a post-storage operation instruction for the file to be stored when there is no abnormality in the storage result; a sending unit, used to send the post-storage operation instruction to the target database when there is a post-storage operation instruction, wherein the target database processes the updated file to be stored according to the post-storage operation instruction.

[0105] It should be noted that the first acquisition unit 41, the first judgment unit 42, the second acquisition unit 43, and the first storage unit 44 correspond to steps S201 to S204 in Embodiment 1, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules may also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0106] Example 3

[0107] An embodiment of the present application may provide an electronic device, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0108] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: when receiving a file storage instruction, obtain the target file information of the file to be stored indicated by the file storage instruction; determine whether there is an abnormality in the file to be stored based on the target file information; when there is no abnormality in the file to be stored, obtain the database information of the target database, and process the file to be stored based on the database information to obtain an updated file to be stored; and store the updated file to be stored in the target database.

[0110] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: obtain the preset file information of the file to be stored in the local memory according to the file name of the file to be stored; or obtain the preset file information of the file to be stored from the file storage instruction; compare the preset file information with the target file information to obtain a comparison result, and when the comparison result indicates that the preset file information and the target file information are the same, determine that there is no abnormality in the file to be stored; when the comparison result indicates that the preset file information and the target file information are different, determine that there is an abnormality in the file to be stored.

[0111] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the preset data structure and preset data type of the data stored in the target database; process the data in the file to be stored according to the preset data structure and preset data type, as well as the initial data structure and initial data type of the data contained in the file to be stored, to obtain an updated file to be stored.

[0112] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the performance information of the target database, and determine the maximum number of records for obtaining data from the target database based on the performance information; obtain the number of rows of data in the updated file to be stored, and compare the number of rows with the maximum number of records; when the number of rows is greater than the maximum number of records, divide the updated file to be stored according to the maximum number of records to obtain multiple sub-files, and store them in the target database in sequence according to the arrangement order of the multiple sub-files in the updated file to be stored; when the number of rows is less than or equal to the maximum number of records, store the updated file to be stored in the target database.

[0113] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine whether it is necessary to process the target data in the updated file to be stored; if it is necessary to process the target data, obtain the row data where the target data is located, and process the row data according to the processing flow to obtain the processed row data; use the processed row data to replace the row data where the target data is located in the updated file to be stored.

[0114] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine whether there is an initial file associated with the file to be stored in the target database; if the initial file exists in the target database, adjust the data in the initial file according to the association relationship between the file to be stored and the initial file; after completing the adjustment operation of the initial file, store the updated file to be stored in the target database; or, obtain the storage conditions required for storing the file to be stored to obtain the target storage conditions; update the target database according to the target storage conditions to obtain the updated target database, and store the updated file to be stored in the updated target database.

[0115] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: receiving the storage results fed back by the target database; if there is an abnormality in the storage results, displaying an alarm message, wherein the alarm message is used to prompt the operation and maintenance personnel of the file storage abnormality; if there is no abnormality in the storage results, determining whether there is a post-storage operation instruction for the file to be stored; if there is a post-storage operation instruction, sending the post-storage operation instruction to the target database, wherein the target database processes the updated file to be stored according to the post-storage operation instruction.

[0116] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations are shown.

[0117] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0118] Example 4

[0119] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the file storage method provided in the first embodiment.

[0120] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0121] An embodiment of the present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the file storage method.

[0122] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned file storage method.

[0123] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0129] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A file storage method, characterized in that: include: When receiving a file storage instruction, obtaining target file information of the file to be stored indicated by the file storage instruction; Determine whether the file to be stored is abnormal according to the target file information; When there is no abnormality in the file to be stored, obtaining database information of the target database, and processing the file to be stored according to the database information to obtain an updated file to be stored; The updated files to be stored are stored in the target database.

2. The method according to claim 1, characterized in that Judging whether the to-be-stored file is abnormal according to the target file information includes: Obtaining preset file information of the file to be stored in the local memory according to the file name of the file to be stored; or obtaining preset file information of the file to be stored from the file storage instruction; Comparing the preset file information with the target file information to obtain a comparison result, and determining that there is no abnormality in the file to be stored when the comparison result indicates that the preset file information and the target file information are the same; When the comparison result indicates that the preset file information and the target file information are different, it is determined that an abnormality exists in the file to be stored.

3. The method according to claim 1, characterized in that The to-be-stored file is processed according to the database information to obtain an updated to-be-stored file including: Determining a preset data structure and a preset data type of data stored in the target database; According to the preset data structure and the preset data type, and the initial data structure and the initial data type of the data contained in the file to be stored, the data in the file to be stored is processed to obtain the updated file to be stored.

4. The method according to claim 1, characterized in that Storing the updated to-be-stored file in the target database includes: Determine performance information of the target database, and determine a maximum number of records of data to be acquired by the target database according to the performance information; Obtaining the number of rows of data in the updated file to be stored, and comparing the number of rows with the maximum number of records; In the case where the number of rows is greater than the maximum number of records, the updated file to be stored is divided according to the maximum number of records to obtain a plurality of sub-files, and the plurality of sub-files are sequentially stored in the target database according to the arrangement order of the plurality of sub-files in the updated file to be stored; When the number of rows is less than or equal to the maximum number of records, the updated file to be stored is stored in the target database.

5. The method according to claim 1, characterized in that Before storing the updated to-be-stored file in the target database, the method further includes: Determine whether it is necessary to process the updated target data in the to-be-stored file; When the target data needs to be processed, the row data where the target data is located is obtained, and the row data is processed according to the processing flow to obtain the processed row data; The processed row data is used to replace the row data where the target data is located in the updated file to be stored.

6. The method according to claim 1, characterized in that Before storing the updated to-be-stored file in the target database, the method further includes: Determine whether there is an initial file associated with the file to be stored in the target database; When the initial file exists in the target database, adjusting the data in the initial file according to the association relationship between the file to be stored and the initial file; After completing the adjustment operation of the initial file, storing the updated file to be stored in the target database; or, Acquire storage conditions required for storing the files to be stored, and obtain target storage conditions; The target database is updated according to the target storage condition to obtain an updated target database, and the updated files to be stored are stored in the updated target database.

7. The method according to claim 1, characterized in that After storing the updated to-be-stored file in the target database, the method further includes: Receiving the storage result fed back by the target database; If there is an abnormality in the storage result, an alarm message is displayed, wherein the alarm message is used to remind the operation and maintenance personnel of the abnormality in file storage; If there is no abnormality in the storage result, determining whether there is a post-storage operation instruction for the file to be stored; In the case where the post-storage operation instruction exists, the post-storage operation instruction is sent to the target database, wherein the target database processes the updated file to be stored according to the post-storage operation instruction.

8. A file storage device, characterized in that: include: A first acquisition unit is used to acquire target file information of a file to be stored indicated by the file storage instruction when receiving the file storage instruction; A first judgment unit, used for judging whether the to-be-stored file is abnormal according to the target file information; A second acquisition unit is used to acquire database information of a target database when there is no abnormality in the file to be stored, and process the file to be stored according to the database information to obtain an updated file to be stored; The first storage unit is used to store the updated file to be stored in the target database.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the file storage method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the file storage method described in any one of claims 1 to 7 when running.

Images