Business data file loading method and device, storage medium and processor
By filtering and segmenting business data files and using different loading methods to handle abnormal and normal data, the problem of a single set of data occupying multiple rows in the banking system was solved, improving the efficiency and accuracy of data loading and avoiding a decrease in speed.
Patent Information
- Application Number
- CN202511410618.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-29
- Publication Date
- 2025-12-30
AI Technical Summary
Existing technologies are incompatible with solving the problems of a single set of data occupying multiple rows and slow loading speed, leading to business data loading failures and reduced speed in banking systems.
By receiving files from the upstream system, the first line number with a number of delimiters less than the number of constituent fields is selected, and the business data file is split into first business data and second business data. Different loading methods are used to process them respectively. The second business data is loaded using an existing database loading tool, and the first business data is loaded sequentially based on the number of constituent fields.
While solving the problem of a set of data occupying multiple rows, it avoids reducing loading speed and improves the efficiency and accuracy of data loading.
Smart Images
Figure CN121233656A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a method, apparatus, storage medium, and processor for loading business data files. Background Technology
[0002] There are upstream and downstream dependencies between various systems (services) within a bank. Downstream systems (services) rely on data provided by upstream systems (services). For example, the cash position management system depends on daily customer payment transaction data provided by the upstream accounting engine system to calculate the bank's cash position. In medium and large-sized banks, due to the large customer base, high daily transaction volume, and the resulting large and complex volume of business data, downstream systems must load the files provided by the upstream system on time and correctly at the end of each day to ensure the normal operation of downstream systems.
[0003] In existing technical solutions, upstream and downstream systems pre-agree on data formats, and the downstream system loads files and performs database operations according to the predetermined format. Due to the unpredictability of customer-entered data, downstream systems may encounter loading errors or even system errors when loading upstream data according to a single, predetermined loading rule. Examples include: 1. Changes in the upstream system's business operations lead to changes in the data format, requiring adjustments in the downstream system, resulting in high interconnectivity; 2. The upstream system provides data partially entered by bank customers, which is unpredictable. Customers may input newlines during information entry, causing a single set of data to occupy two lines in the data file, leading to loading failures and program errors in the downstream system. For cases where a single set of data occupies multiple lines, existing memory loading techniques (e.g., Java memory file loading) can read the file line by line into memory, identify the data according to rules, and then insert it into the database. However, this method is slow and unsuitable for loading large amounts of business data in banking systems.
[0004] In summary, existing technologies cannot reconcile the issues of a single set of data occupying multiple rows and slow loading speed. Summary of the Invention
[0005] The purpose of this application is to provide a business data file loading method, apparatus, storage medium, and processor to solve the problem of a set of data occupying multiple lines while avoiding a reduction in loading speed.
[0006] To achieve the above objectives, a first aspect of this application provides a method for loading a business data file, comprising: receiving a file provided by an upstream system, including a business data file and a data format file; obtaining the number of constituent fields of a set of data from the data format file; selecting a first line number in the business data file whose number of delimiters for each line of data is less than the number of constituent fields; dividing the business data file into first business data and second business data, wherein the first business data is the data corresponding to the first line number, and the second business data is the data corresponding to other line numbers; loading the first business data sequentially based on the number of constituent fields; and loading the second business data using an existing database loading tool.
[0007] In this embodiment of the application, selecting the first row number where the number of delimiters in each row of data in the business data file is less than the number of constituent fields includes: obtaining delimiters between fields from the data format file; counting the number of delimiters in each row of data in the business data file; and comparing the number of delimiters in each row of data with the number of constituent fields, and recording the row number where the number of delimiters is less than the number of constituent fields as the first row number.
[0008] In this embodiment of the application, dividing the business data file into first business data and second business data includes: using the shell split command to divide the business data file according to the first line number to obtain the data corresponding to each line number in the first line number; and dividing the data corresponding to each line number in the first line number into the first business data, and dividing the remaining data into the second business data.
[0009] In this embodiment of the application, loading the first business data sequentially based on the number of constituent fields includes: starting from the first row of the first business data, executing the following steps sequentially, row by row, until the last row of the first business data: accumulating the number of fields up to the current row, determining whether the accumulated number has reached the number of constituent fields; if it has, loading the accumulated fields as a group of data, clearing the accumulated number to zero, and moving to the next row to start accumulating again; if it has not reached the number, moving to the next row to continue accumulating.
[0010] In this embodiment of the application, the method further includes: before filtering the first row number, determining whether there is a group of data occupying multiple rows; if so, continuing to execute; if not, directly loading the business data file.
[0011] In this embodiment of the application, determining whether there is a set of data occupying multiple lines includes: receiving a data verification file provided by an upstream system; determining whether the number of lines in the business data file is consistent with the number of business data recorded in the data verification file; and if they are consistent, determining that there is no set of data occupying multiple lines, and if they are inconsistent, determining that there is a set of data occupying multiple lines.
[0012] In this embodiment of the application, loading the second service data includes: comparing the size of the second service data with a preset first threshold; and if the size of the second service data is less than or equal to the first threshold, then directly loading the second service data; if the size of the second service data is greater than the first threshold, then slicing the second service data to obtain several slice data not greater than a preset second threshold, and loading the slice data in a multi-threaded concurrent manner.
[0013] A second aspect of this application provides a business data file loading apparatus, comprising: a receiving module for receiving a file provided by an upstream system, including a business data file and a data format file; a reading module for obtaining the number of constituent fields of a set of data from the data format file; a filtering module for filtering out a first line number in the business data file whose number of delimiters for each line of data is less than the number of constituent fields; a segmentation module for segmenting the business data file into first business data and second business data, wherein the first business data is the data corresponding to the first line number, and the second business data is the data corresponding to other line numbers; a first loading module for loading the first business data sequentially based on the number of constituent fields; and a second loading module for loading the second business data using an existing database loading tool.
[0014] In this embodiment of the application, loading the first business data sequentially based on the number of constituent fields includes: starting from the first row of the first business data, executing the following steps sequentially, row by row, until the last row of the first business data: accumulating the number of fields up to the current row, determining whether the accumulated number has reached the number of constituent fields; if it has, loading the accumulated fields as a group of data, clearing the accumulated number to zero, and moving to the next row to start accumulating again; if it has not reached the number, moving to the next row to continue accumulating.
[0015] A third aspect of this application provides a processor configured to execute the above-described business data file loading method.
[0016] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned business data file loading method.
[0017] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned business data file loading method.
[0018] The above technical solution can filter out abnormal data from the business data file where the number of delimiters is less than the number of constituent fields, i.e., a group of data occupying multiple rows. Then, based on the filtering results, the business data file is divided into normal second business data and abnormal first business data. Finally, the corresponding methods are used for loading, thereby solving the problem of a group of data occupying multiple rows while avoiding a decrease in loading speed.
[0019] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description
[0020] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a process flow of a business data file loading method according to an embodiment of this application; Figure 2 This schematic diagram illustrates a structural block diagram of a service data file loading apparatus according to an embodiment of this application; Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.
[0021] Explanation of reference numerals in the attached figures 200 - Business data file loading device; 202-Receiver Module; 204 - Read module; 206 - Filtering module; 208 - Segmentation Module; 210 - First Loading Module; 212 - Second Loading Module. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0023] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0024] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0025] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. Furthermore, it should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.
[0026] Figure 1 A schematic diagram illustrating a process flow of a business data file loading method according to an embodiment of this application is provided. Figure 1 As shown in one embodiment of this application, a method for loading a business data file is provided, including the following steps: Step 102: Receive files provided by the upstream system, including business data files and data format files.
[0027] Step 104: Obtain the number of constituent fields of a set of data from the data format file.
[0028] Step 106: Filter out the first row number of each row in the business data file whose number of delimiters is less than the number of constituent fields.
[0029] Step 108: Divide the business data file into first business data and second business data, wherein the first business data is the data corresponding to the first line number, and the second business data is the data corresponding to other line numbers.
[0030] Step 110: Based on the number of constituent fields, load the first business data sequentially.
[0031] Step 112: Use an existing database loading tool to load the second business data.
[0032] The above technical solution provides a file transfer standard and a file loading mechanism. Upon receiving a file from the upstream system, the number of constituent fields of a set of data is first obtained from the data format file, thus revealing the composition rules of each set of data. For normal data, each set of data occupies one line, so the number of delimiters in that line should match the number of constituent fields. However, for abnormal data, each set of data occupies multiple lines, so the number of delimiters in each line of that set of data is less than the number of constituent fields. This invention uses this logic to filter out abnormal data that occupies multiple lines. Based on the filtering results, the business data file is split to obtain normal second business data and abnormal first business data. For the second business data, existing database loading tools can be used, such as the MySQL database's `load data` tool, thus avoiding a decrease in loading speed. For the first business data, it can be loaded sequentially according to the number of constituent fields. Since the amount of abnormal data is much smaller than the amount of normal data, the method proposed in this invention can solve the problem of a set of data occupying multiple lines while avoiding a decrease in loading speed.
[0033] Figure 1 This is a flowchart illustrating a business data file loading method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0034] According to a specific implementation method, selecting the first row number where the number of delimiters in each row of data in the business data file is less than the number of constituent fields includes: obtaining the delimiters between fields from the data format file; counting the number of delimiters in each row of data in the business data file; and comparing the number of delimiters in each row of data with the number of constituent fields, and recording the row number where the number of delimiters is less than the number of constituent fields as the first row number.
[0035] After finding the delimiters between fields in the data format file, the delimiting rules between fields in the business data file can be determined. This allows for the counting of delimiters in each row of data in the business data file to find row numbers where the number of delimiters is less than the number of constituent fields. For example, the shell awk command can be used to count the number of delimiters (y) in each row, comparing y with the number of constituent fields (x). If y < x, then that row is designated as the first row.
[0036] According to one specific implementation method, dividing the business data file into first business data and second business data includes: using the shell split command to divide the business data file according to the first line number to obtain the data corresponding to each line number in the first line number; and dividing the data corresponding to each line number in the first line number into the first business data, and dividing the remaining data into the second business data.
[0037] For example, if the first row number selected is (x1, x2, ..., x...) n After splitting using the shell split command, we get [1, x1), [x1, x1+1), [x1+1, x2), [x2, x2+1), [x2+1, x3), [x3, x3+1),…,[x n , x n +1), [x n +1, Max], where x1,x2,x3,…,x n Greater than or equal to 0, Max is the maximum valid line number in the business data file. After the above segmentation, the first business data [x1, x1+1), [x2, x2+1), [x3, x3+1),…,[x n , x n +1), and the second business data [1, x1), [x1+1, x2), [x2+1, x3),…, [x n +1, Max].
[0038] According to a specific implementation method, loading the first business data sequentially based on the number of constituent fields includes: starting from the first row of the first business data, executing the following steps sequentially, row by row, until the last row of the first business data: accumulating the number of fields up to the current row, determining whether the accumulated number has reached the number of constituent fields; if it has, loading the accumulated fields as a group of data, clearing the accumulated number to zero, and moving to the next row to start accumulating again; if it has not reached the number, moving to the next row to continue accumulating.
[0039] It's understandable that although a set of data may be distributed across multiple rows, the total number of fields in the set remains constant, and their distribution order is fixed. The data in the first business data is arranged sequentially by row number; therefore, all abnormal data in the first business data is also arranged sequentially. Thus, when the number of constituent fields is N, starting from the first row of the first business data, accumulating N fields yields a complete set of data.
[0040] According to one specific implementation, the method further includes: before filtering the first row number, determining whether there is a group of data occupying multiple rows; if so, continuing the process; if not, directly loading the business data file.
[0041] To improve processing speed, before filtering, it's advisable to determine if any data sets occupy multiple rows. If not, existing database loading tools can be used to load the business data file, further enhancing processing speed.
[0042] According to one specific implementation method, determining whether there is a set of data occupying multiple rows includes: receiving a data verification file provided by an upstream system; determining whether the number of rows in the business data file is consistent with the number of business data recorded in the data verification file; and if they are consistent, determining that there is no set of data occupying multiple rows, and if they are inconsistent, determining that there is a set of data occupying multiple rows.
[0043] The upstream system also provides a data verification file, which contains information such as the quantity of business data, MD5 checksums, and generation time. If all data in the business data file is normal, then its row count should match the quantity of business data. Therefore, by obtaining the quantity of business data from the data verification file and comparing it with the row count of the business data file, it can be determined whether there is a case where a set of data occupies multiple rows.
[0044] According to one specific implementation, loading the second service data includes: comparing the size of the second service data with a preset first threshold; and if the size of the second service data is less than or equal to the first threshold, then directly loading the second service data; if the size of the second service data is greater than the first threshold, then slicing the second service data to obtain several slice data not greater than a preset second threshold, and loading the slice data in a multi-threaded concurrent manner.
[0045] Understandably, when business data files are exceptionally large, such as reaching hundreds of gigabytes in size, loading them serially would significantly reduce processing speed. Therefore, the technical solution proposed in this invention sets a first threshold. When the size of the second business data exceeds this first threshold, the second business data is sliced to obtain multiple slices, each of which is no larger than the second threshold. Since the second business data consists of normal data, direct slicing avoids splitting a single set of data into two slices. Finally, existing database loading tools, such as the MySQL database's `load data` tool, are used to load the sliced data concurrently, thus avoiding data anomalies and improving processing speed.
[0046] In one embodiment, such as Figure 2 As shown, a business data file loading device 200 is provided, including a receiving module, a reading module, a filtering module, a segmentation module, a first loading module, and a second loading module, wherein: The receiving module 202 is used to receive files provided by the upstream system, including business data files and data format files.
[0047] The reading module 204 is used to obtain the number of constituent fields of a set of data from the data format file.
[0048] The filtering module 206 is used to filter out the first row number of each row of data in the business data file whose number of delimiters is less than the number of constituent fields.
[0049] The segmentation module 208 is used to segment the business data file into first business data and second business data, wherein the first business data is the data corresponding to the first line number, and the second business data is the data corresponding to other line numbers.
[0050] The first loading module 210 is used to load the first business data sequentially based on the number of constituent fields.
[0051] The second loading module 212 is used to load the second business data using an existing database loading tool.
[0052] According to a specific implementation method, loading the first business data sequentially based on the number of constituent fields includes: starting from the first row of the first business data, executing the following steps sequentially, row by row, until the last row of the first business data: accumulating the number of fields up to the current row, determining whether the accumulated number has reached the number of constituent fields; if it has, loading the accumulated fields as a group of data, clearing the accumulated number to zero, and moving to the next row to start accumulating again; if it has not reached the number, moving to the next row to continue accumulating.
[0053] The business data file loading device includes a processor and a memory. The receiving module, reading module, filtering module, segmentation module, first loading module and second loading module are all stored in the memory as program units. The processor executes the program modules stored in the memory to implement the corresponding functions.
[0054] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the method for loading business data files can be implemented by adjusting kernel parameters.
[0055] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0056] This application provides a storage medium on which a program is stored, which, when executed by a processor, implements the above-described business data file loading method.
[0057] This application provides a processor for running a program, wherein the program executes the above-described business data file loading method during runtime.
[0058] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores business data files. The network interface A02 communicates with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a business data file loading method.
[0059] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0060] In one embodiment, the business data file loading device provided in this application can be implemented as a computer program, and the computer program can be configured as follows: Figure 3 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the business data file loading device, for example, Figure 2 The diagram shows a receiving module, a reading module, a filtering module, a segmentation module, a first loading module, and a second loading module. The computer program comprised of these modules causes the processor to execute the steps of the service data file loading methods described in the various embodiments of this application.
[0061] Figure 3 The computer device shown can be used as follows Figure 2 The receiving module 202 in the business data file loading device shown executes step 102, the reading module 204 executes step 104, the filtering module 206 executes step 106, the segmentation module 208 executes step 108, the first loading module 210 executes step 110, and the second loading module 212 executes step 112.
[0062] This application provides an apparatus, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the business data file loading method described in any embodiment of the present invention.
[0063] This application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program for initializing the business data file loading method according to any embodiment of the present invention.
[0064] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0065] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0069] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0070] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0071] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0072] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A service data file loading method characterized by comprising: The method comprises: receiving a file provided by an upstream system, including a business data file and a data format file; obtaining a number of constituent fields of a group of data from the data format file; screening out a first line number of each line of data in the business data file, the number of the split symbol of which is less than the number of constituent fields; segmenting the business data file into first business data and second business data, wherein the first business data is the data corresponding to the first line number, and the second business data is the data corresponding to other line numbers; loading the first business data in turn based on the number of constituent fields; and loading the second business data by using an existing database loading tool.
2. The method of claim 1, wherein, The screening out of the first line number of each line of data in the business data file, the number of the split symbol of which is less than the number of constituent fields, comprises: obtaining the split symbol between fields from the data format file; counting the number of split symbols of each line of data in the business data file; and comparing the number of split symbols of each line of data with the number of constituent fields, and recording each line number of which the number of split symbols is less than the number of constituent fields as the first line number.
3. The method of claim 1, wherein, The segmentation of the business data file into first business data and second business data comprises: segmenting the business data file according to the first line number by using a shell split command to obtain the data corresponding to each line number in the first line number; and dividing the data corresponding to each line number in the first line number into the first business data, and dividing the remaining data into the second business data.
4. The method of claim 1, wherein, The loading of the first business data in turn based on the number of constituent fields comprises: starting from the first line of data of the first business data, executing the following steps in turn in a line unit until the last line of the first business data: accumulating the number of fields up to the current line, judging whether the accumulated number reaches the number of constituent fields, if yes, loading the accumulated fields as a group of data, and clearing the accumulated number and starting to accumulate from the next line; if not, continuing to accumulate from the next line.
5. The method of claim 1, wherein, The method further comprises: before screening the first line number, judging whether there is a case that a group of data occupies multiple lines, if yes, continuing to execute, if not, directly loading the business data file.
6. The method of claim 5, wherein, The judgment of whether there is a case that a group of data occupies multiple lines comprises: receiving a data check file provided by an upstream system; judging whether the number of lines of the business data file and the number of business data recorded in the data check file are consistent; and if yes, judging that there is no case that a group of data occupies multiple lines, if not, judging that there is a case that a group of data occupies multiple lines.
7. The method of claim 1, wherein, The loading of the second business data comprises: comparing the size of the second business data with a preset first threshold; and If the size of the second service data is less than or equal to the first threshold, the second service data is directly loaded; if the size of the second service data is greater than the first threshold, the second service data is sliced to obtain a plurality of slice data not greater than a preset second threshold, and the slice data is loaded based on a multi-thread concurrent mode.
8. A service data file loading apparatus characterized by comprising: The apparatus comprises: A receiving module configured to receive a file provided by an upstream system, the file comprising a service data file and a data format file; A reading module configured to obtain a number of constituent fields of a group of data from the data format file; A screening module configured to screen out a first row number of the service data file in which a number of split symbols of each row of data is less than the number of constituent fields; A splitting module configured to split the service data file into first service data and second service data, wherein the first service data is data corresponding to the first row number, and the second service data is data corresponding to other row numbers; A first loading module configured to load the first service data in turn based on the number of constituent fields; and A second loading module configured to load the second service data using an existing database loading tool.
9. The apparatus of claim 8, wherein, Loading the first service data in turn based on the number of constituent fields comprises: Starting from a first row of data of the first service data, executing the following steps in turn in a unit of row until a last row of the first service data: Accumulating a number of fields up to a current row, determining whether the accumulated number reaches the number of constituent fields, if yes, loading the accumulated fields as a group of data, and clearing the accumulated number and starting to accumulate from a next row; if no, proceeding to the next row to continue accumulating.
10. A processor, comprising: The computer program is configured to implement the service data file loading method according to any one of claims 1 to 7 when executed by a processor.
11. A machine-readable storage medium having stored thereon instructions, the instructions being executable by a machine to cause the machine to: The computer program is configured to implement the service data file loading method according to any one of claims 1 to 7 when executed by a processor.
12. A computer program product comprising a computer program, characterized in that, The computer program is configured to implement the service data file loading method according to any one of claims 1 to 7 when executed by a processor.