File import method, computer device, storage medium and program product

Abstract

This embodiment discloses a file import method, a computer device, a storage medium and a program product. The method includes: obtaining a pre-processed target import file; dynamically adjusting the size of the first-level shards of the target import file according to a first dynamic adjustment strategy; performing a shard reading operation on the target import file according to the dynamically adjusted first-level shard size to obtain a number of first shard data; dynamically adjusting the number of pre-built transmission threads according to a second dynamic adjustment strategy; performing shard processing on a number of first shard data through a number of transmission threads according to a fixed second-level shard size to obtain a number of second shard data, and transmitting through a number of transmission threads. The method performs feedback adjustment on the transmission thread, reasonably shards the data and dynamically creates threads, fully utilizes CPU performance and improves resource utilization, reduces data loss, and improves data import efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a file importing method, a computer device, a storage medium, and a program product. Background Art

[0002] With the rapid development of the information age, data analysis systems need to receive data pushed by various data sources. When a data source pushes a large amount of data in a short period of time, the import system needs to import this data.

[0003] Most of the data import methods in the related art adopt a multi-threaded fixed-slice reading and importing method. Each slice occupies CPU resources exclusively before the data import is completed. Moreover, since different files have different optimal slice sizes, the fixed-slice data import method is not conducive to fully utilizing CPU resources. In the process of data import, a fixed thread is usually used or the transmission thread is determined according to the number of slices, resulting in surplus CPU resources during the transmission of some data. The data import method has low CPU resource utilization, low data import efficiency, and data is easily lost during the import process. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a file import method, a computer device, a storage medium and a program product, which can dynamically perform feedback adjustment on the transmission thread according to the data transmission quality, reasonably segment the data and dynamically create threads, reduce data loss and improve data import efficiency while fully utilizing CPU performance and improving CPU resource utilization.

[0005] In a first aspect, the present disclosure provides a method for importing a file, which adopts the following technical solution:

[0006] Get the preprocessed target import file;

[0007] Dynamically adjusting the size of the first-level shards of the target imported file according to a first dynamic adjustment strategy;

[0008] Performing a shard reading operation on the target import file according to the dynamically adjusted first-level shard size to obtain a plurality of first shard data;

[0009] Dynamically adjusting the number of pre-built transmission threads according to a second dynamic adjustment strategy;

[0010] According to the fixed secondary shard size, the plurality of first shard data are subjected to shard processing operations by a plurality of transmission threads whose number is dynamically adjusted to obtain a plurality of second shard data;

[0011] The plurality of second fragment data are transmitted through the plurality of transmission threads.

[0012] In some embodiments, obtaining a preprocessed target import file includes:

[0013] Obtain the file to be imported through the data import interface;

[0014] Performing format verification or field verification on the file to be imported according to pre-configured verification rules to obtain a verification result;

[0015] Storing the file to be imported and the verification result in a received file directory;

[0016] The received file directory is scanned according to a preset scanning period, and the target import file is searched according to the timestamp when the file to be imported is received in the received file directory.

[0017] In some embodiments, dynamically adjusting the size of the first-level shards of the target imported file according to the first dynamic adjustment strategy includes:

[0018] Obtaining at least one of a current system performance parameter, a file type or a file size of the target import file;

[0019] Based on the current system performance parameter and at least one of the file type or the file size of the target imported file, the primary fragment size of the target imported file is dynamically adjusted.

[0020] In some embodiments, transmitting the plurality of second fragment data through the plurality of transmission threads includes:

[0021] Use several asynchronous threads in the pre-built asynchronous thread pool as transmission threads;

[0022] Performing data verification processing and data cleaning conversion processing on the plurality of second shard data through the plurality of transmission threads;

[0023] According to the current system performance parameters, the processed plurality of second slice data are transmitted through the plurality of transmission threads;

[0024] When lost data of the transmitted plurality of second fragmented data is detected, a retransmission mechanism is started to continue transmitting the lost data from a data transmission position corresponding to the time of transmission failure.

[0025] In some embodiments, after performing data verification processing and data cleaning conversion processing on the plurality of second shard data through the plurality of transmission threads, the method further includes:

[0026] Aggregating the data processing results of the plurality of second shard data to obtain an aggregated result;

[0027] Based on the aggregation result, obtaining first data that has passed verification and has been cleaned and converted, and second data that has failed verification;

[0028] Calculate the data ratio between the first data and the second data;

[0029] Marking the data quality of the second shard data based on the data ratio;

[0030] According to the data quality, the number of transmission threads used to transmit the second fragment data is dynamically adjusted.

[0031] In some embodiments, the method further comprises:

[0032] Use several synchronization threads in the pre-built synchronization thread pool as monitoring threads;

[0033] The data transmission process of the plurality of transmission threads is monitored by the plurality of monitoring threads to obtain a monitoring result;

[0034] When it is confirmed according to the monitoring result that the plurality of second fragment data transmitted by the plurality of transmission threads is greater than or equal to a preset data transmission threshold, negative feedback adjustment is performed on the plurality of transmission threads to control the plurality of second fragment data transmitted by the plurality of transmission threads.

[0035] In some embodiments, the method further comprises:

[0036] Determining the transmission quality of the plurality of second fragmented data according to the single-piece transmission time length and transmission integrity of the plurality of second fragmented data;

[0037] When the transmission quality is greater than or equal to a preset transmission quality threshold, gradually increase the transmission value of the plurality of second fragmented data, and record the current fragment sizes of the plurality of second fragmented data;

[0038] When the transmission quality is less than a preset transmission quality threshold, the sending of the plurality of second fragment data is rolled back.

[0039] In a second aspect, the present disclosure also provides a file importing device, which adopts the following technical solution:

[0040] An acquisition unit is configured to acquire a preprocessed target import file;

[0041] A first adjustment unit is configured to dynamically adjust the size of the first-level fragments of the target imported file according to a first dynamic adjustment strategy;

[0042] A first sharding unit is configured to perform a sharding reading operation on the target import file according to the dynamically adjusted primary sharding size to obtain a plurality of first sharding data;

[0043] A second adjustment unit is configured to dynamically adjust the number of pre-built transmission threads according to a second dynamic adjustment strategy;

[0044] The second sharding unit is configured to perform sharding processing operations on the plurality of first sharded data through a plurality of transmission threads whose numbers are dynamically adjusted according to a fixed secondary sharding size, to obtain a plurality of second sharded data;

[0045] The transmission unit is configured to transmit the plurality of second fragment data through the plurality of transmission threads.

[0046] In a third aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:

[0047] The computer device comprises:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned file importing methods.

[0051] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned file importing methods.

[0052] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0053] The embodiments of the present disclosure provide a file import method, a computer device, a storage medium, and a program product, which can dynamically perform feedback adjustment on the transmission thread according to the data transmission quality, reasonably segment the data and dynamically create threads, dynamically adjust the size of the first-level segment, and use a fixed second-level segment size for multi-threaded transmission, thereby reducing data loss and improving data import efficiency while fully utilizing CPU performance and improving CPU resource utilization.

[0054] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A flowchart of a file import method provided in an embodiment of the present disclosure;

[0057] Figure 2 A schematic diagram of the structure of a file importing device provided by an embodiment of the present disclosure;

[0058] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0060] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0061] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0062] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0063] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0064] Figure 1 A flowchart of a file import method provided in an embodiment of the present disclosure is provided. A file import method provided in an embodiment of the present disclosure includes the following steps:

[0065] S101, obtaining a preprocessed target import file.

[0066] For example, a file to be imported is obtained through a data import interface, format verification and field verification are performed on the file to be imported according to a predefined interface specification, and the received file to be imported is stored.

[0067] S102: Dynamically adjust the size of the first-level fragments of the target imported file according to a first dynamic adjustment strategy.

[0068] For example, two shard reading and shard processing operations of indefinite length are performed according to the file type of the target import file, the file quality of the target import file, or system performance parameters.

[0069] S103: Perform a shard reading operation on the target import file according to the dynamically adjusted first-level shard size to obtain a plurality of first shard data.

[0070] The system shard size has a baseline shard value (baseline shard size value) at the beginning of the system. The baseline shard value is determined based on the file type, the number of lines in the file, and the system memory size.

[0071] For example, for an Excel file, the baseline segment size value = 1024 + memory * 2.

[0072] S104: Dynamically adjust the number of pre-built transmission threads according to a second dynamic adjustment strategy.

[0073] For example, the number of transmission threads can be adjusted based on system performance parameters and the data processing quality of the target import file to fully utilize system resources, speed up data import, and improve data import efficiency.

[0074] S105 . According to the fixed secondary shard size, a plurality of transmission threads whose numbers are dynamically adjusted are used to perform shard processing operations on the plurality of first shard data to obtain a plurality of second shard data.

[0075] S106: Transmit the plurality of second fragment data through the plurality of transmission threads.

[0076] The disclosed embodiments can dynamically perform feedback adjustment on the transmission threads according to the data transmission quality, reasonably segment the data and dynamically create threads, dynamically adjust the size of the first-level segment, and use a fixed second-level segment size for multi-threaded transmission, thereby reducing data loss and improving data import efficiency while fully utilizing CPU performance and improving CPU resource utilization.

[0077] In some embodiments, obtaining a preprocessed target import file includes:

[0078] Obtain the file to be imported through the data import interface;

[0079] Performing format verification or field verification on the file to be imported according to pre-configured verification rules to obtain a verification result;

[0080] Storing the file to be imported and the verification result in a received file directory;

[0081] The received file directory is scanned according to a preset scanning period, and the target import file is searched according to the timestamp when the file to be imported is received in the received file directory.

[0082] Optionally, the format check or field check includes a blacklist check on the source of the imported file, a check on the type of the received file, and a check on whether the server returned by the sender is busy.

[0083] Optionally, a scheduled task can be started in the background. For example, with a preset scanning period of every second, a scheduled scan is performed on the received file directory to find the earliest received target import file, and the target import file is read in segments. The size of the read segments is dynamically adjusted according to the file type of the target import file, the current system performance parameters, and the file quality of the processed file.

[0084] In some embodiments, dynamically adjusting the size of the first-level shards of the target imported file according to the first dynamic adjustment strategy includes:

[0085] Obtaining at least one of a current system performance parameter, a file type or a file size of the target import file;

[0086] Based on the current system performance parameter and at least one of the file type or the file size of the target imported file, the primary fragment size of the target imported file is dynamically adjusted.

[0087] In some embodiments, transmitting the plurality of second fragment data through the plurality of transmission threads includes:

[0088] Use several asynchronous threads in the pre-built asynchronous thread pool as transmission threads;

[0089] Performing data verification processing and data cleaning conversion processing on the plurality of second shard data through a plurality of transmission threads;

[0090] According to the current system performance parameters, the processed plurality of second slice data are transmitted through a plurality of transmission threads.

[0091] Optionally, an asynchronous thread pool and a synchronous thread pool are constructed in the system, and a plurality of asynchronous threads in the asynchronous thread pool perform secondary sharding processing operations on a plurality of first sharded data to obtain a plurality of second sharded data. The number of threads of the asynchronous thread can be dynamically adjusted according to the transmission quality of the target imported file and the system performance parameters to achieve the optimal thread data.

[0092] In some embodiments, after performing data verification processing and data cleaning conversion processing on a plurality of second shard data through a plurality of transmission threads, the method further includes:

[0093] Aggregating the data processing results of the plurality of second shard data to obtain an aggregated result;

[0094] Based on the aggregation result, obtaining first data that has passed verification and has been cleaned and converted, and second data that has failed verification;

[0095] Calculate the data ratio between the first data and the second data;

[0096] Marking the data quality of the second shard data based on the data ratio;

[0097] According to the data quality, the secondary shard size is dynamically adjusted, thereby establishing a corresponding number of multi-threaded transmission connections to the file server according to the secondary shard size of the second shard data of the transmitted file, and dynamically adjusting the number of transmission threads used to transmit the second shard data.

[0098] Optionally, the second sharding operation of the embodiment of the present disclosure is a sharding of a fixed number of rows, and an asynchronous thread is started to perform sharding operations on a number of first sharding data to obtain a number of second sharding data after sharding, and pre-processing operations such as verification, cleaning, and conversion are performed on the number of second sharding data. After all the second sharding data are pre-processed, the data pre-processing results of each second sharding data are aggregated to obtain the first data that has passed the verification and the cleaning and conversion processing and the second data that failed the verification. The data quality of the second sharding data is marked according to the data ratio of the first data that has passed the verification and the second data that has not passed the verification, and the value of the sharding size of the next round is dynamically adjusted according to the data quality of this round (that is, the secondary sharding size is dynamically adjusted). For example, the higher the data quality, the faster the data processing speed, and the value of the sharding size can be appropriately increased. Conversely, the lower the data quality, the slower the data processing speed, and the value of the sharding size can be appropriately reduced.

[0099] Optionally, the numerical adjustment formula for the shard size is as follows:

[0100] The value of the fragment size = the value of the initial fragment size * data quality / ideal quality ratio.

[0101] Optionally, a corresponding number of multi-threaded transmission connections are established to the file server according to the number of slices of the second slice data of the target imported file to be transmitted (ie, the value of the slice size).

[0102] In some embodiments, the method further comprises:

[0103] Use several synchronization threads in the pre-built synchronization thread pool as monitoring threads;

[0104] The data transmission process of the plurality of transmission threads is monitored by a plurality of monitoring threads to obtain a monitoring result;

[0105] When it is confirmed according to the monitoring result that the plurality of second fragment data transmitted by the plurality of transmission threads is greater than or equal to a preset data transmission threshold, negative feedback adjustment is performed on the plurality of transmission threads to control the plurality of second fragment data transmitted by the plurality of transmission threads.

[0106] It should be noted that the user can set the data transmission threshold according to actual business needs, and the embodiments of the present disclosure are not limited to this.

[0107] Optionally, the asynchronous thread controls the transmitted data of several second shards through an atomic custom current limiter, and when the transmitted data of several second shards is greater than or equal to a preset data transmission threshold, negative feedback regulation is performed on the asynchronous thread that reads and processes the data.

[0108] In some embodiments, the method further comprises:

[0109] Determining the transmission quality of the plurality of second fragmented data according to the single-piece transmission time length and transmission integrity of the plurality of second fragmented data;

[0110] When the transmission quality is greater than or equal to a preset transmission quality threshold, gradually increase the transmission value of the plurality of second fragmented data, and record the current fragment sizes of the plurality of second fragmented data;

[0111] When the transmission quality is less than a preset transmission quality threshold, the sending of the plurality of second fragment data is rolled back.

[0112] Optionally, the value of the fragment size is dynamically adjusted according to the integrity of the receiving party. When the transmission quality of several second fragment data is excellent, the TCP congestion control algorithm can be used to gradually increase the transmission value of several second fragment data, and record the value of the current fragment size of several second fragment data; after several second fragment data reach the transmission peak, the transmission quality is evaluated according to the single-chip transmission time length and transmission integrity of several second fragment data. When the transmission quality of several second fragment data decreases, the congestion avoidance algorithm is used to send back, and the transmission quality of several second fragment data is continuously monitored.

[0113] Optionally, the TCP congestion control algorithm of the embodiment of the present disclosure may include a slow start algorithm, a congestion avoidance algorithm, a fast retransmit algorithm, a fast recovery algorithm, etc.

[0114] It should be noted that the user can set the transmission quality threshold according to actual business needs, and the embodiments of the present disclosure are not limited to this.

[0115] In some embodiments, when the data transmission of the second shard is abnormal, the synchronization thread initiates an asynchronous scan to determine the thread survival status of the synchronization thread. If the survival status of the synchronization thread is a non-survival state, a new data synchronization thread is established through the synchronization process to prevent the transmission thread from being stuck or frozen.

[0116] In some embodiments, when data loss of the transmitted second slice data is detected, a retransmission mechanism is started, and the lost data is continued to be transmitted from the data transmission position corresponding to the transmission failure. In the embodiment of the present disclosure, when data loss is found in the transmitted second slice data, a retransmission mechanism is started, and the lost data is continued to be transmitted from the transmission position of the second slice data corresponding to the transmission failure. According to the system definition, the second slice data is imported into a specified location or a corresponding database.

[0117] Figure 2 This is a schematic diagram of the structure of a file importing device provided in an embodiment of the present disclosure. The present disclosure also provides a file importing device, including:

[0118] An acquisition unit 21 is configured to acquire a preprocessed target import file;

[0119] A first adjustment unit 22 is configured to dynamically adjust the size of the first-level fragments of the target imported file according to a first dynamic adjustment strategy;

[0120] The first sharding unit 23 is configured to perform a sharding reading operation on the target import file according to the dynamically adjusted primary sharding size to obtain a plurality of first sharding data;

[0121] A second adjustment unit 24 is configured to dynamically adjust the number of pre-built transmission threads according to a second dynamic adjustment strategy;

[0122] The second sharding unit 25 is configured to perform sharding processing operations on the plurality of first sharded data through a plurality of transmission threads whose numbers are dynamically adjusted according to a fixed secondary sharding size, to obtain a plurality of second sharded data;

[0123] The transmission unit 26 is configured to transmit the plurality of second fragment data through the plurality of transmission threads.

[0124] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0125] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device performs all or part of the steps of the file import method of each embodiment of the present disclosure.

[0126] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0127] like Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Figure 3 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Figure 3 As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0129] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 3 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0130] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the file import method of the embodiment of the present disclosure are executed.

[0131] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0132] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the file importing method of each embodiment of the present disclosure are executed.

[0133] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0134] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0135] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0136] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0137] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0138] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0139] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0140] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0141] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A file import method, characterized in that: include: Build asynchronous thread pools and synchronous thread pools in the system; Get the preprocessed target import file; Dynamically adjusting the size of the first-level shards of the target imported file according to a first dynamic adjustment strategy; Performing a shard reading operation on the target import file according to the dynamically adjusted first-level shard size to obtain a plurality of first shard data; Dynamically adjusting the number of pre-built transmission threads according to a second dynamic adjustment strategy; According to the fixed secondary shard size, the plurality of first shard data are subjected to shard processing operations by a plurality of transmission threads whose number is dynamically adjusted to obtain a plurality of second shard data; Transmitting the plurality of second fragment data through the plurality of transmission threads includes: Use several asynchronous threads in the pre-built asynchronous thread pool as transmission threads; Performing data verification processing and data cleaning conversion processing on the plurality of second shard data through the plurality of transmission threads; According to the current system performance parameters, the processed plurality of second slice data are transmitted through the plurality of transmission threads; When lost data of the transmitted data of the plurality of second fragments is detected, a retransmission mechanism is started to continue transmitting the lost data from a data transmission position corresponding to when the transmission failed; After performing data verification processing and data cleaning conversion processing on the plurality of second shard data through the plurality of transmission threads, the method further includes: Aggregating the data processing results of the plurality of second shard data to obtain an aggregated result; Based on the aggregation result, obtaining first data that has passed verification and has been cleaned and converted, and second data that has failed verification; Calculate the data ratio between the first data and the second data; Marking the data quality of the second shard data based on the data ratio; Dynamically adjust the number of transmission threads used to transmit the second fragment data according to the data quality; The method further comprises: Use several synchronization threads in the pre-built synchronization thread pool as monitoring threads; The data transmission process of the plurality of transmission threads is monitored by the plurality of monitoring threads to obtain a monitoring result; When it is confirmed according to the monitoring result that the plurality of second fragment data transmitted by the plurality of transmission threads is greater than or equal to a preset data transmission threshold, negative feedback adjustment is performed on the plurality of transmission threads to control the plurality of second fragment data transmitted by the plurality of transmission threads.

2. The file importing method according to claim 1, characterized in that: Get the preprocessed target import file, including: Obtain the file to be imported through the data import interface; Performing format verification or field verification on the file to be imported according to pre-configured verification rules to obtain a verification result; Storing the file to be imported and the verification result in a received file directory; The received file directory is scanned according to a preset scanning period, and the target import file is searched according to the timestamp when the file to be imported is received in the received file directory.

3. The file importing method according to claim 1, characterized in that: Dynamically adjusting the size of the first-level fragments of the target imported file according to the first dynamic adjustment strategy includes: Obtaining at least one of a current system performance parameter, a file type or a file size of the target import file; Based on the current system performance parameter and at least one of the file type or the file size of the target imported file, the primary fragment size of the target imported file is dynamically adjusted.

4. The file importing method according to claim 1, characterized in that: The method further comprises: Determining the transmission quality of the plurality of second fragmented data according to the single-piece transmission time length and transmission integrity of the plurality of second fragmented data; When the transmission quality is greater than or equal to a preset transmission quality threshold, gradually increasing the transmission value of the plurality of second fragmented data, and recording the current fragmented sizes of the plurality of second fragmented data; When the transmission quality is less than a preset transmission quality threshold, the sending of the plurality of second fragment data is rolled back.

5. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the file import method described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the file importing method described in any one of claims 1 to 4.

7. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images