Data import method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to the technical field of big data processing, and particularly discloses a data importing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: dividing a to-be-imported data set to obtain respective service data sets of various service types; based on the data dependency relationship among the respective service data of the plurality of service types, determining the import priorities of the respective service data sets of the plurality of service types, and according to the import priorities, executing data import by taking the respective service data sets of the plurality of service types as target data sets in sequence; for each time of data import, when the data size of the target data set meets a splitting import condition, splitting the target data set to obtain a plurality of target data subsets; and respectively executing data import on the plurality of target data subsets to obtain a data import result of the target data set. By adopting the method, the data importing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a data import method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of big data processing technology, data import technology has emerged. Data import technology refers to the process of migrating data from external data sources to the target system or database. This process usually involves data cleaning, conversion, and loading to ensure the accuracy and availability of the data. Data import technology has a wide range of applications in many fields, including but not limited to accurate recommendations, knowledge graphs, etc. In traditional technology, data import is done by importing and calculating all data items sequentially in the same execution task.

[0003] However, although the traditional method has a relatively simple operation logic when importing data, the resource utilization efficiency of the device performing the data import is relatively low, the entire import operation is very time-consuming, and there is a problem of low data import efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data import method, device, computer equipment, computer-readable storage medium and computer program product that can improve data import efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data import method, comprising:

[0006] When a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of respective multiple business types;

[0007] Determining the import priorities of the business data sets of the multiple business types based on the data dependency relationships between the business data of the multiple business types, and importing the business data sets of the multiple business types in sequence as target data sets according to the import priorities;

[0008] For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets;

[0009] Data import is performed on the multiple target data subsets respectively to obtain data import results of the target data set.

[0010] In one embodiment, data import is performed on the multiple target data subsets respectively to obtain the data import result of the target data set, including:

[0011] For each of the target data subsets, create a first sub-thread corresponding to the target data subset, and use the first sub-thread to perform data import on the target data subset to obtain a data import result of the target data subset;

[0012] Based on the data import results of each of the multiple target data subsets, the data import result of the target data set is obtained.

[0013] In one embodiment, the data importing of the target data subset is performed by the first sub-thread to obtain the data import result of the target data subset, including:

[0014] When the data amount of the target data subset meets the splitting and importing condition, the target data subset is split by the first sub-thread to obtain a plurality of split data subsets;

[0015] For each of the split data subsets, create a second sub-thread corresponding to the split data subset, and use the second sub-thread to perform data import on the split data subset to obtain a data import result for the split data subset;

[0016] Based on the data import results of each of the multiple split data subsets, the data import result of the target data subset is obtained.

[0017] In one embodiment, the data import method further includes:

[0018] When the data volume of the target data set meets the direct import condition, data import is performed on the target data set to obtain a data import result of the target data set.

[0019] In one embodiment, the data import method further includes:

[0020] Get the number of cores of the processor used to perform data import;

[0021] Determine the amount of data to be imported by the thread based on the amount of data in the target data set and the number of cores;

[0022] When the thread import data volume is greater than a preset thread data processing volume threshold, it is determined that the data volume of the target data set meets the split import condition.

[0023] In one embodiment, the data set to be imported is divided to obtain business data sets of various business types, including:

[0024] Identify data identifiers of respective ones of the plurality of data to be imported in the data set to be imported;

[0025] Based on the data identifiers of the multiple data to be imported and the corresponding relationship between the data identifiers and the business types, the multiple data to be imported are divided to obtain business data sets of the multiple business types.

[0026] In a second aspect, the present application also provides a data import device, comprising:

[0027] A data partitioning module is used to, when receiving a data import request, obtain a set of data to be imported corresponding to the data import request, and partition the set of data to be imported to obtain business data sets of various business types;

[0028] A sorting module, configured to determine the import priority of the business data sets of the multiple business types based on the data dependency relationship between the business data of the multiple business types, and to import the business data sets of the multiple business types in sequence as target data sets according to the import priority;

[0029] A splitting module is used for splitting the target data set to obtain multiple target data subsets for each data import when the data volume of the target data set meets the splitting and importing conditions;

[0030] The import module is used to perform data import on the multiple target data subsets respectively to obtain the data import result of the target data set.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] When a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of respective multiple business types;

[0033] Determining the import priorities of the business data sets of the multiple business types based on the data dependency relationships between the business data of the multiple business types, and importing the business data sets of the multiple business types in sequence as target data sets according to the import priorities;

[0034] For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets;

[0035] Data import is performed on the multiple target data subsets respectively to obtain data import results of the target data set.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] When a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of respective multiple business types;

[0038] Determining the import priorities of the business data sets of the multiple business types based on the data dependency relationships between the business data of the multiple business types, and importing the business data sets of the multiple business types in sequence as target data sets according to the import priorities;

[0039] For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets;

[0040] Data import is performed on the multiple target data subsets respectively to obtain data import results of the target data set.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] When a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of respective multiple business types;

[0043] Determining the import priorities of the business data sets of the multiple business types based on the data dependency relationships between the business data of the multiple business types, and importing the business data sets of the multiple business types in sequence as target data sets according to the import priorities;

[0044] For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets;

[0045] Data import is performed on the multiple target data subsets respectively to obtain data import results of the target data set.

[0046] The above-mentioned data import method, device, computer equipment, computer-readable storage medium and computer program product, when receiving a data import request, obtains the to-be-imported data set corresponding to the data import request, divides the to-be-imported data set, can use multiple business types to realize the classification of the to-be-imported data, obtains the business data sets of the multiple business types respectively, and determines the import priority of the business data sets of the multiple business types respectively based on the data dependency relationship between the business data of the multiple business types respectively. It is possible to use the data dependency relationship to realize the determination of the import priority, so that the business data sets of the multiple business types respectively can be used as the target data sets in turn to execute data import according to the import priority, so as to avoid the lack of dependent data when importing data. It is necessary to pause and then continue to execute data import after importing the business data sets of other businesses. This can improve the efficiency of data import. For each data import, when the data volume of the target data set meets the split import conditions, the target data set is split to obtain multiple target data subsets. The target data set can be quickly grouped by splitting, thereby realizing the control of the data volume of a single data import. Then, data import can be performed on multiple target data subsets separately to obtain the data import result of the target data set, realizing parallel data import based on controlling the data volume, avoiding the situation where too much data during data import causes memory overflow and then causes data import failure, and making full use of parallel computing capabilities to improve data import efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is an application environment diagram of a data import method in one embodiment;

[0049] FIG2 is a schematic diagram of a flow chart of a data import method in one embodiment;

[0050] FIG3 is a schematic diagram of determining an import priority in one embodiment;

[0051] FIG4 is a schematic diagram of executing data import of a target data subset in one embodiment;

[0052] FIG5 is a schematic diagram of determining whether the split import condition is met in one embodiment;

[0053] FIG6 is a schematic diagram of a flow chart of a data import method in another embodiment;

[0054] FIG7 is a schematic diagram of a flow chart of a data import method in yet another embodiment;

[0055] FIG8 is a schematic diagram of executing data import after recursive segmentation in one embodiment;

[0056] FIG9 is a structural block diagram of a data import device in one embodiment;

[0057] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0059] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0060] The data import method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. When receiving the data import request sent by the terminal 102, the server 104 obtains the data set to be imported corresponding to the data import request, divides the data set to be imported, obtains the business data sets of each of the multiple business types, and determines the import priority of the business data sets of each of the multiple business types based on the data dependency relationship between the business data of each of the multiple business types. According to the import priority, the business data sets of each of the multiple business types are sequentially used as the target data sets to perform data import. For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets, and data import is performed on the multiple target data subsets respectively to obtain the data import result of the target data set. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0061] In an exemplary embodiment, as shown in FIG2 , a data import method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps 202 to 208. Among them:

[0062] Step 202: when a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of various business types.

[0063] Among them, a data import request refers to a request to import data from an external data source into a server. The data source may specifically be a database, a file system, etc. A set of data to be imported refers to a set of multiple data to be imported, and the data to be imported refers to data that needs to be imported into the server but has not yet been imported. For example, the data to be imported may specifically be a data file, which is an electronic document mainly used to store data in a computer. The data may be in various forms such as numbers, text, images, audio, and video. Data files have a specific file format to ensure the correct storage and reading of data. Through data files, users can easily share and transfer data between different programs and systems.

[0064] Among them, business type refers to the type of business provided by an organization to meet different market needs, customer groups or goals. It is understandable that the business type will vary according to the business scope and market positioning of different organizations. For example, if the organization is a bank, the business types provided may include but are not limited to deposit business, payment and settlement services, wealth management business, etc. Business data set refers to a collection of a series of data related to the business type, which is organized and stored for analysis, reporting and decision support.

[0065] Exemplarily, when receiving a data import request, the server obtains the data set to be imported corresponding to the data import request, and divides the data set to be imported based on the data identifiers of the data to be imported in the data set to be imported to obtain business data sets for each of the multiple business types.

[0066] In a specific application, when receiving a data import request, the server can determine the data range of the data to be imported according to the data import request, and then obtain the to-be-imported data set corresponding to the data import request based on the data range. The data range refers to the breadth and depth of the data involved in the data import operation, and can include at least one of the data type, data source, and data time span.

[0067] In a specific application, the type of data refers to the classification or nature of the data, which determines how the data is stored, processed and analyzed. In this embodiment, the type of data can be a variety of business types, that is, through the data import request, it can be specified which business types of data to import. The source of the data refers to the place where the data is generated or obtained. For example, the source of the data can be an internal system, an external source, etc. The time span of the data refers to the time range covered by the data, which depends on the analysis requirements and the timeliness of the data.

[0068] Step 204, based on the data dependency relationship between the business data of the various business types, determine the import priority of the business data sets of the various business types, and according to the import priority, perform data import on the business data sets of the various business types in turn as the target data sets.

[0069] The data dependency relationship between business data refers to the logical connection and interaction between different data sets or data items in business processes and data processing, which determines how the data is generated, processed and used. In this embodiment, the data dependency relationship mainly refers to the reference relationship between business data, that is, one business data references or depends on another business data.

[0070] For example, a business data references or depends on another business data, which specifically means that a business data contains another business data. For example, if business data A contains business data B, then business data A can be considered to reference business data B. For further example, business data A can be a sales record, and business data B can be a product catalog. The sales record will reference the product ID in the product catalog to identify which products are sold. Business data A can be a reimbursement application, and business data B can be an expense type. The reimbursement application will reference a predefined expense type, such as rate difference, office supplies, etc.

[0071] For example, one business data refers to or depends on another business data, which specifically means that the calculation of one business data depends on the other business data, that is, the value of one business data is obtained through a certain calculation or formula based on the value of another business data. If the value of business data C is obtained through a certain calculation or formula based on the value of business data D, then it can be considered that the calculation of business data C depends on business data D. For further example, business data C can be specifically the sales tax of goods, and business data D can be specifically the sales amount of goods, then the calculation of sales tax of goods needs to depend on the sales amount of goods. Business data C can be specifically the profit, and business data D can be specifically the sales revenue and cost, then the calculation of profit needs to depend on the sales revenue and cost.

[0072] The import priority refers to the order in which the business data sets of various business types are imported into the server. It is understandable that since the business data sets of some business types may depend on the existence of business data sets of other business types, the dependent business data sets need to be imported first.

[0073] Exemplarily, based on the data dependency relationship between the business data of multiple business types, the server can sort the business data sets of the multiple business types, determine the import priority of the business data sets of the multiple business types, and then according to the import priority, perform data import with the business data sets of the multiple business types as target data sets in turn.

[0074] In a specific application, the data dependency relationship between the business data of all business types is pre-stored in the server. After obtaining the business data sets of multiple business types, the server can determine the data dependency relationship between the business data of multiple business types by querying the pre-stored data dependency relationship between the business data of all business types.

[0075] In specific applications, the business data sets of the various business types obtained each have an initial order. When sorting, the server will, based on data dependencies, sort the business data sets that reference or depend on other business data sets after the referenced or dependent business data sets, thereby obtaining the import priority of the business data sets of the various business types.

[0076] In a specific application, the initial order can be determined according to the data volume order of the service data sets of the various service types, specifically, the data volume can be in ascending order or in descending order. In addition, the initial order can also be determined according to the service level of the various service types, where the service data sets of the service types with high service levels are arranged in front, and the service data sets of the service types with low service levels are arranged in the back. Among them, the service levels of all service types can be configured according to the actual application scenarios.

[0077] In a specific application, taking the business data sets of multiple business types including business data set 1, business data set 2, business data set 3, business data set 4, business data set 5, and business data set 6, and the initial order is determined by the ascending order of data volume, and the ascending order of data volume is business data set 1-business data set 3-business data set 6-business data set 5-business data set 4-business data set 2, and business data set 3 references business data set 5, and business data set 6 depends on business data set 4 as an example, the initial order can be as shown in Figure 3, which is business data set 1-business data set 3-business data set 6-business data set 5-business data set 4-business data set 2. After sorting based on the data dependency relationship, the resulting import priority is shown in Figure 3, which is business data set 1-business data set 5-business data set 3-business data set 4-business data set 6-business data set 2.

[0078] Step 206 , for each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets.

[0079] The data volume of the target data set refers to the number of data items included in the target data set, and in this embodiment, it refers to the number of data to be imported included in the target data set. For example, if the data to be imported is a data file, the data volume of the target data set refers to the number of data files included in the target data set.

[0080] The split import condition refers to the condition for splitting a data set before importing it when performing data import on any data set, which can be configured according to the actual application scenario. The target data subset refers to the subset of the target data set obtained after splitting the target data set, which contains a part of the data to be imported in the target data set.

[0081] Exemplarily, for each data import, the server will determine the data volume of the target data set, determine whether the data volume of the target data set meets the split import condition, and if the data volume of the target data set meets the split import condition, split the target data set to obtain multiple target data subsets. In a specific application, the split import condition may specifically be that the thread import data volume determined based on the data volume of the data set is greater than a preset thread data processing volume threshold. In this case, when the thread import data volume determined based on the data volume of the target data set is greater than the preset thread data processing volume threshold, the server will split the target data set to obtain multiple target data subsets. Among them, the thread data processing volume threshold refers to the maximum amount of data that can be efficiently processed by a single thread when processing data, and can be configured according to the actual application scenario. The thread import data volume can be the ratio of the data volume of the data set to the number of cores of the processor.

[0082] In a specific application, the server can split the target data set according to a preset split number, and split the target data set into multiple target data subsets. Specifically, the server calculates the ratio between the data volume of the target data set and the preset split number, and then determines the data volume of each target data subset according to the ratio, and finally splits the target data set based on the data volume of each target data subset to obtain multiple target data subsets. Among them, the preset split number can be configured according to the actual application scenario.

[0083] In a specific application, when determining the data amount of each target data subset based on a ratio, if the ratio is a positive integer, the ratio can be directly used as the data amount of each target data subset. If the ratio is not a positive integer, the data amount of each target data subset needs to be allocated based on the ratio, and there may be a situation where the data amount of at least one target data subset is greater than the ratio.

[0084] For example, assuming that the preset number of splits is 2, and the data volume of the target data set is (2N+1), N is a positive integer, and the ratio of the data volume of the target data set to the preset number of splits is (N+0.5) through calculation, which is not a positive integer. The server needs to allocate the data volume of each target data subset based on the ratio, specifically, the data volume of one target data subset is N (less than the ratio N+0.5), and the data volume of another target data subset is N+1 (greater than the ratio N+0.5). To further illustrate, if the data volume of the target data set is 2N, the ratio obtained by calculation is N, which is a positive integer, and the server can directly use N as the data volume of each target data subset.

[0085] Step 208: perform data import on multiple target data subsets respectively to obtain data import results of the target data set.

[0086] Exemplarily, the server will perform data import on multiple target data subsets respectively, obtain the data import results of each of the multiple target data subsets, and then obtain the data import result of the target data set based on the data import results of each of the multiple target data subsets. In a specific application, for each target data subset, the server will create a subthread to perform data import on the target data subset and obtain the data import result of the target data subset.

[0087] The above-mentioned data import method, when receiving a data import request, obtains the to-be-imported data set corresponding to the data import request, divides the to-be-imported data set, can use multiple business types to realize the classification of the to-be-imported data, obtains the business data sets of the multiple business types respectively, and determines the import priority of the business data sets of the multiple business types respectively based on the data dependency relationship between the business data of the multiple business types respectively. The data dependency relationship can be used to realize the determination of the import priority, so that the business data sets of the multiple business types respectively can be used as the target data sets to execute data import in turn according to the import priority, so as to avoid the need to pause and pause the data import due to the lack of dependent data during the data import and import the business data of other businesses. After the service data set is imported, continuing to import data can improve the data import efficiency. For each data import, when the data volume of the target data set meets the split import conditions, the target data set is split to obtain multiple target data subsets. The target data set can be quickly grouped by splitting, thereby realizing the control of the data volume of a single data import. Then, data import can be performed on multiple target data subsets respectively to obtain the data import result of the target data set, realizing parallel data import based on controlling the data volume, avoiding the situation where the memory overflow caused by excessive data volume during data import and then the data import failure is caused, and making full use of parallel computing capabilities can improve the data import efficiency.

[0088] In an exemplary embodiment, data import is performed on multiple target data subsets respectively to obtain data import results of the target data set, including:

[0089] For each target data subset, create a first sub-thread corresponding to the target data subset, and execute data import of the target data subset through the first sub-thread to obtain a data import result of the target data subset;

[0090] Based on the data import results of each of the multiple target data subsets, the data import result of the target data set is obtained.

[0091] Among them, the sub-thread, also called auxiliary thread or working thread, is a thread used to perform a specific task relative to the main thread that executes the data set to be imported. In this embodiment, the first sub-thread refers to a thread used to perform the data import task for the target data subset. It can be understood that the creation and use of sub-threads can improve parallel processing capabilities, improve the responsiveness of data import, and more effectively utilize the computing resources of multiple processors.

[0092] Exemplarily, for each target data subset, the server creates a first subthread corresponding to the target data subset, and executes data import for the target data subset through the first subthread to obtain the data import result of the target data subset. After obtaining the data import results of each of the multiple target data subsets, the server obtains the data import result of the target data set based on the data import results of each of the multiple target data subsets.

[0093] In a specific application, when the first sub-thread executes data import of the target data subset, the first sub-thread will determine whether the data volume of the target data subset meets the direct import condition. If the data volume of the target data subset meets the direct import condition, the first sub-thread can directly import the data of the target data subset. If the data of the target data subset does not meet the direct import condition, that is, it meets the split import condition, the first sub-thread needs to split the target data subset first, and then import the split data subset. Among them, the direct import condition refers to the condition of directly importing the data set without splitting it when executing data import on any data set, which can be configured according to the actual application scenario. For example, the direct import condition can specifically be that the thread import data volume determined based on the data volume of the data set is less than or equal to the preset thread data processing volume threshold. Among them, the thread import data volume can be the ratio of the data volume of the data set to the number of cores of the processor. In this embodiment, multiple first sub-threads are created, and multiple target data subsets are imported respectively by using the multiple first sub-threads. After obtaining the data import results of each of the multiple target data subsets, the data import results of the target data set are obtained based on the data import results of each of the multiple target data subsets. Multiple first sub-threads can be used to realize parallel processing of multiple target data subsets, thereby improving data import efficiency.

[0094] In an exemplary embodiment, as shown in FIG4 , the data import of the target data subset is executed by the first subthread to obtain the data import result of the target data subset, including steps 402 to 406. Among them:

[0095] Step 402: when the data volume of the target data subset meets the splitting and importing condition, the target data subset is split by the first subthread to obtain a plurality of split data subsets;

[0096] Step 404: for each split data subset, create a second sub-thread corresponding to the split data subset, and use the second sub-thread to perform data import on the split data subset to obtain a data import result for the split data subset;

[0097] Step 406: Based on the data import results of each of the multiple split data subsets, the data import result of the target data subset is obtained.

[0098] The split data subset refers to a subset of the target data subset obtained after the target data subset is split. The second sub-thread refers to a thread for executing the data import task for the split data subset.

[0099] Exemplarily, when the first sub-thread is used to perform data import of the target data subset, the first sub-thread in the server will determine whether the data volume of the target data subset meets the split import condition. If the data volume of the target data subset meets the split import condition, the target data subset will be split to obtain multiple split data subsets, and for each split data subset, a second sub-thread corresponding to the split data subset will be created, and the data import of the split data subset will be performed by the second sub-thread to obtain the data import result of the split data subset. After obtaining the data import results of each of the multiple split data subsets, the first sub-thread will summarize the data import results of each of the multiple split data subsets to obtain the data import result of the target data subset.

[0100] In a specific application, when the second sub-thread executes the data import of the split data subset, the second sub-thread will determine whether the data volume of the split data subset meets the direct import condition. If the data volume of the split data subset meets the direct import condition, the second sub-thread can directly import the data of the split data subset. If the data volume of the split data subset does not meet the direct import condition, that is, it meets the split import condition, the second sub-thread needs to split the split data subset first, and then import the three split data subsets obtained after the split.

[0101] In a specific application, if the data volume of the split data subset meets the splitting and importing conditions, the second sub-thread, after splitting the split data subset to obtain multiple three-time split data subsets, will create a third sub-thread corresponding to each three-time split data subset for each three-time split data subset, and use the third sub-thread to execute data import of the three-time split data subset to obtain the data import results of the three-time split data subset, and then summarize the data import results of the multiple three-time split data subsets to obtain the data import results of the split data subset.

[0102] It is understandable that, similarly, in the third sub-thread, it is also necessary to first determine whether the data volume of the three-split data subsets meets the direct import condition or the split import condition, and then import the three-split data subsets according to the satisfied import condition. That is, the first splitting of the data subsets and then creating sub-threads for processing in this application is executed recursively. If the data volume of the data set meets the direct import condition, the data set can be directly imported in the corresponding sub-thread. If the data volume of the data set meets the split import condition, the data set needs to be split in the corresponding sub-thread and then a sub-thread needs to be created for import.

[0103] Recursion is a method in computer science to solve a problem by repeatedly decomposing the problem into similar sub-problems. In this embodiment, the method solves the problem of importing the target data set by repeatedly decomposing the import of the target data set into the import of similar sub-sets obtained by each split.

[0104] In this embodiment, when the data amount of the target data subset meets the splitting and importing conditions, the target data subset is split by the first sub-thread to obtain multiple split data subsets, and the data amount of a single data import can be controlled by splitting. Then, multiple second sub-threads can be created, and multiple second sub-threads can be used to import multiple split data subsets respectively. After obtaining the data import results of the split data subsets, the data import results of the target data subset are obtained based on the data import results of each of the multiple split data subsets. Multiple second sub-threads can be used to realize parallel processing of multiple split data subsets, thereby improving data import efficiency.

[0105] In an exemplary embodiment, the data import method further includes:

[0106] When the data volume of the target data set meets the direct import condition, the number of cores of the processor used to execute the data import is determined, and a data import thread is created according to the number of cores;

[0107] The data import thread is used to import data to the target data set, and the data import result of the target data set is obtained.

[0108] The direct import condition refers to the condition for directly importing data without splitting the data set when performing data import on any data set, which can be configured according to the actual application scenario. For example, the direct import condition can be that the thread import data volume determined based on the data volume of the data set is less than or equal to the preset thread data processing volume threshold. The thread import data volume can be the ratio of the data volume of the data set to the number of cores of the processor.

[0109] Among them, the processor is one of the core components of the computer device, that is, one of the core components of the server in this embodiment. It is responsible for executing instructions in the program, processing data, and controlling other components of the computer device, and can execute data import threads. For example, the processor can specifically refer to a central processing unit. The number of cores of the processor refers to the number of cores (i.e., independent processing units) inside the processor, and each core can independently execute program instructions, perform data processing and perform tasks. The increase in the number of cores can improve the computing power and multi-tasking processing capabilities of the processor, allowing more efficient execution of parallel computing and multi-threaded applications. For example, the common number of cores can be four, six, eight, etc. Exemplarily, the server will determine whether the data volume of the target data set meets the direct import condition. If the data volume of the target data set meets the direct import condition, the number of cores of the processor used to execute the data import is determined, and a data import thread is created according to the number of cores. The data import thread executes data import on the target data set to obtain the data import result of the target data set.

[0110] In specific applications, when judging whether the data volume of the target data set meets the direct import conditions, the server will obtain the core number of the processor used to execute the data import, and determine the thread import data volume based on the data volume and core number of the target data set. If the thread import data volume is less than or equal to the preset thread data processing volume threshold, it is determined that the data volume of the target data set meets the direct import conditions.

[0111] In a specific application, creating data import threads according to the number of cores means creating data import threads with a number of threads equal to the number of cores, so as to utilize the cores of the processor to execute the data import threads, and to perform data import on the target data set through the data import threads to obtain the data import result of the target data set. Each core of the processor can independently execute one thread.

[0112] In this embodiment, when the data volume of the target data set meets the direct import conditions, by utilizing the number of cores of the processor to create data import threads, it is possible to create data import threads with the same number of cores, and then the cores can be utilized to execute the data import threads, so that parallel data import can be performed on the target data set through the data import threads, which can fully utilize the cores of the processor to work simultaneously and improve data import efficiency.

[0113] In an exemplary embodiment, as shown in FIG5 , the method of determining whether the data volume of the target data set meets the split import condition includes steps 502 to 506. Among them:

[0114] Step 502, obtaining the number of cores of the processor used to execute data import;

[0115] Step 504, determining the amount of data to be imported by the thread based on the amount of data in the target data set and the number of cores;

[0116] Step 506: When the thread import data volume is greater than the preset thread data processing volume threshold, it is determined that the data volume of the target data set meets the split import condition.

[0117] The thread import data volume refers to the estimated amount of data that a single thread needs to process when executing data import. The thread data processing volume threshold refers to the maximum amount of data that a single thread can efficiently process when processing data, and can be configured according to the actual application scenario.

[0118] Exemplarily, when judging whether the data volume of the target data set meets the split import condition, the server will obtain the number of cores of the processor used to execute the data import, calculate the ratio of the data volume of the target data set to the number of cores, use the ratio as the thread import data volume, compare the thread import data volume with the preset thread data processing volume threshold, and when the thread import data volume is greater than the preset thread data processing volume threshold, it means that the amount of data to be processed exceeds the maximum amount of data that a single thread can efficiently process when processing data, and the target data set needs to be split and then processed separately, that is, the data volume of the target data set meets the split import condition. In a specific application, the calculation formula for the thread import data volume T can be: T=m / n, where m is the data volume of the target data set and n is the number of cores of the processor.

[0119] In this embodiment, based on the number of cores of the processor used to execute data import, the data volume and core number of the target data set can be used to determine the amount of data imported by the thread, and the amount of data processed by the thread can be estimated. Furthermore, by comparing the amount of data imported by the thread with the preset thread data processing volume threshold, it can be determined whether the target data set needs to be further split, thereby accurately determining the import conditions satisfied by the target data set.

[0120] In an exemplary embodiment, the data set to be imported is divided to obtain business data sets of various business types, including:

[0121] Identify data identifiers of respective ones of the plurality of data to be imported in the data set to be imported;

[0122] Based on the data identifiers of the multiple data to be imported and the corresponding relationship between the data identifiers and the business types, the multiple data to be imported are divided to obtain business data sets of the multiple business types.

[0123] The data identifier refers to a unique identifier of the data to be imported, which can facilitate the identification, location and management of the data to be imported. For example, the data to be imported can be a data file, and the data identifier can specifically refer to the file identifier of the data file, such as the file name of the data file, the file format suffix, etc. The file format suffix is ​​also called the file extension, which refers to a string of characters in the file name used to indicate the file format and purpose.

[0124] Among them, the correspondence between the data identifier and the business type refers to the mapping relationship between the data identifier and the business type. The data identifiers of the business data of different business types are different. By using this correspondence, the business data belonging to each business type can be distinguished by the data identifier. In this embodiment, the correspondence between the data identifier and the business type can be configured according to the actual application scenario. For example, the data identifier can be a file format suffix, and the correspondence between the file format suffix and the business type can be established to distinguish the data files of each business type.

[0125] Exemplarily, after obtaining the data set to be imported, the server will identify the data identifiers of each of the multiple data to be imported in the data set to be imported, and use the data identifiers of each of the multiple data to be imported to respectively query the correspondence between the data identifiers and the business types, determine the business types to which the multiple data to be imported belong, and use the business types to which the multiple data to be imported belong to divide the multiple data to be imported to obtain business data sets of each of the multiple business types.

[0126] In this embodiment, by identifying the data identifiers of each of the multiple data to be imported in the data set to be imported, the data identifiers of each of the multiple data to be imported and the correspondence between the data identifiers and the business types can be utilized to achieve division of the multiple data to be imported, and obtain business data sets of each of the multiple business types, thereby facilitating the import of the business data sets of each of the multiple business types and improving the data import efficiency.

[0127] In an exemplary embodiment, as shown in FIG6 , a data import method of the present application is described by a flowchart, and the data import method includes steps 602 to 624. Among them:

[0128] Step 602: Acquire a data set to be imported, and identify data identifiers of respective data to be imported in the data set to be imported.

[0129] When receiving a data import request, the server will obtain the to-be-imported data set corresponding to the data import request, and identify the data identifiers of the multiple to-be-imported data in the to-be-imported data set.

[0130] Step 604: based on the data identifiers, the multiple data to be imported are divided to obtain business data sets of the respective multiple business types.

[0131] The server divides the multiple data to be imported based on their respective data identifiers and the corresponding relationship between the data identifiers and the business types, and obtains business data sets of the multiple business types.

[0132] Step 606: Determine the import priority, and determine the target data set to perform data import according to the import priority.

[0133] Among them, the server will determine the import priority of the business data sets of various business types based on the data dependency relationship between the business data of various business types, and according to the import priority, perform data import on the business data sets of various business types in turn as the target data sets.

[0134] Step 608: For each data import, obtain the number of cores of the processor used to execute the data import.

[0135] Step 610: determine the thread import data volume based on the data volume of the target data set and the number of cores, and determine the import condition satisfied by the data volume of the target data set based on the data import data volume.

[0136] Step 612: If the direct import condition is met, determine the number of cores of the processor used to execute the data import, and create a data import thread according to the number of cores.

[0137] When the amount of thread imported data is less than or equal to a preset thread data processing amount threshold, it is determined that the amount of data in the target data set meets the direct import condition.

[0138] Step 614: Import data to the target data set through a data import thread to obtain a data import result of the target data set.

[0139] Step 616: If the split import condition is met, the target data set is split to obtain multiple target data subsets.

[0140] When the amount of thread imported data is greater than a preset thread data processing amount threshold, it is determined that the amount of data in the target data set meets the split import condition.

[0141] Step 618: for each target data subset, create a first sub-thread, and when the data volume of the target data subset meets the splitting and importing condition, split the target data subset through the first sub-thread to obtain multiple split data subsets.

[0142] Step 620: for each split data subset, create a second sub-thread, and use the second sub-thread to perform data import on the split data subset to obtain the data import result of the split data subset.

[0143] Step 622, based on the data import results of each of the multiple split data subsets, obtain the data import result of the target data subset.

[0144] Step 624, based on the data import results of each of the multiple target data subsets, obtain the data import result of the target data set.

[0145] In an exemplary embodiment, the data import method of the present application, the processing logic of each data import is as shown in FIG7:

[0146] Step 702, when executing data import, first determine whether the data volume of the data set meets the direct import conditions.

[0147] Step 704: If the data volume of the data set meets the direct import condition, directly import the data for the data set and collect the data import result.

[0148] If the thread import data volume determined based on the data volume of the data set and the number of processor cores is less than or equal to a preset thread data processing volume threshold, it is considered that the data volume of the data set meets the direct import condition.

[0149] Step 706: If the data volume of the data set meets the split import condition, the data set is split into multiple data subsets.

[0150] Step 708, creating an independent child thread, recursively executing steps 702 to 708 in the child thread, and collecting data import results after the child thread finishes executing.

[0151] If the amount of thread imported data is greater than a preset thread data processing amount threshold, it is considered that the amount of data in the target data set meets the split import condition.

[0152] Step 710, collect all data import results and complete the data import.

[0153] In an exemplary embodiment, taking the target data set with a data volume of 8 and a preset thread data processing volume threshold of 2 as an example, the schematic diagram of executing data import after recursive splitting can be shown in Figure 8. The target data set will be split first to obtain 2 target data subsets (the data volume is 4), and then for each target data subset, a first sub-thread corresponding to the target data subset will be created. The target data subset will be split through the first sub-thread to obtain 2 split data subsets. For each split data subset, a second sub-thread corresponding to the split data subset will be created. The data import of the split data subset will be executed through the second sub-thread, that is, the execution calculation task in Figure 8.

[0154] In an exemplary embodiment, the data import method of the present application can be applied to the business system of an organization that provides a variety of business services. When a new business system goes online, it is necessary to initially import business data to support the development of the business. In the process of initially importing business data, the time window for the entire import operation is usually extremely limited, and the format of the data file before import is significantly different from the business format of the new business, requiring transformation calculations. In addition, the methods of transformation calculations for business data of different business types are different, and the use of traditional methods has the problem of low data import efficiency.

[0155] Based on this, the present application provides a data import method. When a data import request is received, the data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided. A plurality of business types can be used to classify the data to be imported, and business data sets of the plurality of business types are obtained. Thus, the business data sets of the plurality of business types can be processed respectively, thereby speeding up the transformation calculation. The import priority of the business data sets of the plurality of business types is determined based on the data dependency relationship between the business data of the plurality of business types. The data dependency relationship can be used to determine the import priority, so that the business data sets of the plurality of business types can be used as the target data sets in turn to execute data import according to the import priority, thereby avoiding errors during data import. In order to improve the efficiency of data import, the data import process needs to be paused for lack of dependent data and then continue to execute data import after importing the business data sets of other businesses. For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets. The target data set can be quickly grouped by splitting, thereby realizing the control of the data volume of a single data import. Then, the data import result of the target data set can be obtained by executing data import on multiple target data subsets respectively, realizing parallel data import based on the control of data volume, avoiding the situation that the data volume during data import is too large, causing memory overflow and thus causing data import failure, and making full use of parallel computing capabilities, which can improve the efficiency of data import. That is, the entire data import process, by controlling the execution data volume of a single thread computing task, can achieve the satisfaction of using a multi-core computer to realize parallel computing tasks, and can control the memory usage constraints within a controllable range while computing the computing tasks.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a data import device for implementing the data import method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data import device embodiments provided below can refer to the limitations on the data import method above, and will not be repeated here.

[0158] In an exemplary embodiment, as shown in FIG. 9 , a data import device is provided, including: a data partitioning module 902, a sorting module 904, a splitting module 906 and an importing module 908, wherein:

[0159] The data partitioning module 902 is used to, when receiving a data import request, obtain a set of data to be imported corresponding to the data import request, and partition the set of data to be imported to obtain business data sets of various business types;

[0160] The sorting module 904 is used to determine the import priority of the business data sets of the multiple business types based on the data dependency relationship between the business data of the multiple business types, and to import the business data sets of the multiple business types as target data sets in sequence according to the import priority;

[0161] A splitting module 906 is used for splitting the target data set to obtain multiple target data subsets when the data volume of the target data set meets the splitting and importing conditions for each data import;

[0162] The import module 908 is used to perform data import on multiple target data subsets respectively to obtain data import results of the target data set.

[0163] The above-mentioned data import device, when receiving a data import request, obtains the to-be-imported data set corresponding to the data import request, divides the to-be-imported data set, can use multiple business types to classify the to-be-imported data, obtain the business data sets of the multiple business types, and determine the import priority of the business data sets of the multiple business types based on the data dependency relationship between the business data of the multiple business types. The data dependency relationship can be used to determine the import priority, so that the business data sets of the multiple business types can be used as target data sets in turn to execute data import according to the import priority, so as to avoid the need to pause and pause the data import due to the lack of dependent data during the data import. After the service data set is imported, continuing to import data can improve the data import efficiency. For each data import, when the data volume of the target data set meets the split import conditions, the target data set is split to obtain multiple target data subsets. The target data set can be quickly grouped by splitting, thereby realizing the control of the data volume of a single data import. Then, data import can be performed on multiple target data subsets respectively to obtain the data import result of the target data set, realizing parallel data import based on controlling the data volume, avoiding the situation where the memory overflow caused by excessive data volume during data import and then the data import failure is caused, and making full use of parallel computing capabilities can improve the data import efficiency.

[0164] In an exemplary embodiment, the import module includes:

[0165] The sub-thread unit is used to create a first sub-thread corresponding to each target data subset, and execute data import of the target data subset through the first sub-thread to obtain the data import result of the target data subset;

[0166] The aggregation unit is used to obtain the data import result of the target data set based on the data import results of each of the multiple target data subsets.

[0167] In an exemplary embodiment, the sub-thread unit is specifically used for:

[0168] When the data volume of the target data subset meets the splitting and importing conditions, the target data subset is split by the first sub-thread to obtain multiple split data subsets;

[0169] For each split data subset, create a second sub-thread corresponding to the split data subset, and use the second sub-thread to perform data import on the split data subset to obtain a data import result for the split data subset;

[0170] Based on the data import results of each of the multiple split data subsets, the data import result of the target data subset is obtained.

[0171] In an exemplary embodiment, the import module is further used to:

[0172] When the data volume of the target data set meets the direct import condition, data import is performed on the target data set through the main thread to obtain the data import result of the target data set.

[0173] In an exemplary embodiment, the device also includes a judgment module, which is used to obtain the number of cores of a processor used to execute data import, determine the thread import data volume based on the data volume and core number of the target data set, and when the thread import data volume is greater than a preset thread data processing volume threshold, determine that the data volume of the target data set meets the split import condition.

[0174] In an exemplary embodiment, the data partitioning module includes:

[0175] An identification unit, used to identify data identifications of respective data of a plurality of data to be imported in the data set to be imported;

[0176] The division unit is used to divide the multiple data to be imported based on the data identifiers of the multiple data to be imported and the corresponding relationship between the data identifiers and the business types to obtain business data sets of multiple business types.

[0177] Each module in the above data import device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0178] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal. Taking the computer device as a server as an example, its internal structure diagram can be shown in Figure 10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as data import results of a target data set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data import method is implemented.

[0179] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0182] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0183] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data import method, characterized in that: The method comprises: When a data import request is received, a data set to be imported corresponding to the data import request is obtained, and the data set to be imported is divided to obtain business data sets of respective multiple business types; Determining the import priorities of the business data sets of the multiple business types based on the data dependency relationships between the business data of the multiple business types, and importing the business data sets of the multiple business types in sequence as target data sets according to the import priorities; For each data import, when the data volume of the target data set meets the split import condition, the target data set is split to obtain multiple target data subsets; Data import is performed on the multiple target data subsets respectively to obtain data import results of the target data set.

2. The method according to claim 1, characterized in that The step of respectively importing data on the plurality of target data subsets to obtain a data import result for the target data set comprises: For each of the target data subsets, create a first sub-thread corresponding to the target data subset, and use the first sub-thread to perform data import on the target data subset to obtain a data import result of the target data subset; Based on the data import results of each of the multiple target data subsets, the data import result of the target data set is obtained.

3. The method according to claim 2, characterized in that The importing of data of the target data subset by the first sub-thread to obtain the data import result of the target data subset includes: When the data amount of the target data subset meets the splitting and importing condition, the target data subset is split by the first sub-thread to obtain a plurality of split data subsets; For each of the split data subsets, create a second sub-thread corresponding to the split data subset, and use the second sub-thread to perform data import on the split data subset to obtain a data import result for the split data subset; Based on the data import results of each of the multiple split data subsets, the data import result of the target data subset is obtained.

4. The method according to claim 1, characterized in that: The method further comprises: When the data volume of the target data set meets the direct import condition, determining the number of cores of the processor used to execute the data import, and creating a data import thread according to the number of cores; The data import thread executes data import on the target data set to obtain a data import result of the target data set.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Get the number of cores of the processor used to perform data import; Determine the amount of data to be imported by the thread based on the amount of data in the target data set and the number of cores; When the thread import data volume is greater than a preset thread data processing volume threshold, it is determined that the data volume of the target data set meets the split import condition.

6. The method according to any one of claims 1 to 4, characterized in that: The data set to be imported is divided to obtain business data sets of various business types, including: Identifying respective data identifiers of a plurality of to-be-imported data in the to-be-imported data set; Based on the data identifiers of the multiple data to be imported and the corresponding relationship between the data identifiers and the business types, the multiple data to be imported are divided to obtain business data sets of the multiple business types.

7. A data import device, characterized in that: The device comprises: A data partitioning module is used to, when receiving a data import request, obtain a set of data to be imported corresponding to the data import request, and partition the set of data to be imported to obtain business data sets of various business types; A sorting module, configured to determine the import priority of the business data sets of the multiple business types based on the data dependency relationship between the business data of the multiple business types, and to import the business data sets of the multiple business types in sequence as target data sets according to the import priority; A splitting module is used for splitting the target data set to obtain multiple target data subsets for each data import when the data volume of the target data set meets the splitting and importing conditions; The import module is used to perform data import on the multiple target data subsets respectively to obtain the data import result of the target data set.

8. The device according to claim 7, characterized in that The import module includes: A sub-thread unit is used to create, for each of the target data subsets, a first sub-thread corresponding to the target data subset, and execute data import of the target data subset through the first sub-thread to obtain a data import result of the target data subset; The summarizing unit is used to obtain the data import result of the target data set based on the data import results of each of the multiple target data subsets.

9. The device according to claim 8, characterized in that The sub-thread unit is specifically used for: When the data amount of the target data subset meets the splitting and importing condition, the target data subset is split by the first sub-thread to obtain a plurality of split data subsets; For each of the split data subsets, create a second sub-thread corresponding to the split data subset, and use the second sub-thread to perform data import on the split data subset to obtain a data import result for the split data subset; Based on the data import results of each of the multiple split data subsets, the data import result of the target data subset is obtained.

10. The device according to claim 7, characterized in that The import module is also used to: When the data volume of the target data set meets the direct import condition, data import is performed on the target data set to obtain a data import result of the target data set.

11. The device according to any one of claims 7 to 10, characterized in that The device also includes a judgment module, which is used to obtain the number of cores of the processor used to execute data import, and determine the thread import data volume based on the data volume of the target data set and the number of cores. When the thread import data volume is greater than a preset thread data processing volume threshold, it is determined that the data volume of the target data set meets the split import condition.

12. The device according to any one of claims 7 to 10, characterized in that The data partitioning module comprises: An identification recognition unit, used to identify the data identification of each of the plurality of to-be-imported data in the to-be-imported data set; The division unit is used to divide the multiple data to be imported based on the data identifiers of the multiple data to be imported and the corresponding relationship between the data identifiers and the business types to obtain business data sets of multiple business types.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images