A data export method, system and device

Abstract

The application relates to the field of data processing and discloses a data export method, system and device, wherein the method comprises the following steps: in response to a data export request of a source data table, extracting all export data groups in the data export request and an export requirement identifier corresponding to each export data group; obtaining an export template identifier of each export template in a preset export template library; based on the export requirement identifier and the template identifier, screening a target export template of each export data group; based on the export data group, constructing a query statement; using all the query statements to perform concurrent query on the source data table to obtain export data of each export data group; and filling the export data of the export data group into the target export template of the export data group to complete data export of the source data table. The application mainly aims to improve the data export efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to a data export method, system and device, and belongs to the field of data processing. BACKGROUND

[0002] With the increasing number of samples and types of immunological detection projects in laboratories, the amount of stored sample detection data is also increasing, and there is a lack of corresponding data processing systems. Therefore, in order to understand the detection data list of different sample quantities, the sample detection data of the corresponding sample in the stored sample detection data needs to be exported.

[0003] The data export method needs to query the detection data of each item to be exported in the stored sample detection data, and then fill in the corresponding list template, so as to complete the data export and obtain the data list of different items. However, the existing data export method needs to query a large number of items, and needs to use a single thread to query the detection data of each item in turn, thereby reducing the efficiency of data export. SUMMARY

[0004] The present application provides a data export method, system and device, which mainly aims to improve the efficiency of data export.

[0005] To achieve the above purpose, the present application provides a data export method, which comprises:

[0006] In response to a data export request of a source data table, all export data groups in the data export request and the export requirements corresponding to each export data group are extracted;

[0007] The list template identifier of each list template in the preset list template library is obtained, and the template identifier is matched and screened based on the export requirements, so as to screen the list template corresponding to each export data group and obtain the target list template of each export data group;

[0008] The number of export data groups is calculated to obtain the number of export tasks, and a query statement is constructed based on the export data groups;

[0009] The number of idle threads in the target thread pool is obtained, and it is judged whether the number of idle threads is greater than or equal to the number of export tasks;

[0010] When the number of idle threads is greater than or equal to the number of export tasks, the idle threads of the number of export tasks are selected in the target thread pool to obtain data query threads;

[0011] When the number of idle threads is less than the number of export tasks, new threads are created in the target thread pool based on the number of export tasks, and idle threads of the number of export tasks are selected in the target thread pool to obtain data query threads;

[0012] All the data query threads are started to perform concurrent queries on the source data table by all the query statements, and export data of each export data group is obtained;

[0013] The export data of the export data group is filled into the target manifest template of the export data group to complete the data export of the source data table.

[0014] Optionally, based on the export requirement, the template identifier is matched and screened to screen the manifest template corresponding to each export data group, and the target manifest template of each export data group is obtained, comprising:

[0015] The export requirement is converted into a vector to obtain an export requirement vector;

[0016] The template identifier is converted into a vector to obtain a template identifier vector;

[0017] Based on the template identifier vector and the export requirement vector, a matching coefficient of each template identifier corresponding to the export requirement is obtained;

[0018] Based on the matching coefficient, all the template identifiers are screened to obtain a target template identifier;

[0019] The export target corresponding to the target template identifier is determined as the target manifest template of the export data group corresponding to the export requirement.

[0020] Optionally, the export requirement is converted into a vector to obtain an export requirement vector, comprising:

[0021] The export requirement is segmented, and the segmented word is determined as a first segmented word;

[0022] Each first segmented word is converted into a vector to obtain a first segmented word vector;

[0023] All the first segmented word vectors are used for vector calculation to obtain a first requirement feature vector;

[0024] The export requirement is converted by language translation to obtain a translated export requirement;

[0025] The translated export requirement is segmented, and the segmented word is determined as a second segmented word;

[0026] convert the second segmented word into a vector to obtain a second segmented word vector;

[0027] perform vector calculation on all the second segmented word vectors to obtain a second requirement feature vector;

[0028] horizontally splice the first requirement feature vector and the second requirement feature vector to obtain a requirement feature matrix;

[0029] perform dimension reduction on the requirement feature matrix to obtain the derived requirement vector.

[0030] Optionally, the performing vector calculation on all the first segmented word vectors to obtain a first requirement feature vector comprises:

[0031]

[0032] wherein y is the first requirement feature vector, m is the total number of the first segmented word vectors, j is the serial number of the first segmented word vector, x j is the first segmented word vector with the serial number j.

[0033] Optionally, the performing calculation based on the template identification vector and the derived requirement vector to obtain a matching coefficient of each template identification corresponding to the derived requirement comprises:

[0034]

[0035] wherein a i is an element with the serial number i in the derived requirement vector a, b i is an element with the serial number i in the template identification vector b, n is the vector dimension of the derived requirement vector and the template identification vector, i is the serial number of the element in the derived requirement vector and the template identification vector, the derived requirement vector a is a derived requirement vector corresponding to a derived requirement A, the template identification vector b is a template identification vector corresponding to a template identification B, and p is a matching coefficient of the template identification B corresponding to the derived requirement A.

[0036] Optionally, the selecting the number of idle threads in the target thread pool to obtain a data query thread comprises:

[0037] obtaining all idle threads in the target thread pool and the creation time of each idle thread;

[0038] sorting all idle threads in the target thread pool based on the creation time to obtain an idle thread sequence;

[0039] The preset rank in the idle thread program sequence and an idle thread before the preset rank are determined as the data query thread, and a value of the preset rank is the same as a value of the export task quantity.

[0040] Optionally, the creating of the new thread in the target thread pool based on the export task quantity comprises:

[0041] The quantity of the created threads of the target thread pool is acquired to obtain a created thread quantity;

[0042] The maximum thread quantity of the target thread pool is acquired;

[0043] The difference between the maximum thread quantity and the created thread quantity is calculated to obtain a theoretical thread creation quantity;

[0044] The quantity of the idle threads in the target thread pool is acquired to obtain an idle thread quantity;

[0045] The difference between the export task quantity and the idle thread quantity is calculated to obtain an actual thread creation quantity;

[0046] It is judged whether the theoretical thread creation quantity is greater than or equal to the actual thread creation quantity;

[0047] When the theoretical thread creation quantity is less than the actual thread creation quantity, the target thread pool is updated in maximum thread quantity based on the theoretical thread creation quantity and the actual thread creation quantity, and the new thread is created in the target thread pool, and the quantity of the created new thread is greater than or equal to the actual thread quantity;

[0048] When the theoretical thread creation quantity is greater than or equal to the actual thread creation quantity, the new thread is created in the target thread pool, and the quantity of the created new thread is greater than or equal to the actual thread quantity.

[0049] Optionally, the updating of the target thread pool in maximum thread quantity comprises:

[0050] The difference between the actual thread creation quantity and the theoretical thread creation quantity is calculated to obtain an adjustment difference value;

[0051] The sum of the maximum thread quantity and the adjustment difference value is calculated to obtain a target adjustment maximum thread quantity;

[0052] The target thread pool is adjusted in maximum thread quantity so that the adjusted maximum thread quantity is greater than or equal to the target adjustment maximum thread quantity.

[0053] Optionally, the starting all the data query threads to perform concurrent queries on the source data table by all the query statements to obtain the export data of each export data group comprises:

[0054] Traversing all the data query threads and allocating one query statement to each data query thread;

[0055] Starting all the data query threads, and executing the query statement allocated to each data query thread by the started data query thread to query the source data table, and saving the query result corresponding to each query statement;

[0056] Determining the saved query result corresponding to each query statement as the export data of the export data group corresponding to the query statement.

[0057] To solve the above problems, the present application further provides a data export system, the system comprises:

[0058] A template matching module is configured to extract all export data groups and export requirements corresponding to each export data group in a data export request of a source data table in response to the data export request, obtain a list template identifier of each list template in a preset list template library, and match and filter the template identifiers based on the export requirements to filter a list template corresponding to each export data group to obtain a target list template of each export data group.

[0059] An export data query module is configured to calculate the number of export data groups to obtain an export task number, construct a query statement based on the export data groups, obtain the number of idle threads in a target thread pool, determine whether the number of idle threads is greater than or equal to the export task number, select idle threads of the export task number in the target thread pool to obtain data query threads when the number of idle threads is greater than or equal to the export task number, create new threads in the target thread pool based on the export task number when the number of idle threads is less than the export task number, select idle threads of the export task number in the target thread pool to obtain data query threads, and start all the data query threads to perform concurrent queries on the source data table by all the query statements to obtain the export data of each export data group.

[0060] A data export module is configured to fill the export data of the export data group into the target list template of the export data group to complete the data export of the source data table.

[0061] To solve the above problems, the present application further provides an electronic device, the electronic device comprises:

[0062] at least one processor; and

[0063] a memory in communication with the at least one processor; wherein

[0064] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the data export method described above.

[0065] To solve the above problems, the present application also provides a computer readable storage medium, the computer readable storage medium stores at least one instruction, the at least one instruction is executed by the processor in the electronic device to implement the data export method described above.

[0066] Compared with the problems described in the background art, the embodiment of the application first calculates the number of the export data sets to obtain the number of export tasks, and constructs a query statement based on the export data sets; obtains the number of idle threads in the target thread pool, and judges whether the number of idle threads is greater than or equal to the number of export tasks; when the number of idle threads is greater than or equal to the number of export tasks, select the number of idle threads of the export tasks in the target thread pool to obtain data query threads;

[0067] When the number of idle threads is less than the number of export tasks, create new threads in the target thread pool based on the number of export tasks, and select the number of idle threads of the export tasks in the target thread pool to obtain data query threads; start all the data query threads to execute concurrent queries of all the query statements on the source data table to obtain the export data of each export data set; fill the export data of the export data set into the target manifest template of the export data set to complete the data export of the source data table. The embodiment of the application dynamically matches the execution of all query statements by filtering or creating threads according to the number of data categories to be queried, so that all query statements can be executed simultaneously, thereby realizing concurrent query. Compared with the background technology in which only single thread can be executed sequentially to complete data query of each query statement, the query speed is faster, the data query speed is improved, and the queried data can be exported faster, thereby improving the efficiency of data export. Therefore, the data export method, system, electronic device and computer readable storage medium provided by the present application mainly aim to improve the efficiency of data export. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The flowchart of the data export method provided by an embodiment of the present application is shown;

[0069] Figure 2This is a schematic diagram of the process of obtaining exported data in a data export method provided in an embodiment of the present invention;

[0070] Figure 3 This is a functional block diagram of a data export system provided in an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data export method according to an embodiment of the present invention.

[0072] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0073] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0074] This application provides a data export method. The execution subject of the data export method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices that can be configured to execute the method provided in this application. In other words, the data export method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster.

[0075] Example 1:

[0076] Reference Figure 1 The diagram shown is a flowchart illustrating a data export method according to an embodiment of the present invention. In this embodiment, the data export method includes:

[0077] S1. In response to the data export request of the source data table, extract all export data groups in the data export request and the export requirements corresponding to each export data group;

[0078] In this embodiment of the invention, the source data table is a table from which data needs to be exported. It can be an Excel spreadsheet or a table in a database, and this embodiment of the invention does not impose any limitations on this. The data export request is a request to export the data in the source data table into multiple table lists, categorized by type. The exported data group is the data requirements for export (such as exported data fields, etc.). Based on the conditions corresponding to each exported data group, the data in the source data table that meets the conditions is exported, resulting in a corresponding table list. The export requirements are the type requirements for exporting the data in the source data table that meets the conditions according to the conditions corresponding to the exported data group, resulting in the table list type requirements.

[0079] S2, obtain a list template identifier of each list template in a preset list template library, and perform matching screening on the template identifier based on the export requirement to screen a list template corresponding to each of the export data groups, so as to obtain a target list template of each of the export data groups;

[0080] In the embodiment of the present application, the data type of the table list obtained by the export data is determined by the export data template written by the export data, so it is necessary to determine the list template matched with the export requirement. Therefore, the list template corresponding to each of the export data groups is screened based on the export requirement and the template identifier, so as to obtain the target list template of each of the export data groups. The list template identifier is information describing the type of the list template, and the list template is a preset blank table list template. The list template library contains at least one list template and the template identifier of each list template.

[0081] In detail, in the embodiment of the present application, the matching screening on the template identifier based on the export requirement to screen a list template corresponding to each of the export data groups, so as to obtain a target list template of each of the export data groups, includes:

[0082] Converting the export requirement into a vector to obtain an export requirement vector;

[0083] Converting the template identifier into a vector to obtain a template identifier vector;

[0084] Based on the template identifier vector and the export requirement vector, a matching coefficient of each of the template identifiers corresponding to the export requirement is obtained;

[0085] Based on the matching coefficient, all of the template identifiers are screened to obtain a target template identifier;

[0086] The export target corresponding to the target template identifier is determined as the target list template of the export data group corresponding to the export requirement.

[0087] Further, in the embodiment of the present application, the converting the export requirement into a vector to obtain an export requirement vector includes:

[0088] The export requirement is segmented, and the segmented word is determined as a first segmented word;

[0089] Each of the first segmented words is converted into a vector to obtain a first segmented word vector;

[0090] All of the first segmented word vectors are used for vector calculation to obtain a first requirement feature vector;

[0091] The export requirement is converted by language translation to obtain a translated export requirement;

[0092] The translation export requirement is segmented, and the segmented words are determined as second segmented words;

[0093] The second segmented words are converted into vectors to obtain a second segmented word vector;

[0094] All the second segmented word vectors are used for vector calculation to obtain a second requirement feature vector;

[0095] The first requirement feature vector and the second requirement feature vector are transversely spliced to obtain a requirement feature matrix;

[0096] The requirement feature matrix is reduced in dimension to obtain the export requirement vector.

[0097] It should be explained that in the embodiment of the application, the segmented words can be converted into vectors by using a bag-of-words model, a BERT model, a one-hot algorithm or other models or algorithms, and the embodiment of the application does not limit this. Further, in order to more accurately represent the semantic features of the export requirement, the export requirement is converted by language translation to translate the export requirement into a text in another language (for example, the export requirement is a Chinese text, and then the export can be translated into English), and then the feature vectors of the two languages are combined, thereby improving the accuracy of the export requirement vector in representing the semantic features of the export requirement. The dimension of the export requirement vector is the same as that of the template identification vector.

[0098] Specifically, in the embodiment of the application, the use of all the first segmented word vectors for vector calculation to obtain a first requirement feature vector comprises:

[0099]

[0100] wherein y is the first requirement feature vector, m is the total number of the first segmented word vectors, j is the serial number of the first segmented word vector, and x j is the first segmented word vector with serial number j.

[0101] In detail, in the embodiment of the application, the calculation based on the template identification vector and the export requirement vector to obtain the matching coefficient of each template identification corresponding to the export requirement comprises:

[0102]

[0103] wherein a i is the element with serial number i in the export requirement vector a, b i, n is the vector dimension of the derivation requirement vector and the template identification vector, i is the serial number of the element in the derivation requirement vector and the template identification vector, the derivation requirement vector a is a derivation requirement vector corresponding to derivation requirement A, the template identification vector b is a template identification vector corresponding to template identification B, and p is a matching coefficient of template identification B corresponding to derivation requirement A.

[0104] In another embodiment of the present application, the vector calculation is performed using all the first segmented word vectors to obtain a first requirement feature vector, which includes:

[0105] All the first segmented word vectors are combined horizontally according to the order of the first segmented word vectors corresponding to the first segmented word vectors in the derivation requirement to obtain a first requirement feature matrix.

[0106] The first requirement matrix is input into a pre-constructed feature extraction model, wherein the first feature extraction network is a convolutional neural network.

[0107] The output value of each output node in the last full connection layer of the feature extraction model is extracted.

[0108] The output values are combined into a vector form according to the order of the output nodes corresponding to each output value in the last full connection layer to obtain the first requirement feature vector.

[0109] Further, the method of converting the template identification into a vector is similar to the method of converting the derivation requirement into a vector, and the present embodiment is not limited in this regard.

[0110] S3, the number of derivation data groups is calculated to obtain the number of derivation tasks, and a query statement is constructed based on the derivation data groups.

[0111] In the embodiment of the present application, the number of derivation data groups is calculated to obtain the number of derivation tasks, and a query statement is constructed based on the derivation data groups.

[0112] Optionally, the embodiment of the present application can construct a query statement according to the type of the source data table and the derivation data groups, for example, if the source data table is a data table in a SQL database, a select function and the derivation data groups can be used to construct a query statement for data query on the source data table.

[0113] S4, the number of idle threads in the target thread pool is obtained, and it is determined whether the number of idle threads is greater than or equal to the number of derivation tasks.

[0114] The target thread pool in the embodiment of the application is a preset multi-thread pool, such as a ThreadPoolExecutor thread pool in Java.

[0115] Further, in the embodiment of the application, in order to concurrently query the exported data corresponding to each query statement in the source data table by using the idle threads, it is necessary to determine whether the number of idle threads in the target thread pool meets the requirement of data export, and therefore, it is determined whether the number of idle threads is greater than or equal to the number of export tasks.

[0116] S5, when the number of idle threads is greater than or equal to the number of export tasks, idle threads in the target thread pool are selected to obtain data query threads;

[0117] In the embodiment of the application, in order to concurrently execute all query statements, idle threads in the target thread pool are selected to obtain data query threads.

[0118] In detail, in the embodiment of the application, idle threads in the target thread pool are selected to obtain data query threads, including:

[0119] All idle threads in the target thread pool and the creation time of each idle thread are obtained;

[0120] All idle threads in the target thread pool are sorted based on the creation time to obtain an idle thread sequence;

[0121] Idle threads with a preset ranking and before the preset ranking in the idle thread sequence are determined as the data query threads, wherein the value of the preset ranking is the same as the value of the number of export tasks.

[0122] S6, when the number of idle threads is less than the number of export tasks, new threads are created in the target thread pool based on the number of export tasks, and idle threads in the target thread pool are selected to obtain data query threads;

[0123] In the embodiment of the application, each idle thread can execute a query statement, and in order to enable all query statements to be concurrently executed, when the number of idle threads is less than the number of export tasks, new threads are created in the target thread pool based on the number of export tasks, and idle threads in the target thread pool are selected to obtain data query threads.

[0124] In detail, when the total number of threads created in the thread pool is greater than the maximum number of threads set in the thread pool, the thread creation fails in the embodiment of the present application. Therefore, the new thread is created in the target thread pool based on the number of exported tasks, which comprises:

[0125] Obtaining the number of threads created in the target thread pool to obtain the number of created threads;

[0126] Obtaining the maximum number of threads of the target thread pool;

[0127] Calculating the difference between the maximum number of threads and the number of created threads to obtain the theoretical thread creation number;

[0128] Obtaining the number of idle threads in the target thread pool to obtain the number of idle threads;

[0129] Calculating the difference between the number of exported tasks and the number of idle threads to obtain the actual thread creation number;

[0130] Determining whether the theoretical thread creation number is greater than or equal to the actual thread creation number;

[0131] When the theoretical thread creation number is less than the actual thread creation number, the target thread pool is updated in terms of the maximum number of threads based on the theoretical thread creation number and the actual thread creation number, and the new thread is created in the target thread pool, and the number of created new threads is greater than or equal to the actual thread number;

[0132] When the theoretical thread creation number is greater than or equal to the actual thread creation number, the new thread is created in the target thread pool, and the number of created new threads is greater than or equal to the actual thread number.

[0133] Further, the target thread pool is updated in terms of the maximum number of threads in the embodiment of the present application, which comprises:

[0134] Calculating the difference between the actual thread creation number and the theoretical thread creation number to obtain an adjustment difference;

[0135] Calculating the sum of the maximum number of threads and the adjustment difference to obtain a target adjustment maximum number of threads;

[0136] The maximum number of threads of the target thread pool is adjusted so that the adjusted maximum number of threads is greater than or equal to the target adjustment maximum number of threads.

[0137] S7, starting all the data query threads to perform concurrent query on the source data table by all the query statements to obtain the exported data of each exported data group;

[0138] In the embodiment of the present application, in order to query the data to be exported corresponding to each export data group in the source data table, all the data query threads are started to execute all the query statements to perform concurrent query on the source data table, and the export data of each export data group is obtained.

[0139] In detail, in the embodiment of the present application, the starting of all the data query threads to execute all the query statements to perform concurrent query on the source data table to obtain the export data of each export data group comprises:

[0140] S71, all data query threads are traversed, and each data query thread is allocated a query statement;

[0141] S72, all data query threads are started, each data query thread after being started is used to execute the query statement allocated to it to query the source data table, and the query result corresponding to each query statement is saved;

[0142] S73, the saved query result corresponding to each query statement is determined as the export data of the export data group corresponding to the query statement.

[0143] In the embodiment of the present application, the query statement allocated to each data query thread is unique.

[0144] S8, the export data of the export data group is filled into the target list template of the export data group to complete the data export of the source data table.

[0145] In the embodiment of the present application, the export data of the export data group is filled into the target list template of the export data group to complete the data export of the source data table.

[0146] Compared with the problems in the background art, the embodiment of the present application first calculates the number of export data groups to obtain the number of export tasks, and constructs a query statement based on the export data group; obtains the number of idle threads in the target thread pool, judges whether the number of idle threads is greater than or equal to the number of export tasks; when the number of idle threads is greater than or equal to the number of export tasks, the number of idle threads of the export tasks is selected in the target thread pool to obtain data query threads.

[0147] When the number of idle threads is less than the number of export tasks, new threads are created in the target thread pool based on the number of export tasks, and idle threads of the number of export tasks are selected in the target thread pool to obtain data query threads; all the data query threads are started to perform concurrent query on the source data table by all the query statements to obtain export data of each export data group; and the export data of the export data group is filled into the target list template of the export data group to complete data export of the source data table. According to the number of data categories to be queried, the embodiment of the application filters or creates corresponding threads to dynamically match the execution of all query statements, so that all query statements can be executed simultaneously, thereby realizing concurrent query. Compared with the background art in which each query statement is executed sequentially by single thread to complete data query, the query speed is faster, the data query speed is improved, and the data export speed is further improved, thereby improving the data export efficiency.

[0148] Embodiment 2

[0149] As Figure 3 shown is a functional module diagram of a data export system provided by an embodiment of the application.

[0150] The data export system 100 can be installed in an electronic device. According to the implemented functions, the data export system 100 can include a template matching module 101, an export data query module 102, and a data export module 103. The modules of the application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0151] The template matching module 101 is configured to extract all export data groups in a data export request and export requirements corresponding to each export data group in response to a data export request of a source data table; obtain a list template identifier of each list template in a preset list template library; and perform matching and filtering on the template identifier based on the export requirements to filter a list template corresponding to each export data group to obtain a target list template of each export data group.

[0152] The export data query module 102 is configured to calculate the number of the export data groups, obtain the number of export tasks, and construct query statements based on the export data groups; obtain the number of idle threads in a target thread pool, and determine whether the number of idle threads is greater than or equal to the number of export tasks; when the number of idle threads is greater than or equal to the number of export tasks, select the number of idle threads in the target thread pool, and obtain data query threads; when the number of idle threads is less than the number of export tasks, create new threads in the target thread pool based on the number of export tasks, and select the number of idle threads in the target thread pool, and obtain data query threads; start all the data query threads to perform concurrent queries on the source data table by using all the query statements, and obtain export data of each export data group.

[0153] The data export module 103 is configured to fill the export data of the export data group into a target manifest template of the export data group, so as to complete data export of the source data table.

[0154] In detail, the modules in the data export system 100 in the embodiment of the present application use the same technical means as the data export method in the above-mentioned Figure 1 , and can produce the same technical effects, which will not be described herein again.

[0155] Embodiment 3

[0156] As shown in Figure 4 , it is a structural schematic diagram of an electronic device for implementing a data export method according to an embodiment of the present application.

[0157] The electronic device 1 can include a processor 10, a memory 11, a bus 12, and a communication interface 13, and can further include a computer program, such as a data export program, stored in the memory 11 and executable on the processor 10.

[0158] The memory 11 includes at least one type of readable storage medium, such as a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 can be used to store application software and various data installed in the electronic device 1, such as the code of the data export program, etc., and can also be used to temporarily store data that has been output or will be output.

[0159] The processor 10 can be composed of an integrated circuit in some embodiments, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of a central processing unit (CPU), a microprocessor, a digital processing chip, a graphics processor, and various control chips, etc. The processor 10 is the control unit of the electronic device, which connects various components of the electronic device through various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules stored in the memory 11 (such as the data export program, etc.) and calling data stored in the memory 11.

[0160] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0161] Figure 4 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0162] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0163] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0164] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0165] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0166] The data export program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:

[0167] In response to a data export request from the source data table, extract all export data groups in the data export request and the export requirements corresponding to each export data group;

[0168] Obtain the list template identifier of each list template in the preset list template library, and perform matching and filtering on the template identifier based on the export requirements to filter the list template corresponding to each of the exported data groups, so as to obtain the target list template for each of the exported data groups.

[0169] Calculate the number of exported data groups to obtain the number of exported tasks, and construct a query statement based on the exported data groups;

[0170] obtaining the number of idle threads in the target thread pool, and determining whether the number of idle threads is greater than or equal to the number of exported tasks;

[0171] when the number of idle threads is greater than or equal to the number of exported tasks, selecting the number of idle threads in the target thread pool, and obtaining data query threads;

[0172] when the number of idle threads is less than the number of exported tasks, creating new threads in the target thread pool based on the number of exported tasks, and selecting the number of idle threads in the target thread pool, and obtaining data query threads;

[0173] starting all the data query threads to perform concurrent queries on the source data table by using all the query statements, and obtaining exported data of each exported data group;

[0174] filling the exported data of the exported data group into the target manifest template of the exported data group, so as to complete data export of the source data table.

[0175] Specifically, the specific implementation method of the processor 10 to the above instructions can refer to Figures 1 to 3 The description of related steps in the corresponding embodiments will not be repeated here.

[0176] Further, the modules / units integrated in the electronic device 1 can be stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

[0177] The application also provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0178] in response to a data export request of a source data table, extracting all exported data groups in the data export request and export requirements corresponding to each exported data group;

[0179] obtaining a manifest template identifier of each manifest template in a preset manifest template library, and performing matching and screening on the template identifier based on the export requirements, so as to screen a manifest template corresponding to each exported data group, and obtain a target manifest template of each exported data group.

[0180] The number of the derived data groups is calculated to obtain a derived task number, and a query statement is constructed based on the derived data groups;

[0181] The number of idle threads in a target thread pool is obtained, and it is determined whether the number of idle threads is greater than or equal to the derived task number;

[0182] When the number of idle threads is greater than or equal to the derived task number, a number of idle threads equal to the derived task number is selected from the target thread pool to obtain data query threads;

[0183] When the number of idle threads is less than the derived task number, new threads are created in the target thread pool based on the derived task number, and a number of idle threads equal to the derived task number is selected from the target thread pool to obtain data query threads;

[0184] All the data query threads are started to perform concurrent queries on the source data table by using all the query statements, and the derived data of each derived data group is obtained;

[0185] The derived data of the derived data group is filled into a target manifest template of the derived data group to complete data derivation of the source data table.

[0186] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0187] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0188] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A data export method, characterized by, The method comprises: in response to a data export request of a source data table, extracting all export data groups in the data export request and export requirements corresponding to each of the export data groups; obtaining the list template identifier of each list template in the preset list template library, and performing matching screening on the template identifier based on the export requirements to screen the list template corresponding to each of the export data groups and obtain the target list template of each of the export data groups; wherein the matching screening of the template identifier based on the export requirements to screen the list template corresponding to each of the export data groups and obtain the target list template of each of the export data groups comprises: transforming the export requirements into a vector to obtain an export requirement vector; transforming the template identifier into a vector to obtain a template identifier vector; based on the template identifier vector and the export requirement vector, calculating a matching coefficient of each of the template identifiers corresponding to the export requirements; based on the matching coefficient, screening all the template identifiers to obtain a target template identifier; determining the export target corresponding to the target template identifier as the target list template of the export array corresponding to the export requirements; wherein the transformation of the export requirements into a vector to obtain an export requirement vector comprises: performing word segmentation on the export requirements, and determining the segmented words as first segmented words; converting each of the first segmented words into a vector to obtain a first segmented word vector; performing vector calculation using all the first segmented word vectors to obtain a first requirement feature vector; performing language translation conversion on the export requirements to obtain a translated export requirement; performing word segmentation on the translated export requirement, and determining the segmented words as second segmented words; converting the second segmented words into a vector to obtain a second segmented word vector; performing vector calculation using all the second segmented word vectors to obtain a second requirement feature vector; performing horizontal splicing on the first requirement feature vector and the second requirement feature vector to obtain a requirement feature matrix; performing dimension reduction on the requirement feature matrix to obtain the export requirement vector; calculating the number of export data groups to obtain the number of export tasks, and constructing a query statement based on the export data groups; obtaining the number of idle threads in a target thread pool, and determining whether the number of idle threads is greater than or equal to the number of export tasks; when the number of idle threads is greater than or equal to the number of export tasks, selecting the number of idle threads in the target thread pool to obtain data query threads; when the number of idle threads is less than the number of export tasks, creating new threads in the target thread pool based on the number of export tasks, and selecting the number of idle threads in the target thread pool to obtain data query threads; starting all the data query threads to perform concurrent queries on the source data table based on all the query statements to obtain export data of each of the export data groups; The export data of the export data group is filled into a target list template of the export data group to complete data export of the source data table.

2. The data export method of claim 1, wherein, The vector calculation is performed by using all the first segmented word vectors to obtain a first required feature vector, including: ； wherein y is the first required feature vector, m is the total number of the first segmented word vectors, j is the serial number of the first segmented word vector, x j is the first segmented word vector with serial number j.

3. The data export method of claim 1, wherein, The matching coefficients of each template identifier corresponding to the export requirement are obtained by performing calculation based on the template identifier vector and the export requirement vector, including: ； wherein a i is the element with index i in the derivation requirement vector a i is the element with index i in the template identification vector b, n is the vector dimension of the derivation requirement vector and the template identification vector, i is the index of the element in the derivation requirement vector and the template identification vector, the derivation requirement vector a is the derivation requirement vector corresponding to the derivation requirement A, the template identification vector b is the template identification vector corresponding to the template identification B, and p is the matching coefficient of the template identification B corresponding to the derivation requirement A.

4. The data export method of claim 1, wherein, The data query threads are obtained by selecting the idle threads in the target thread pool according to the number of export tasks, including: All idle threads in the target thread pool and the creation time of each idle thread are obtained. All idle threads in the target thread pool are sorted according to the creation time to obtain an idle thread sequence. The idle thread with a preset ranking and the idle thread before the preset ranking in the idle thread sequence are determined as the data query thread, where the value of the preset ranking is the same as the value of the number of export tasks.

5. The data export method of claim 1, wherein, The new threads in the target thread pool are created based on the number of export tasks, including: The number of created threads in the target thread pool is obtained to obtain a created thread number. The maximum thread number of the target thread pool is obtained. The difference between the maximum thread number and the created thread number is calculated to obtain a theoretical thread creation number. The number of idle threads in the target thread pool is obtained to obtain an idle thread number. The difference between the number of export tasks and the idle thread number is calculated to obtain an actual thread creation number. It is judged whether the theoretical thread creation number is greater than or equal to the actual thread creation number. When the theoretical thread creation number is less than the actual thread creation number, the target thread pool is updated in the maximum thread number based on the theoretical thread creation number and the actual thread creation number, and new threads are created in the target thread pool, and the number of created new threads is greater than or equal to the actual thread number. When the theoretical thread creation number is greater than or equal to the actual thread creation number, new threads are created in the target thread pool, and the number of created new threads is greater than or equal to the actual thread number.

6. The data export method of claim 5, wherein, The target thread pool is updated in the maximum thread number, including: The difference between the actual thread creation number and the theoretical thread creation number is calculated to obtain an adjustment difference. The sum of the maximum thread number and the adjustment difference is calculated to obtain a target adjustment maximum thread number. The maximum thread number of the target thread pool is adjusted so that the adjusted maximum thread number is greater than or equal to the target adjustment maximum thread number.

7. The data export method of any one of claims 1 to 6, wherein, All the data query threads are started to perform concurrent queries on the source data table by using all the query statements to obtain the export data of each export data group, including: All data query threads are traversed, and each data query thread is allocated a query statement; All data query threads are started, and each data query thread executes the query statement allocated to it to query the source data table, and saves the query result corresponding to each query statement; The saved query result corresponding to each query statement is determined as the exported data of the exported data group corresponding to the query statement.

8. A data export system, characterized by The system comprises: The template matching module is configured to, in response to a data export request of a source data table, extract all exported data groups in the data export request and an export requirement corresponding to each of the exported data groups; obtain a list template identifier of each list template in a preset list template library; based on the export requirement, perform matching screening on the template identifiers to screen a list template corresponding to each of the exported data groups, and obtain a target list template of each of the exported data groups; wherein the matching screening of the template identifiers based on the export requirement to screen the list template corresponding to each of the exported data groups and obtain the target list template of each of the exported data groups comprises: converting the export requirement into a vector to obtain an export requirement vector; converting the template identifier into a vector to obtain a template identifier vector; based on the template identifier vector and the export requirement vector, calculating a matching coefficient of each of the template identifiers corresponding to the export requirement; based on the matching coefficient, screening all of the template identifiers to obtain a target template identifier; determining an export target corresponding to the target template identifier as a target list template of the exported data group corresponding to the export requirement; wherein the converting the export requirement into a vector to obtain an export requirement vector comprises: performing word segmentation on the export requirement, and determining a word obtained through the word segmentation as a first segmented word; converting each of the first segmented words into a vector to obtain a first segmented word vector; performing vector calculation using all of the first segmented word vectors to obtain a first requirement feature vector; performing language translation conversion on the export requirement to obtain a translated export requirement; performing word segmentation on the translated export requirement, and determining a word obtained through the word segmentation as a second segmented word; converting the second segmented word into a vector to obtain a second segmented word vector; performing vector calculation using all of the second segmented word vectors to obtain a second requirement feature vector; performing horizontal splicing on the first requirement feature vector and the second requirement feature vector to obtain a requirement feature matrix; performing dimension reduction on the requirement feature matrix to obtain the export requirement vector; The data export query module is configured to calculate the number of the export data groups, obtain the number of export tasks, construct a query statement based on the export data groups, obtain the number of idle threads in a target thread pool, determine whether the number of idle threads is greater than or equal to the number of export tasks, select idle threads corresponding to the number of export tasks in the target thread pool when the number of idle threads is greater than or equal to the number of export tasks, obtain data query threads, create new threads in the target thread pool based on the number of export tasks when the number of idle threads is less than the number of export tasks, select idle threads corresponding to the number of export tasks in the target thread pool, obtain data query threads, and start all the data query threads to perform concurrent queries on the source data table by using all the query statements to obtain export data of each export data group. The data export module is configured to fill the export data of the export data group into a target manifest template of the export data group to complete data export of the source data table.

9. An electronic device, comprising: The electronic device includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the data export method of any one of claims 1-6.

Images