Data import method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a data import method, which establishes a connection with a server and a resource end based on a third-party front end; receives the original data sent by the resource end, and decomposes the original data according to a preset decomposition dimension to obtain decomposed data; enters the decomposed data into a pool to be imported according to a preset data import rule; and imports the decomposed data into the server end through the pool to be imported. The present invention can establish a connection with a server and a resource end through a third-party front end, so that data import does not depend on the front end of the server end. After receiving the original data, the original data is decomposed, and corresponding decomposed data can be obtained according to different decomposition dimensions, so that the decomposed data can be tracked during the import process, thereby obtaining the import status of the decomposed data. In the case of import failure, the corresponding decomposed data can be re-imported.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a data import method, device, electronic equipment and storage medium. Background Art

[0002] With the development of image processing technology, a large amount of image data is needed to support various image processing scenarios. Image data is obtained by capturing different scenes with camera equipment, such as capturing faces and vehicles with cameras to obtain corresponding image data. Image data is generally stored in the database of the camera owner and used for various image-based services. For example, identity data and corresponding landing services are required for the implementation of identity recognition. The landing services can be provided by the service provider's server, but in this process, the identity data needs to be imported into the server. However, the existing server has a relatively simple support for data import. It needs to be imported at the front end of the server and imported using a gateway. The import process is not transparent. Identity data is used for identity implementation and needs to be deconstructed into multiple types of static data, such as pictures, texts, and numbers. Each time the import fails, the specific import situation is unknown, and all identity data can only be re-imported. When the amount of imported data is too large, after the import fails, it is not known which part of the data has a problem, and the time cost of re-importing through the front end of the server is high, making the import efficiency low. Summary of the invention

[0003] An embodiment of the present invention provides a data import method, which can establish a connection with a server and a resource end through a third-party front end, so that data import does not rely on the front end of the server. After receiving the original data, the original data is decomposed, and corresponding decomposed data can be obtained according to different decomposition dimensions, so that the decomposed data can be tracked during the import process, thereby obtaining the import status of the decomposed data. In the case of import failure, the corresponding decomposed data can be re-imported, thereby improving the efficiency of importing data.

[0004] In a first aspect, an embodiment of the present invention provides a data import method, the method comprising:

[0005] Based on the third-party front end, a connection is established with the server and the resource end. The resource end provides the original data for identity landing, and the server provides identity landing services based on the imported original data;

[0006] Receive the original data sent by the resource end, and decompose the original data according to a preset decomposition dimension to obtain decomposed data;

[0007] Entering the decomposed data into a pool to be imported according to a preset data import rule;

[0008] The decomposed data is imported into the server through the pool to be imported.

[0009] Optionally, the decomposition dimension includes a classification dimension and a granularity dimension, and decomposing the original data according to the preset decomposition dimension to obtain the decomposed data includes:

[0010] Performing a first decomposition on the original data according to the classification dimension to obtain classified data;

[0011] Performing a second decomposition on the classified data according to the granularity dimension to obtain decomposed data;

[0012] The first decomposition and the second decomposition are recorded to obtain a decomposition record of the decomposition data.

[0013] Optionally, the method further includes:

[0014] Processing the decomposition records according to the preset data import rules to obtain a decomposition record table;

[0015] According to the decomposition record table, the import status of the decomposition data is judged to obtain an import result table, wherein the import result table includes import success information and import failure information of the decomposition data;

[0016] The import is resumed according to the import failure information.

[0017] Optionally, the data import rule includes an import priority, and the processing of the decomposition record according to the preset data import rule to obtain a decomposition record table further includes:

[0018] Determining a first priority of the decomposed data according to a classification dimension of the decomposed data;

[0019] Determining a second priority of the decomposed data according to a granularity dimension of the decomposed data;

[0020] The decomposition records are sorted according to the first priority of the decomposition data and the second priority of the decomposition data to obtain the decomposition record table.

[0021] Optionally, the performing breakpoint-resume import according to the import failure information includes:

[0022] Determine the import data to be resumed that needs to be resumed according to the import failure information and the decomposition record table;

[0023] Determining the import priority of the data to be continued according to a preset continuation strategy;

[0024] According to the import priority of the data to be continued, a corresponding continued guidance pool is opened to import the data to be continued.

[0025] Optionally, the import failure information includes a first import failure reason based on a classification dimension and a second import failure reason based on a granularity dimension, and determining the import priority of the data to be continued according to a preset continued import strategy includes:

[0026] Determining the first priority of the data to be continued according to the first priority of the decomposed data corresponding to the data to be continued and the first import failure reason;

[0027] Determining the second priority of the data to be continued according to the second priority of the decomposed data corresponding to the data to be continued and the second import failure reason;

[0028] The import priority of the data to be continued is determined according to the first priority of the data to be continued and the second priority of the data to be continued.

[0029] Optionally, the step of opening a corresponding resumed-guided pool to import the data to be resumed according to the import priority of the data to be resumed includes:

[0030] According to the import priority of the data to be continued, a first continued guidance pool corresponding to the server and a second continued guidance pool corresponding to the local storage end or the mobile storage end are respectively opened;

[0031] Importing the to-be-resumed guidance data in the first guidance resuming pool to the server through the first thread pool, and updating the guidance resuming result of the server to the import result table;

[0032] The data to be continued in the second continuation pool is imported into the local storage end or the mobile storage end through the second thread pool, and the continuation result of the local storage end or the mobile storage end is updated to the import result table, and the continuation result of the local storage end or the mobile storage end includes a storage address.

[0033] In a second aspect, an embodiment of the present invention provides a data import device, the device comprising:

[0034] A connection module is used to establish a connection with a server and a resource end based on a third-party front end. The resource end provides raw data for identity landing, and the server end provides identity landing services based on the imported raw data;

[0035] A decomposition module, used for receiving the original data sent by the resource end, and decomposing the original data according to a preset decomposition dimension to obtain decomposed data;

[0036] An input module, used for inputting the decomposed data into a pool to be imported according to a preset data import rule;

[0037] An import module is used to import the decomposed data into the server through the to-be-imported pool.

[0038] In a third aspect, an embodiment of the present invention provides a data import system, including a third-party front end, a server end, a data end, and a data import device described in an embodiment of the present invention;

[0039] The resource end provides original data for identity landing, and the service end provides identity landing service according to the imported original data.

[0040] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data import method provided in the embodiment of the present invention when executing the computer program.

[0041] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data import method provided in the embodiment of the invention are implemented.

[0042] In an embodiment of the present invention, a connection with a server and a resource end is established based on a third-party front end, the resource end provides original data for identity landing, and the server end provides identity landing services based on the imported original data; the original data sent by the resource end is received, and the original data is decomposed according to a preset decomposition dimension to obtain decomposed data; the decomposed data is entered into a pool to be imported according to a preset data import rule; and the decomposed data is imported into the server end through the pool to be imported. The present invention can establish a connection with a server and a resource end through a third-party front end, so that data import does not depend on the front end of the server end. After receiving the original data, the original data is decomposed, and corresponding decomposed data can be obtained according to different decomposition dimensions, so that the decomposed data can be tracked during the import process, thereby obtaining the import status of the decomposed data. In the case of import failure, the corresponding decomposed data can be re-imported, thereby improving the efficiency of importing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is an architecture diagram of a data import system provided by an embodiment of the present invention;

[0045] Figure 2 is a flow chart of a data import method provided by an embodiment of the present invention;

[0046] Figure 3 It is a structural schematic diagram of a data import device provided by an embodiment of the present invention;

[0047] Figure 4 is a structural diagram of a decomposition module provided by an embodiment of the present invention;

[0048] Figure 5 is a structural schematic diagram of another data import device provided by an embodiment of the present invention;

[0049] Figure 6 is a structural schematic diagram of a processing module provided by an embodiment of the present invention;

[0050] Figure 7 It is a structural schematic diagram of a conduction module provided by an embodiment of the present invention;

[0051] Figure 8 is a structural schematic diagram of a fourth determination unit provided by an embodiment of the present invention;

[0052] Fig. 9 It is a structural schematic diagram of a development unit provided by an embodiment of the present invention;

[0053] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 , Figure 1 is an architecture diagram of a data import system provided by an embodiment of the present invention, such as Figure 1As shown, the data import system includes: a third-party front end 1, a server 2, a data end 3 and a data import device 4. The resource end provides the original data for identity landing, and the server 2 provides identity landing services according to the imported original data. The third-party front end 1 is used as a temporary front end of the server 2 to provide interface support for the server 2.

[0056] The above-mentioned server 2 can be a cloud server, the above-mentioned data terminal 3 can be the database of the data owner, the above-mentioned data import device 4 can be a device equipped with the data import method provided in the embodiment of the present invention, the above-mentioned data import device 4 can also be a third-party front end 1 equipped with the data import method provided in the embodiment of the present invention, and can also be an electronic device or an external storage medium equipped with the data import method provided in the embodiment of the present invention. Furthermore, the data import method provided in the embodiment of the present invention can be encapsulated as a data import tool, a data import component or a data import software and loaded in any third-party front end 1. Optionally, the data import method provided in the above-mentioned embodiment of the present invention can be encapsulated based on the monolithic architecture built by springboot, so that it can run independently based on the jdk1.8 environment, and the services that do not depend on the server 2 can be deployed independently, and can support: pure pictures, txt, csv, excel multi-type and multi-scenario import formats.

[0057] In a possible embodiment, after the third-party front desk 1 mounts the above-mentioned data import device 4, the above-mentioned data import device 4 can call the general interface protocol of the third-party front desk 1 to build a data terminal 3 interface connected to the data terminal 3 and a server 2 interface connected to the server 2. Further, the third-party front desk 1 displays the login interface by loading the login component, and logs in through the login interface displayed by the third-party front desk 1, so that the data import device 4 is connected to the data terminal 3 and the server 2 respectively through the general interface protocol. Specifically, a token can be obtained according to the login credential and the key key to log in.

[0058] The data import device 4 receives the above raw data sent by the resource end, decomposes the above raw data according to the preset decomposition dimension to obtain decomposed data; enters the above decomposed data into the to-be-imported pool according to the preset data import rule; and imports the above decomposed data into the above server end 2 through the to-be-imported pool. The above decomposition dimension includes a classification dimension and a granularity dimension. The above classification dimension may refer to the classification attribute of the data, such as classification by unit, region, image, text, number, etc.; the above granularity dimension may refer to the import magnitude of the data, such as decomposing 1T data into 1024 1G data.

[0059] The above-mentioned data import rules can be the data import priority. Different classification attributes correspond to different first priorities, and different import levels correspond to different second priorities. The import priority of the decomposed data can be determined based on the first priority and the second priority, and the import can be performed according to the import priority of the decomposed data.

[0060] The present invention can establish a connection with the server and the resource end through a third-party front end, so that data import does not rely on the front end of the server. After receiving the original data, the original data is decomposed, and corresponding decomposed data can be obtained according to different decomposition dimensions. Therefore, the decomposed data can be tracked during the import process, so as to obtain the import status of the decomposed data. In the case of import failure, the corresponding decomposed data can be re-imported, thereby improving the efficiency of importing data.

[0061] Optional, see Figure 2 , Figure 2 is a flow chart of a data import method provided by an embodiment of the present invention, the data import method is run in the above-mentioned data import system, such as Figure 2 As shown, the data import method includes the following steps:

[0062] 101. Based on the third-party front end, establish connections with the server and resource ends.

[0063] In the embodiment of the present invention, the resource end provides raw data for identity landing, and the server end provides identity landing service based on the imported raw data. The third-party front desk is an on-site third-party front desk, which is used as a temporary front end of the server end and provides interface support for the server end. The server end can be a cloud server end, and the data end can be a database of the data owner.

[0064] The third-party front-end's universal interface protocol can be used to construct a data-end interface connected to the data end and a server-end interface connected to the server end. Furthermore, the third-party front-end displays the login interface by loading the login component, and logs in through the login interface displayed by the third-party front-end, thereby connecting with the data end and the server end respectively through the universal interface protocol. Specifically, a token can be obtained according to the login credentials and key to log in.

[0065] 102. Receive the original data sent by the resource end, and decompose the original data according to a preset decomposition dimension to obtain decomposed data.

[0066] In the embodiment of the present invention, the above original data is the data in the database of the data owner, and the user can request data import and view the data import process through the third-party front-end interface. The above decomposition dimension includes classification dimension and granularity dimension. The above classification dimension can refer to the classification attribute of data, such as classification attribute by unit, region, image, text, number, etc.; the above granularity dimension can refer to the import level of data, for example, decomposing 1T data into 1024 1G data.

[0067] Furthermore, different classification attributes may correspond to different import levels. Specifically, different import levels may be determined according to the amount of data corresponding to different classification attributes. The amount of the above data is determined by the structure of the data. For example, the structure of image data is generally matrix data, which occupies a large amount of data and has a correspondingly large import level.

[0068] Specifically, the above-mentioned original data can be first decomposed according to the above-mentioned classification dimension to obtain classified data; the above-mentioned classified data can be secondly decomposed according to the above-mentioned granularity dimension to obtain decomposed data; the above-mentioned first decomposition and the above-mentioned second decomposition can be recorded to obtain a decomposition record of the above-mentioned decomposed data. The above-mentioned decomposition record is used to record the specific situation of each decomposed data, including recording the classification attribute and import magnitude corresponding to each decomposed data. In this way, the corresponding decomposed data can be found according to the decomposition record. In the case of import failure, the amount of data that failed to be imported can also be limited within the import magnitude, and the decomposed data that failed to be imported can be quickly found according to the decomposition record.

[0069] 103. Enter the decomposed data into the pool to be imported according to the preset data import rules.

[0070] In an embodiment of the present invention, the above-mentioned import rules include import priorities, and the above-mentioned pool to be imported is used to temporarily store the above-mentioned decomposed data. Different import threads or import timings can be assigned according to the import priority, for example, an import thread with a high import speed is assigned to the decomposed data with a high import priority, and an import thread with a low import speed is assigned to the decomposed data with a low import priority, or for example, the decomposed data with a high import priority is imported first in the import timing, and the decomposed data with a low import priority is imported later in the import timing. It is understandable that in the case of multi-threading, different import threads can be assigned according to the import priority, and in the case of single-threading, different import timings can be assigned according to the import priority. Multi-threading and single-threading can be determined according to the specific configuration of the third-party foreground. When the third-party foreground supports multi-threading, different import threads can be assigned according to the import priority. When the third-party foreground only supports multi-threading, different import timings can be assigned according to the import priority.

[0071] Optionally, the decomposition records can be processed according to the preset data import rules to obtain a decomposition record table; based on the decomposition record table, the import status of the decomposition data can be judged to obtain an import result table, which includes the import success information and import failure information of the decomposition data; and the import can be resumed according to the import failure information. In this way, the import order can be determined according to the import priority to ensure that important decomposition data is imported first. At the same time, the decomposition record table can be synchronized with the import order of the decomposition data. In the event of an import failure, the corresponding decomposition data can be directly found in the decomposition record table according to the import order.

[0072] Specifically, the first priority of the decomposed data can be determined according to the classification dimension of the decomposed data; the second priority of the decomposed data can be determined according to the granularity dimension of the decomposed data; and the decomposed records can be sorted according to the first priority of the decomposed data and the second priority of the decomposed data to obtain the decomposition record table.

[0073] The above-mentioned classification dimension may refer to the classification attribute of the data, such as classification attributes by unit, region, image, text, number, etc.; the first priority of decomposing the data may be determined according to the data volume of the classification attribute and the weight of the classification attribute. The larger the data volume of the classification attribute and the higher the weight of the classification attribute, the higher the first priority of the classification attribute. The above-mentioned granularity dimension may refer to the import level of the data. The second priority of decomposing the data may be determined according to the import level of the decomposition. The larger the import level, the higher the second priority.

[0074] In an embodiment of the present invention, the above-mentioned pool to be imported is opened according to the hardware resources of the third-party front-end and is used for temporarily storing decomposed data. The above-mentioned temporarily stored decomposed data is entered according to preset data import rules.

[0075] 104. Import the decomposed data to the server through the pool to be imported.

[0076] In the embodiment of the present invention, the decomposed data may be imported into the server according to the import rules through the pool to be imported.

[0077] In an embodiment of the present invention, a connection with a server and a resource end is established based on a third-party front end, the resource end provides original data for identity landing, and the server end provides identity landing services based on the imported original data; the original data sent by the resource end is received, and the original data is decomposed according to a preset decomposition dimension to obtain decomposed data; the decomposed data is entered into a pool to be imported according to a preset data import rule; and the decomposed data is imported into the server end through the pool to be imported. The present invention can establish a connection with a server and a resource end through a third-party front end, so that data import does not depend on the front end of the server end. After receiving the original data, the original data is decomposed, and corresponding decomposed data can be obtained according to different decomposition dimensions, so that the decomposed data can be tracked during the import process, thereby obtaining the import status of the decomposed data. In the case of import failure, the corresponding decomposed data can be re-imported.

[0078] Optionally, in an embodiment of the present invention, the data to be resumed that needs to be resumed from the breakpoint can be determined based on the above-mentioned import failure information and the above-mentioned decomposition record table; the import priority of the above-mentioned data to be resumed can be determined based on a pre-set resumption strategy; and according to the import priority of the above-mentioned data to be resumed, a corresponding resumption pool is opened to import the above-mentioned data to be resumed. The above-mentioned breakpoint resumption refers to re-importing the data from the disconnected point. Since the corresponding decomposed data can be quickly found from the decomposition record table, the resumption can be carried out in time, and since the import magnitude of the decomposed data is limited, the amount of data that fails to be imported can be limited within the import magnitude, and the decomposed data that fails to be imported can be quickly found according to the decomposition record and quickly resumed.

[0079] Optionally, in an embodiment of the present invention, the above-mentioned import failure information includes a first import failure reason based on a classification dimension and a second import failure reason based on a granularity dimension, and the pre-set continuation strategy may be: determining the first priority of the above-mentioned data to be continued according to the first priority of the decomposed data corresponding to the above-mentioned data to be continued and the first import failure reason; determining the second priority of the above-mentioned data to be continued according to the second priority of the decomposed data corresponding to the above-mentioned data to be continued and the second import failure reason; determining the import priority of the above-mentioned data to be continued according to the first priority of the above-mentioned data to be continued and the second priority of the above-mentioned data to be continued.

[0080] Furthermore, the first import failure reason may be a regular failure reason and an unconventional failure reason of the classification attribute, the regular failure reason being a failure reason that has already occurred, and the unconventional reason being a failure reason that has not occurred. A first weight is set for the regular failure reason to weight the first priority, and a second weight is set for the unconventional failure reason to weight the first priority to obtain the second priority of the data to be continued. The first weight is greater than the second weight. Specifically, the first weight is 0.8-1. The first weight increases with the increase in the number of regular failure reasons, because the more times a regular failure reason occurs, the greater the probability that it will be quickly resolved. The second weight is 0.5. Similarly, the second import failure reason can be a regular failure and an unconventional failure reason of the import level. For the regular failure reason, a third weight is set to weight the above second priority, and for the unconventional failure reason, a fourth weight is set to weight the above second priority to obtain the second priority of the data to be continued. The above third weight is greater than the above fourth weight. Specifically, the above third weight is 0.8-1. The above third weight increases with the increase in the number of regular failure reasons, because the more times the regular failure reasons appear, the greater the probability of being quickly resolved. The above fourth weight is 0.5. The import priority of the data to be continued can be the sum of the first priority of the data to be continued and the second priority of the data to be continued.

[0081] Optionally, in an embodiment of the present invention, a first continuation pool corresponding to the above-mentioned server and a second continuation pool corresponding to the local storage end or the mobile storage end may be opened up respectively according to the import priority of the above-mentioned data to be continued; the data to be continued in the above-mentioned first continuation pool may be imported into the above-mentioned server through the first thread pool, and the continuation result of the above-mentioned server may be updated to the above-mentioned import result table; the data to be continued in the above-mentioned second continuation pool may be imported into the above-mentioned local storage end or the mobile storage end through the second thread pool, and the continuation result of the above-mentioned local storage end or the mobile storage end may be updated to the above-mentioned import result table, and the continuation result of the above-mentioned local storage end or the mobile storage end may include a storage address.

[0082] The decomposed data that failed to be imported on-site can be prompted through the local storage end, mobile storage end or server end after a preset time interval, so as to prompt the user to upload it to the server end within the preset time, which is convenient for the breakpoint resumption through the decomposed data of the local storage end or mobile storage end when the on-site residence time is tight, and also enables the server end to obtain the complete data of the data end. Since the original data is split into decomposed data, a single decomposed data belongs to the case of small data volume. When the amount of unimported data reaches a certain amount or proportion, the staff is prompted to check and maintain the import system or import device, or match the automatic inspection and maintenance plan, which is preset in the plan pool.

[0083] Optionally, in an embodiment of the present invention, the import magnitude of each classification attribute can be determined according to the import efficiency corresponding to different classification attributes. The above import efficiency can be determined according to the import speed and the import success rate. Specifically, the import speed is related to the import thread. Each import thread is used to import the decomposed data of a classification attribute. The import failure rate can be estimated based on the failure rate of historical import data. In this way, the import time can be optimized, thereby improving the efficiency of import. In an embodiment of the present invention, taking the minimum import time as the objective function, an import magnitude optimization model is established as shown in the following formula:

[0084]

[0085] T≤T max

[0086] q min ≤q ij ≤q max

[0087] v min ≤v i ≤v max

[0088]

[0089]

[0090] In the above import level optimization model, minT means the minimum import time, T max represents the on-site residence time, N represents the number of categories of the classification attribute, which can also be understood as the number of import threads, and M i represents the number of decomposed data under the i-th classification attribute, q ij represents the amount of data for the jth decomposition data under the i-th classification attribute, q min Indicates the maximum amount of decomposed data, q max Indicates the maximum amount of decomposed data, v i represents the import speed of the import thread corresponding to the i-th classification attribute, v min Indicates the minimum import speed of the import thread, v max Indicates the maximum import speed of the import thread, O i Indicates the number of decompositions of the to-be-continued data under the i-th classification attribute during the continuation process. represents the amount of data of the jth decomposition data under the i-th classification attribute, λ i Indicates the import failure rate corresponding to the i-th classification attribute.

[0091] It should be noted that the data import method provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform data import.

[0092] Optional, see Figure 3 , Figure 3 is a structural diagram of a data import device provided by an embodiment of the present invention, such as Figure 3 As shown, the device comprises:

[0093] The connection module 301 is used to establish a connection with a server and a resource end based on a third-party front end, wherein the resource end provides original data for identity landing, and the server end provides identity landing services according to the imported original data;

[0094] The decomposition module 302 is used to receive the original data sent by the resource end, and decompose the original data according to a preset decomposition dimension to obtain decomposed data;

[0095] An input module 303 is used to input the decomposed data into a pool to be imported according to a preset data import rule;

[0096] The import module 304 is used to import the decomposed data into the server through the to-be-imported pool.

[0097] Optional, such as Figure 4 As shown, the decomposition module 302 includes:

[0098] A first decomposition unit 3021 is used to perform a first decomposition on the original data according to the classification dimension to obtain classified data;

[0099] A second decomposition unit 3022, configured to perform a second decomposition on the classified data according to the granularity dimension to obtain decomposed data;

[0100] The recording unit 3023 is used to record the first decomposition and the second decomposition to obtain a decomposition record of the decomposition data.

[0101] Optional, such as Figure 5 As shown, the device also includes:

[0102] The processing module 305 is used to process the decomposition record according to the preset data import rule to obtain a decomposition record table;

[0103] A judgment module 306 is used to judge the import status of the decomposed data according to the decomposition record table to obtain an import result table, wherein the import result table includes import success information and import failure information of the decomposed data;

[0104] The resume module 307 is used to resume the breakpoint according to the import failure information.

[0105] Optional, such as Figure 6 As shown, the processing module 305 includes:

[0106] A first determining unit 3051 is configured to determine a first priority of the decomposed data according to a classification dimension of the decomposed data;

[0107] A second determining unit 3052 is configured to determine a second priority of the decomposed data according to a granularity dimension of the decomposed data;

[0108] The sorting unit 3053 is used to sort the decomposed records according to the first priority of the decomposed data and the second priority of the decomposed data to obtain the decomposed record table.

[0109] Optional, such as Figure 7 The said continuation module 307 comprises:

[0110] The third determining unit 3071 is used to determine the to-be-resumed import data that needs to be resumed according to the import failure information and the decomposition record table;

[0111] The fourth determining unit 3072 is used to determine the import priority of the to-be-continued data according to a preset continuation strategy;

[0112] The opening unit 3073 is used to open a corresponding resume pool to import the data to be continued according to the import priority of the data to be continued.

[0113] Optional, such as Figure 8 The fourth determining unit 3072 includes:

[0114] A first determining subunit 30721 is used to determine the first priority of the data to be continued according to the first priority of the decomposed data corresponding to the data to be continued and the first import failure reason;

[0115] The second determining subunit 30722 is used to determine the second priority of the data to be continued according to the second priority of the decomposed data corresponding to the data to be continued and the second import failure reason;

[0116] The third determining subunit 30723 is used to determine the importing priority of the data to be continued according to the first priority of the data to be continued and the second priority of the data to be continued.

[0117] Optional, such as Fig. 9 As shown, the development unit 3073 includes:

[0118] An opening subunit 30731 is used to open a first resuming pool corresponding to the server and a second resuming pool corresponding to the local storage end or the mobile storage end according to the import priority of the data to be resumed;

[0119] A first updating subunit 30732 is used to import the to-be-resumed guidance data in the first guidance resuming pool to the server through the first thread pool, and update the guidance resuming result of the server to the import result table;

[0120] The second update sub-unit 30733 is used to import the data to be continued in the second continuation pool to the local storage end or the mobile storage end through the second thread pool, and update the continuation result of the local storage end or the mobile storage end to the import result table, and the continuation result of the local storage end or the mobile storage end includes a storage address.

[0121] It should be noted that the data import device provided in the embodiment of the present invention can be applied to smart phones, computers, servers and other devices that can perform business analysis at the map level.

[0122] The data import device provided in the embodiment of the present invention can implement each process implemented by the data import method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0123] See also Fig.10 , Fig.10 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, it includes: a memory 1002, a processor 1001, and a computer program of a data import method stored in the memory 1002 and executable on the processor 1001, wherein:

[0124] The processor 1001 is used to call the computer program stored in the memory 1002 and execute the following steps:

[0125] Based on the third-party front end, a connection is established with the server and the resource end. The resource end provides the original data for identity landing, and the server provides identity landing services based on the imported original data;

[0126] Receive the original data sent by the resource end, and decompose the original data according to a preset decomposition dimension to obtain decomposed data;

[0127] Entering the decomposed data into a pool to be imported according to a preset data import rule;

[0128] The decomposed data is imported into the server through the pool to be imported.

[0129] Optionally, the decomposition dimension includes a classification dimension and a granularity dimension, and the processor 1001 performs the decomposition of the original data according to the preset decomposition dimension to obtain the decomposed data, including:

[0130] Performing a first decomposition on the original data according to the classification dimension to obtain classified data;

[0131] Performing a second decomposition on the classified data according to the granularity dimension to obtain decomposed data;

[0132] The first decomposition and the second decomposition are recorded to obtain a decomposition record of the decomposition data.

[0133] Optionally, the processor 1001 executes the method further comprising:

[0134] Processing the decomposition records according to the preset data import rules to obtain a decomposition record table;

[0135] According to the decomposition record table, the import status of the decomposition data is judged to obtain an import result table, wherein the import result table includes import success information and import failure information of the decomposition data;

[0136] The import is resumed according to the import failure information.

[0137] Optionally, the data import rule includes an import priority, and the processor 1001 processes the decomposition record according to the preset data import rule to obtain a decomposition record table, further comprising:

[0138] Determining a first priority of the decomposed data according to a classification dimension of the decomposed data;

[0139] Determining a second priority of the decomposed data according to a granularity dimension of the decomposed data;

[0140] The decomposition records are sorted according to the first priority of the decomposition data and the second priority of the decomposition data to obtain the decomposition record table.

[0141] Optionally, the processor 1001 performs the breakpoint-resume booting according to the import failure information, including:

[0142] Determine the import data to be resumed that needs to be resumed according to the import failure information and the decomposition record table;

[0143] Determining the import priority of the data to be continued according to a preset continuation strategy;

[0144] According to the import priority of the data to be continued, a corresponding continued guidance pool is opened to import the data to be continued.

[0145] Optionally, the import failure information includes a first import failure reason based on a classification dimension and a second import failure reason based on a granularity dimension, and the processor 1001 determines the import priority of the to-be-continued data according to a pre-set continuation strategy, including:

[0146] Determining the first priority of the data to be continued according to the first priority of the decomposed data corresponding to the data to be continued and the first import failure reason;

[0147] Determining the second priority of the data to be continued according to the second priority of the decomposed data corresponding to the data to be continued and the second import failure reason;

[0148] The import priority of the data to be continued is determined according to the first priority of the data to be continued and the second priority of the data to be continued.

[0149] Optionally, the processor 1001 opens a corresponding resumed guidance pool to import the to-be-resumed guidance data according to the import priority of the to-be-resumed guidance data, including:

[0150] According to the import priority of the data to be continued, a first continued guidance pool corresponding to the server and a second continued guidance pool corresponding to the local storage end or the mobile storage end are respectively opened;

[0151] Importing the to-be-resumed guidance data in the first guidance resuming pool to the server through the first thread pool, and updating the guidance resuming result of the server to the import result table;

[0152] The data to be continued in the second continuation pool is imported into the local storage end or the mobile storage end through the second thread pool, and the continuation result of the local storage end or the mobile storage end is updated to the import result table, and the continuation result of the local storage end or the mobile storage end includes a storage address.

[0153] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can import data.

[0154] The electronic device provided in the embodiment of the present invention can implement each process implemented by the data import method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0155] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the data import method or the application-side data import method provided in the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0157] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data import method, characterized in that: The following steps are involved: Based on the third-party front end, a connection is established with the server and the resource end. The resource end provides the original data for identity landing, and the server provides identity landing services based on the imported original data; The original data sent by the resource end is received, and the original data is decomposed according to a preset decomposition dimension to obtain decomposed data; the decomposition dimension includes a classification dimension and a granularity dimension, and the original data is first decomposed according to the classification dimension to obtain classified data; the classified data is secondly decomposed according to the granularity dimension to obtain decomposed data; Recording the first decomposition and the second decomposition to obtain a decomposition record of the decomposition data; Entering the decomposed data into a pool to be imported according to a preset data import rule; The data import rules include import priority; Importing the decomposed data into the server through the pool to be imported; Determining a first priority of the decomposed data according to a classification dimension of the decomposed data; Determining a second priority of the decomposed data according to a granularity dimension of the decomposed data; sorting the decomposition records according to the first priority of the decomposition data and the second priority of the decomposition data to obtain a decomposition record table; According to the decomposition record table, the import status of the decomposition data is judged to obtain an import result table, wherein the import result table includes import success information and import failure information of the decomposition data; the import failure information includes a first import failure reason based on a classification dimension and a second import failure reason based on a granularity dimension; Determine the import data to be resumed that needs to be resumed according to the import failure information and the decomposition record table; Determining the first priority of the data to be continued according to the first priority of the decomposed data corresponding to the data to be continued and the first import failure reason; Determining the second priority of the data to be continued according to the second priority of the decomposed data corresponding to the data to be continued and the second import failure reason; Determining the import priority of the data to be continued according to the first priority of the data to be continued and the second priority of the data to be continued; According to the import priority of the data to be continued, a corresponding continued guidance pool is opened to import the data to be continued.

2. The method according to claim 1, characterized in that The step of opening a corresponding continued guidance pool to import the data to be continued according to the import priority of the data to be continued includes: According to the import priority of the data to be continued, a first continued guidance pool corresponding to the server and a second continued guidance pool corresponding to the local storage end or the mobile storage end are respectively opened; Importing the to-be-resumed guidance data in the first guidance resuming pool to the server through the first thread pool, and updating the guidance resuming result of the server to the import result table; The data to be continued in the second continuation pool is imported into the local storage end or the mobile storage end through the second thread pool, and the continuation result of the local storage end or the mobile storage end is updated to the import result table, and the continuation result of the local storage end or the mobile storage end includes a storage address.

3. A data import device, characterized in that: The device comprises: A connection module is used to establish a connection with a server and a resource end based on a third-party front end. The resource end provides raw data for identity landing, and the server end provides identity landing services based on the imported raw data; A decomposition module is used to receive the original data sent by the resource end, and decompose the original data according to a preset decomposition dimension to obtain decomposed data; the decomposition dimension includes a classification dimension and a granularity dimension, and the original data is first decomposed according to the classification dimension to obtain classified data; the classified data is secondly decomposed according to the granularity dimension to obtain decomposed data; the first decomposition and the second decomposition are recorded to obtain a decomposition record of the decomposed data; An input module, used for inputting the decomposed data into a pool to be imported according to a preset data import rule; the data import rule includes an import priority; An import module, used for importing the decomposed data into the server through the to-be-imported pool; a processing module, configured to determine a first priority of the decomposed data according to a classification dimension of the decomposed data; determine a second priority of the decomposed data according to a granularity dimension of the decomposed data; and sort the decomposed records according to the first priority of the decomposed data and the second priority of the decomposed data to obtain a decomposed record table; a judgment module, configured to judge the import status of the decomposed data according to the decomposition record table, and obtain an import result table, wherein the import result table includes import success information and import failure information of the decomposed data; the import failure information includes a first import failure reason based on a classification dimension and a second import failure reason based on a granularity dimension; The resume module is used to determine the data to be resumed that needs to be resumed based on the import failure information and the decomposition record table; determine the first priority of the data to be resumed based on the first priority of the decomposed data corresponding to the data to be resumed and the first import failure reason; determine the second priority of the data to be resumed based on the second priority of the decomposed data corresponding to the data to be resumed and the second import failure reason; determine the import priority of the data to be resumed based on the first priority of the data to be resumed and the second priority of the data to be resumed; and according to the import priority of the data to be resumed, open up a corresponding resume pool to import the data to be resumed.

4. A data import system, characterized in that: It includes a third-party front end, a service end, a data end, and the data import device as claimed in claim 3; The resource end provides original data for identity landing, and the service end provides identity landing service according to the imported original data.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data import method as claimed in claim 1 or 2 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data import method according to claim 1 or 2 are implemented.

Images