Data acquisition method, device, equipment and computer storage medium

Abstract

The application discloses a data collection method and device, equipment and a computer storage medium. The method comprises the following steps: in the case that a target field is determined from a field mapping relationship, acquiring an original data file based on the target field according to a collection mode and a collection frequency of the original data file, generating a data information file according to a predetermined rule based on the original data file, encrypting the original data file based on a predetermined encryption rule to obtain an encrypted file, and sending the data information file and the encrypted file to a server according to a predetermined path, so that the server decrypts the encrypted file, generates a reconciliation file according to a predetermined rule in the case that a decrypted file is obtained, and completes data collection in the case that the data information file and the reconciliation file are consistent. According to the embodiment of the application, local financial institution supervision data can be collected in time, accurately and efficiently, and the supervision efficiency is improved.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a data acquisition method. Background Technology

[0002] Local financial regulatory authorities, in accordance with relevant laws and regulations and under the guidance of the central statutory financial regulatory authorities, supervise and manage financial institutions, local financial organizations, and financial activities within their jurisdiction.

[0003] However, due to the late start of local financial regulation, data from various departments are distributed across different clusters, forming information silos. Furthermore, local financial organizations in the same industry do not have a unified business system. Currently, the collection of regulatory data mainly relies on traditional manual operation modes, which cannot guarantee the timely, accurate, and efficient collection of regulatory data from local financial institutions, resulting in low regulatory efficiency. Summary of the Invention

[0004] This application provides a data acquisition method, apparatus, device, and computer storage medium that effectively solves the problems of wasted manpower costs and inaccurate data caused by manual input due to multiple data sources. It also ensures the timeliness of data transmission, enabling timely, accurate, and efficient collection of regulatory data from local financial institutions, thereby improving regulatory efficiency.

[0005] In a first aspect, embodiments of this application provide a data acquisition method applied to a client, the method comprising:

[0006] Once the target field is determined from the field mapping relationship, the original data file is obtained based on the target field according to the original data file's acquisition mode and acquisition frequency. The original data file includes structured data.

[0007] A data information file is generated based on the original data file according to the information generation rules. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0008] The original data file is encrypted based on a predetermined encryption rule to obtain an encrypted file;

[0009] The system sends data information files and encrypted files to the server according to a predetermined path, so that the server can decrypt the encrypted files. Once the decrypted files are obtained, a reconciliation file is generated according to the information generation rules. If the data information files and the reconciliation files are consistent, the data collection is completed.

[0010] In one possible implementation, after encrypting the original data file based on a predetermined encryption rule to obtain an encrypted file, the method further includes:

[0011] The data file and the encrypted file are sliced ​​according to a predetermined slice length to obtain a sliced ​​transmission file;

[0012] Data information files and encrypted files are sent to the server according to the predetermined path, including:

[0013] The sliced ​​files are sent to the server according to the predetermined path.

[0014] In one possible implementation, before obtaining the original data file, the method further includes:

[0015] Configure field mapping relationships, acquisition mode, and acquisition frequency;

[0016] In one possible implementation embodiment, it further includes:

[0017] The information generation rules are sent to the server according to the predetermined path.

[0018] Secondly, embodiments of this application provide a data acquisition method applied to a server, the method comprising:

[0019] Receive data files and encrypted files sent by the client;

[0020] The encrypted file is decrypted based on the predetermined decryption rules to obtain the decrypted file;

[0021] A reconciliation file is generated based on the decrypted file according to the information generation rules. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines.

[0022] Data collection is completed when the data information file and the reconciliation file are consistent.

[0023] In one possible implementation, before generating the reconciliation file according to the information generation rules based on the decrypted file, the method further includes:

[0024] Receive information sent by the client to generate rules.

[0025] In one possible implementation embodiment, it further includes:

[0026] If the time taken to receive data files and encrypted files sent by the client exceeds a predetermined threshold, the system sends a first message to the client, which includes the filename, the number of transmissions, and the reason for the error.

[0027] In one possible implementation, after decrypting the encrypted file based on a predetermined decryption rule to obtain a decrypted file, the method further includes:

[0028] The decrypted files are stored in the database.

[0029] In one possible implementation, after saving the decrypted file in the database, the method further includes:

[0030] The check engine checks whether the decrypted file meets the target check rules.

[0031] Data collection is completed once the decrypted file meets the target verification rules.

[0032] Thirdly, embodiments of this application provide a data acquisition device applied to a client, comprising:

[0033] The acquisition module is used to acquire the original data file based on the target field and according to the acquisition mode and acquisition frequency of the original data file, when the target field is determined from the field mapping relationship. The original data file includes structured data.

[0034] The first generation module is used to generate a data information file based on the original data file according to the information generation rules. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0035] The encryption module is used to encrypt the original data file based on a predetermined encryption rule to obtain an encrypted file;

[0036] The sending module is used to send data information files and encrypted files to the server according to a predetermined path, so that the server can decrypt the encrypted files. After obtaining the decrypted files, a reconciliation file is generated according to the information generation rules. If the data information files and the reconciliation files are consistent, the data collection is completed.

[0037] In one possible implementation, the device further includes:

[0038] The slicing module is used to encrypt the original data file based on a predetermined encryption rule, and after obtaining the encrypted file, slice the data information file and the encrypted file according to a predetermined slice length to obtain the sliced ​​transmission file.

[0039] The sending module is used to send data information files and encrypted files to the server according to a predetermined path, including:

[0040] The sliced ​​files are sent to the server according to the predetermined path.

[0041] In one possible implementation, the device further includes:

[0042] The configuration module is used to configure field mapping relationships, acquisition mode, and acquisition frequency before acquiring the original data file;

[0043] In one possible implementation embodiment, it further includes:

[0044] The sending module is also used to send information generation rules to the server according to a predetermined path.

[0045] Fourthly, embodiments of this application provide a data acquisition device applied to a server, comprising:

[0046] The receiving module is used to receive data information files and encrypted files sent by the client;

[0047] The decryption module is used to decrypt encrypted files based on predetermined decryption rules to obtain decrypted files;

[0048] The second generation module is used to generate a reconciliation file according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines.

[0049] The judgment module is used to complete data collection when the data information file and the reconciliation file are consistent.

[0050] In one possible implementation, the device further includes:

[0051] The receiving module is also used to receive the information generation rules sent by the client before generating the reconciliation file according to the information generation rules based on the decrypted file.

[0052] In one possible implementation embodiment, it further includes:

[0053] The information feedback module is used to send a first message to the client if the time taken to receive the data information file and encrypted file sent by the client exceeds a predetermined threshold. The first message includes the file name, the number of transmissions, and the reason for the error.

[0054] In one possible implementation, the device further includes:

[0055] The storage module is used to decrypt encrypted files based on predetermined decryption rules, and after obtaining the decrypted files, save the decrypted files in the database.

[0056] In one possible implementation, the device further includes:

[0057] The verification module is used to verify whether the decrypted file meets the target verification rules after it is saved in the database, through the verification engine.

[0058] The judgment module is also used to complete data collection if the decrypted file meets the target check rules.

[0059] Fifthly, embodiments of this application provide a data acquisition device, the device comprising:

[0060] Processor and memory storing computer program instructions;

[0061] The processor executes computer program instructions to implement any of the above-mentioned data acquisition methods.

[0062] Sixthly, embodiments of this application provide a computer storage medium.

[0063] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement any of the above-mentioned data acquisition methods.

[0064] In a seventh aspect, embodiments of this application provide a computer program product, characterized in that, when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device is able to execute any of the above-mentioned data acquisition methods.

[0065] The data acquisition method, apparatus, device, and computer storage medium of this application embodiment, after determining the target field from the field mapping relationship, acquires the original data file based on the target field according to the acquisition mode and acquisition frequency of the original data file, generates a data information file according to the original data file according to predetermined rules, encrypts the original data file according to predetermined encryption rules to obtain an encrypted file, and sends the data information file and the encrypted file to the server according to a predetermined path for the server to decrypt the encrypted file. Upon obtaining the decrypted file, a reconciliation file is generated according to predetermined rules. If the data information file and the reconciliation file are consistent, the data acquisition is completed. This effectively solves the problem of wasted manpower costs caused by multiple data sources and inaccurate data caused by manual input, while also ensuring the timeliness of data transmission. It enables timely, accurate, and efficient acquisition of regulatory data from local financial institutions, improving regulatory efficiency. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating a data acquisition method provided in one embodiment of this application;

[0068] Figure 2 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0069] Figure 3 This is a flowchart illustrating a data acquisition method provided in yet another embodiment of this application;

[0070] Figure 4 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0071] Figure 5 This is a schematic diagram of the structure of a data acquisition device provided in another embodiment of this application;

[0072] Figure 6 This is a schematic diagram of the structure of a data acquisition device provided in another embodiment of this application;

[0073] Figure 7 This is a schematic diagram of the structure of a data acquisition device provided in another embodiment of this application. Detailed Implementation

[0074] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0076] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0077] Local financial regulatory authorities, in accordance with relevant laws and regulations and under the guidance of the central statutory financial regulatory authorities, supervise and manage financial institutions, local financial organizations, and financial activities within their jurisdiction.

[0078] However, due to the late start of local financial regulation, data from various departments are distributed across different clusters, forming information silos. Furthermore, local financial organizations within the same industry lack a unified business system. Currently, the traditional method for collecting regulatory data mainly relies on manual operation. This traditional method has certain limitations. Faced with a large number of data sources, it requires significant manpower for data entry and reporting. Manual entry and reporting inevitably leads to errors and omissions, compromising data accuracy. Moreover, relying on manual operation can easily result in information delays and untimely data transmission, making it impossible to guarantee the timely, accurate, and efficient collection of regulatory data from local financial institutions, resulting in low regulatory efficiency.

[0079] To address the problems of the prior art, embodiments of this application provide a data acquisition method, apparatus, device, and computer storage medium. The data acquisition method provided in this application embodiment will be described first below.

[0080] Figure 1 A flowchart illustrating a data acquisition method for a client provided in one embodiment of this application is shown.

[0081] like Figure 1 As shown in the embodiments of this application, the data acquisition method applied to the client includes the following steps.

[0082] S110. If the target field is determined from the field mapping relationship, the original data file is obtained based on the target field according to the original data file acquisition mode and acquisition frequency. The original data file includes structured data.

[0083] Here, the acquisition mode can be either full mode or incremental mode. The acquisition frequency is the frequency at which the original data file is acquired.

[0084] In some embodiments, when the target field is determined from the field mapping relationship, the original data file to be acquired can be determined. The original data file is acquired according to the original data acquisition mode and the original data file acquisition frequency. Here, the field mapping relationship can be the relationship between fields and data. Data corresponding to the target field can be acquired based on the target field. The original data file can include the target field and the data corresponding to the target field. The target field can be any one of the fields corresponding to the client data source. The data acquisition mode can be incremental mode (newly added data) or full mode (all data). That is, the acquired data can be newly added data in the client data source compared to the previous acquisition, or it can be all the data in the client data source.

[0085] As an example, structured data can be a table.

[0086] S120. Generate a data information file according to the information generation rules based on the original data file. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0087] Here, the information generation rules can extract information from the original data file and write it into the data information file. The information in the original data file may include, but is not limited to, the digest information MD5 code, file name, file size, and number of lines.

[0088] As an example, the data information file could be an OK file.

[0089] S130. The original data file is encrypted based on the predetermined encryption rules to obtain an encrypted file.

[0090] The encrypted file is the original data file after encryption.

[0091] Here, the pre-defined encryption rules can be stored in the client. The client can store multiple encryption rules, which can be symmetric encryption algorithms or asymmetric encryption algorithms. The pre-defined encryption rules can be one or more of multiple encryption rules, where multiple includes two.

[0092] As an example, the original data file is encrypted using the Advanced Encryption Standard (AES) encryption algorithm to obtain an encrypted file.

[0093] S140. Send the data information file and the encrypted file to the server according to the predetermined path so that the server can decrypt the encrypted file. If the decrypted file is obtained, generate the reconciliation file according to the information generation rules. If the data information file and the reconciliation file are consistent, the data collection is completed.

[0094] Here, the predefined path is a path that can receive path input from the user and save the user-inputted path on the client side, so that the client can send data information files and original data files to the server according to the path.

[0095] In some embodiments, the data information file and the original data file can be stored in a compressed package before being sent. Upon receiving the compressed package from the client, the server performs a decompression operation.

[0096] In this way, after determining the target field from the field mapping relationship, the original data file is obtained based on the target field according to the original data file's collection mode and frequency. A data information file is generated according to predetermined rules based on the original data file. The original data file is then encrypted according to predetermined encryption rules to obtain an encrypted file. Both the data information file and the encrypted file are sent to the server via a predetermined path for the server to decrypt the encrypted file. Once the decrypted file is obtained, a reconciliation file is generated according to predetermined rules. If the data information file and the reconciliation file are consistent, the data collection is complete. This effectively solves the problem of wasted manpower due to multiple data sources and inaccuracies caused by manual input, while also ensuring the timeliness of data transmission. It enables timely, accurate, and efficient collection of regulatory data from local financial institutions, improving regulatory efficiency.

[0097] Based on this, in some embodiments, after S130 above, the method may further include:

[0098] The data file and the encrypted file are sliced ​​according to a predetermined slice length to obtain a sliced ​​transmission file;

[0099] Data information files and encrypted files are sent to the server according to the predetermined path, including:

[0100] The sliced ​​files are sent to the server according to the predetermined path.

[0101] In some embodiments, the data information file and the original data file can be stored in a compressed package, and the compressed package can be sliced ​​according to a predetermined slice length to obtain a sliced ​​transmission file.

[0102] In some embodiments, the slice transfer file can be larger than 200MB.

[0103] This method of splitting the file before transmission prevents transmission difficulties caused by excessively large files.

[0104] Based on this, in some embodiments, before S110 above, the method may further include:

[0105] Configure field mapping relationships, acquisition mode, and acquisition frequency.

[0106] In some embodiments, the client receives user input regarding the field mapping relationship, acquisition mode, and acquisition frequency of the original data file, and in response to the input, saves the input.

[0107] In this way, data can be retrieved according to user needs and correctly according to field mapping relationships.

[0108] Based on this, in some embodiments, the method may further include:

[0109] The information generation rules are sent to the server according to the predetermined path.

[0110] In this way, the server can generate reconciliation files according to the same rules to verify whether the reconciliation files are consistent with the data information files.

[0111] Figure 2 A flowchart illustrating a data acquisition method applied to a server according to an embodiment of this application is shown.

[0112] like Figure 2 As shown in the embodiments of this application, the data collection method applied to the server includes the following steps.

[0113] S210: Receive data information files and encrypted files sent by the client.

[0114] Here, the data file and the encrypted file can be placed in the same compressed package. The compressed package can be decompressed to obtain the data file and the encrypted file.

[0115] S220. Decrypt the encrypted file based on the predetermined decryption rules to obtain the decrypted file.

[0116] Here, the pre-defined decryption rules can be stored in advance on the server. The server can store multiple decryption rules, which can decrypt files encrypted with different symmetric encryption algorithms and / or asymmetric encryption algorithms. The pre-defined decryption rules can be one or more of multiple decryption rules, where multiple includes two.

[0117] S230. Generate a reconciliation file according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines.

[0118] Here, the information generation rules can extract information from the decrypted file and write it into the reconciliation file. The information in the reconciliation file may include, but is not limited to, the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0119] As an example, the reconciliation file could be a Result file.

[0120] S240. If the data information file and the reconciliation file are consistent, complete the data collection.

[0121] In some embodiments, the MD5 hash, file name, file size, and number of lines of the digest information in the data information file and the reconciliation file can be compared. If the MD5 hash, file name, file size, and number of lines of the digest information are consistent, the data acquisition is completed.

[0122] In some embodiments, if the data information file and the reconciliation file are inconsistent, the server sends a second message to the client, which may include information that the original data file was sent incorrectly.

[0123] This ensures the consistency of file transfer, prevents errors from occurring during the process of sending data from the client to the server, and guarantees the accuracy of the data.

[0124] Based on this, in some embodiments, prior to the above-described S230, the method may further include:

[0125] Receive information sent by the client to generate rules.

[0126] In this way, the reconciliation file generated from the decrypted file and the data information file generated from the original data file can be generated according to the same information generation rules.

[0127] Based on this, in some embodiments, the method may further include:

[0128] If the time taken to receive data files and encrypted files sent by the client exceeds a predetermined threshold, the system sends a first message to the client, which includes the filename, the number of transmissions, and the reason for the error.

[0129] In some embodiments, if the time taken to receive the data information file and encrypted file sent by the client exceeds a predetermined threshold, a first message is sent back to the client, which may include at least one error reason.

[0130] In this way, if the server does not receive data from the client for an extended period of time, it will respond to the client, ensuring the timeliness of data collection.

[0131] In some embodiments, such as Figure 3 Following S220 above, the method may further include:

[0132] S221. Save the decrypted file in the database.

[0133] In some embodiments, the server provides a database to store the decrypted files.

[0134] This makes it easier for the server to retrieve data, and it eliminates the need to retrieve the same data from the client when needed. It also makes it easier for the server to verify the data.

[0135] Based on this, in some embodiments, after saving the decrypted file in the database, the method may further include:

[0136] The check engine checks whether the decrypted file meets the target check rules.

[0137] Data collection is completed once the decrypted file meets the target verification rules.

[0138] In some embodiments, the verification engine can use target verification rules to verify the decrypted file. Target verification rules can be verification rules pre-stored in a database. The database can store one or more verification rules, and the target verification rule can be any one of multiple verification rules, including two. Target verification rules can be verification rules to check whether the data content in the decrypted file conforms to industry standards, or they can be verification rules to check whether the decrypted file is acquired according to the field mapping relationship, acquisition mode, and acquisition frequency configured by the client.

[0139] As an example, the target verification rule could be that the data in the third decrypted file can be obtained based on the data in the first decrypted file and the data in the second decrypted file. If the first decrypted file, the second decrypted file, and the third decrypted file are obtained, the target verification rule is used for verification. If the target verification rule is met, the data acquisition is completed. If the target verification rule is not met, the location of the erroneous data in the first decrypted file, the second decrypted file, and the third decrypted file is located, and the location is fed back to the client.

[0140] In this way, the verification engine can verify the decrypted files, ensuring the accuracy of the data.

[0141] Figure 4 A flowchart illustrating a data acquisition method provided in one embodiment of this application is shown.

[0142] like Figure 4 As shown, the data acquisition method provided in this application embodiment may include the following steps.

[0143] S410. When the client determines the target field from the field mapping relationship, it obtains the original data file based on the target field according to the original data file acquisition mode and acquisition frequency. The original data file includes structured data.

[0144] S420: The client generates a data information file according to the information generation rules based on the original data file. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0145] S430: The client encrypts the original data file based on a predetermined encryption rule to obtain an encrypted file.

[0146] S440: The client sends data information files and encrypted files to the server according to the predetermined path.

[0147] S450: The server decrypts the encrypted file based on the predetermined decryption rules to obtain the decrypted file.

[0148] Here, the predefined decryption rules can be used to decrypt files encrypted based on predefined encryption rules.

[0149] S460. The server generates a reconciliation file according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines.

[0150] S470: The server completes data collection when the data information file and the reconciliation file are consistent.

[0151] In the embodiments provided in this application, when the target field is determined from the field mapping relationship, the original data file is obtained based on the target field according to the original data file's acquisition mode and frequency. A data information file is generated according to the original data file according to predetermined rules. The original data file is then encrypted according to predetermined encryption rules to obtain an encrypted file. The data information file and the encrypted file are sent to the server along a predetermined path for the server to decrypt the encrypted file. Upon obtaining the decrypted file, a reconciliation file is generated according to predetermined rules. If the data information file and the reconciliation file are consistent, the data acquisition is complete. This effectively solves the problem of wasted manpower due to multiple data sources and inaccurate data caused by manual input, while also ensuring the timeliness of data transmission. It enables timely, accurate, and efficient collection of regulatory data from local financial institutions, improving regulatory efficiency.

[0152] Based on the data acquisition method provided in the above embodiments, this application also provides specific implementations of the data acquisition device. Please refer to the following embodiments.

[0153] First see Figure 5 The data acquisition device 500 for a client application provided in this application embodiment may include:

[0154] The acquisition module 510 is used to acquire the original data file based on the target field according to the acquisition mode and acquisition frequency of the original data file, when the target field is determined from the field mapping relationship. The original data file includes structured data.

[0155] The first generation module 520 is used to generate a data information file according to the information generation rules based on the original data file. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines.

[0156] The encryption module 530 is used to encrypt the original data file based on a predetermined encryption rule to obtain an encrypted file;

[0157] The sending module 540 is used to send data information files and encrypted files to the server according to a predetermined path, so that the server can decrypt the encrypted files. If the decrypted files are obtained, a reconciliation file is generated according to the information generation rules. If the data information files and the reconciliation files are consistent, the data collection is completed.

[0158] Based on this, in some embodiments, the device 500 may further include:

[0159] The slicing module is used to encrypt the original data file based on a predetermined encryption rule, and after obtaining the encrypted file, slice the data information file and the encrypted file according to a predetermined slice length to obtain the sliced ​​transmission file.

[0160] Sending module 540 is used to send data information files and encrypted files to the server according to a predetermined path, including:

[0161] The sliced ​​files are sent to the server according to the predetermined path.

[0162] Based on this, in some embodiments, the device 500 may further include:

[0163] The configuration module is used to configure field mapping relationships, acquisition mode, and acquisition frequency before acquiring the original data file;

[0164] Based on this, in some embodiments, the device 500 may further include:

[0165] The sending module 540 is also used to send information generation rules to the server according to a predetermined path.

[0166] See Figure 6 The data acquisition device 600 applied to the server provided in this application embodiment may include:

[0167] The receiving module 610 is used to receive data information files and encrypted files sent by the client;

[0168] The decryption module 620 is used to decrypt the encrypted file based on a predetermined decryption rule to obtain the decrypted file;

[0169] The second generation module 630 is used to generate a reconciliation file according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines.

[0170] The judgment module 640 is used to complete data collection when the data information file and the reconciliation file are consistent.

[0171] Based on this, in some embodiments, the device 600 may further include:

[0172] The receiving module 610 is also used to receive the information generation rules sent by the client before generating the reconciliation file according to the information generation rules based on the decrypted file.

[0173] Based on this, in some embodiments, the device 600 may further include:

[0174] The information feedback module is used to send a first message to the client if the time taken to receive the data information file and encrypted file sent by the client exceeds a predetermined threshold. The first message includes the file name, the number of transmissions, and the reason for the error.

[0175] Based on this, in some embodiments, the device 600 may further include:

[0176] The storage module is used to decrypt encrypted files based on predetermined decryption rules, and after obtaining the decrypted files, save the decrypted files in the database.

[0177] Based on this, in some embodiments, the device 600 may further include:

[0178] The verification module is used to verify whether the decrypted file meets the target verification rules after it is saved in the database, through the verification engine.

[0179] The judgment module is also used to complete data collection if the decrypted file meets the target check rules.

[0180] The various modules of the data acquisition device provided in this application embodiment can achieve Figure 1 , Figure 2 , Figure 3 and Figure 4 It provides the functionality for each step of the data acquisition method and enables it to achieve the corresponding technical effects. For the sake of brevity, it will not be elaborated here.

[0181] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0182] Figure 7 A schematic diagram of the hardware structure of the data acquisition device provided in an embodiment of this application is shown.

[0183] The data acquisition device may include a processor 701 and a memory 702 storing computer program instructions.

[0184] Specifically, the processor 701 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0185] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.

[0186] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0187] The processor 701 reads and executes computer program instructions stored in the memory 702 to implement any of the data acquisition methods in the above embodiments.

[0188] In one example, the data acquisition device may further include a communication interface 703 and a bus 710. Wherein, as... Figure 7 As shown, the processor 701, memory 702, and communication interface 703 are connected through bus 710 and complete communication with each other.

[0189] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0190] Bus 710 includes hardware, software, or both, that couples components of a data acquisition device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 710 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnect. The electronic device can perform the data acquisition method described in the embodiments of the present invention, thereby achieving... Figure 1 , Figure 2 , Figure 3 and Figure 4 Describe the data acquisition method.

[0191] Furthermore, in conjunction with the data acquisition methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data acquisition methods described in the above embodiments.

[0192] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described data acquisition method embodiments.

[0193] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0194] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0195] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0196] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0197] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data acquisition method, applied to a client, characterized in that, include: The system receives user input regarding the field mapping relationship, acquisition mode, and acquisition frequency of the original data file. The acquisition mode includes full acquisition mode and incremental acquisition mode. When the target field is determined from the field mapping relationship, the original data file is obtained based on the target field according to the original data file acquisition mode and acquisition frequency, and the original data file includes structured data; A data information file is generated based on the original data file according to the information generation rules. The data information file includes the MD5 hash of the original data file, the file name, the file size, and the number of lines. The original data file is encrypted based on a predetermined encryption rule to obtain an encrypted file; The data information file and the encrypted file are sent to the server according to a predetermined path, so that the server can decrypt the encrypted file. If the decrypted file is obtained, the decrypted file is stored in the database, and a reconciliation file is generated according to the information generation rules. If the data information file and the reconciliation file are consistent, the data collection is completed. The verification engine checks whether the decrypted file meets the target verification rules. These rules include rules that allow data in the third decrypted file to be obtained from data in the first and second decrypted files, and rules that verify whether the decrypted file is acquired according to the field mapping relationship, acquisition mode, and acquisition frequency configured by the client. Given the first, second, and third decrypted files, verification is performed according to the target verification rules. If all three files meet the target verification rules, data acquisition is completed. If only one of the first, second, or third decrypted files meets the target verification rules, the location of erroneous data in each file is identified and reported back to the client.

2. The data acquisition method according to claim 1, characterized in that, After encrypting the original data file based on a predetermined encryption rule to obtain an encrypted file, the method further includes: The data information file and the encrypted file are sliced ​​according to a predetermined slice length to obtain a sliced ​​transmission file; Sending the data information file and the encrypted file to the server according to the predetermined path includes: The slice transfer file is sent to the server according to the predetermined path.

3. The data acquisition method according to claim 1, characterized in that, Also includes: The information generation rules are sent to the server according to the predetermined path.

4. A data acquisition method, applied to a server, characterized in that, include: Receive data files and encrypted files sent by the client; The encrypted file is decrypted based on a predetermined decryption rule to obtain a decrypted file, which is then stored in a database. A reconciliation file is generated according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines. Data collection is completed if the data information file and the reconciliation file are consistent. After saving the decrypted file in the database, the method further includes: using a verification engine to verify whether the decrypted file meets target verification rules. The target verification rules include verification rules that allow data in the third decrypted file to be obtained based on data in the first and second decrypted files, and verification rules for verifying whether the decrypted file is acquired according to the field mapping relationship, acquisition mode, and acquisition frequency configured by the client. The acquisition mode includes full mode and incremental mode. Given the first, second, and third decrypted files, verification is performed according to the target verification rules. If all three files meet the target verification rules, data acquisition is completed. If any one of the first, second, or third decrypted files meets the target verification rules, the location of erroneous data in the first, second, and third decrypted files is located and fed back to the client.

5. The data acquisition method according to claim 4, characterized in that, Before generating the reconciliation file according to the information generation rules based on the decrypted file, the method further includes: Receive the information generation rules sent by the client.

6. The data acquisition method according to claim 4, characterized in that, Also includes: If the time elapsed since the client sent the data file and the encrypted file exceeds a predetermined threshold, a first message is sent to the client. The first message includes the file name, the number of transmissions, and the reason for the error.

7. A data acquisition device, characterized in that, Applied to the client side, including: The acquisition module is used to receive user input regarding the field mapping relationship, acquisition mode, and acquisition frequency of the original data file. The acquisition mode includes full mode and incremental mode. When the target field is determined from the field mapping relationship, the original data file is acquired based on the target field according to the acquisition mode and acquisition frequency of the original data file. The original data file includes structured data. The first generation module is used to generate a data information file according to the information generation rules based on the original data file. The data information file includes the MD5 hash of the digest information of the original data file, the file name, the file size, and the number of lines. The encryption module is used to encrypt the original data file based on a predetermined encryption rule to obtain an encrypted file; The sending module is used to send the data information file and the encrypted file to the server according to a predetermined path, so that the server can decrypt the encrypted file. If a decrypted file is obtained, it is saved in a database, and a reconciliation file is generated according to the information generation rules. If the data information file and the reconciliation file are consistent, data acquisition is completed. A verification engine checks whether the decrypted file meets target verification rules. The target verification rules include rules that allow data in a third decrypted file to be obtained from data in the first and second decrypted files, and rules that check whether the decrypted file is obtained according to the field mapping relationship, acquisition mode, and acquisition frequency configured by the client. If the first, second, and third decrypted files are obtained, verification is performed according to the target verification rules. If all three files meet the target verification rules, data acquisition is completed. If any one of the first, second, or third decrypted files meets the target verification rules, the location of erroneous data in the first, second, and third decrypted files is located and fed back to the client.

8. The data acquisition device according to claim 7, characterized in that, The device further includes: The slicing module is used to encrypt the original data file based on a predetermined encryption rule to obtain an encrypted file, and then slice the data information file and the encrypted file according to a predetermined slice length to obtain a sliced ​​transmission file. The sending module is used to send the data information file and the encrypted file to the server according to a predetermined path, including: The slice transfer file is sent to the server according to the predetermined path.

9. The data acquisition device according to claim 7, characterized in that, Also includes: The sending module is also used to send the information generation rule to the server according to a predetermined path.

10. A data acquisition device, applied to a server, characterized in that, include: The receiving module is used to receive data information files and encrypted files sent by the client; The decryption module is used to decrypt the encrypted file based on a predetermined decryption rule, obtain a decrypted file, and save the decrypted file in a database; The second generation module is used to generate a reconciliation file according to the information generation rules based on the decrypted file. The reconciliation file includes the MD5 hash of the decrypted file, the file name, the file size, and the number of lines. The judgment module is used to complete data collection when the data information file and the reconciliation file are consistent; The device further includes: The verification module, after saving the decrypted file in the database, uses a verification engine to verify whether the decrypted file meets target verification rules. These target verification rules include rules that allow data in the third decrypted file to be obtained from data in the first and second decrypted files, as well as rules for verifying whether the decrypted file is acquired according to the field mapping relationship, acquisition mode, and acquisition frequency configured by the client. The acquisition mode includes full acquisition mode and incremental acquisition mode. Given the first, second, and third decrypted files, verification is performed according to the target verification rules. The judgment module is further configured to complete data acquisition if the first decrypted file, the second decrypted file, and the third decrypted file all meet the target verification rules; and to locate the position where erroneous data is generated in the first decrypted file, the second decrypted file, and the third decrypted file if the first decrypted file, the second decrypted file, or the third decrypted file all meet the target verification rules, and to feed this information back to the client.

11. The data acquisition device according to claim 10, characterized in that, The device further includes: The receiving module is further configured to receive the information generation rules sent by the client before generating the reconciliation file according to the information generation rules based on the decrypted file.

12. The data acquisition device according to claim 10, characterized in that, Also includes: The information feedback module is used to send first information to the client when the time for receiving the data information file and encrypted file sent by the client exceeds a predetermined threshold. The first information includes the file name, the number of transmissions, and the reason for the error.

13. A data acquisition device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data acquisition method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data acquisition method as described in any one of claims 1-6.

15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data acquisition method as described in any one of claims 1-6.

Images