Data storage method, device, equipment and medium

Abstract

The embodiment of the present application provides a method, device, equipment and medium for data storage. The method includes: obtaining the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition; based on the preset verification rule, using the Spark engine to perform a quality inspection on the first data to generate a quality inspection report, and when the data verification result included in the quality inspection report meets the second preset condition, controlling the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database reads and stores the first data from the formal area of ​​the dedicated network device. According to the embodiment of the present application, not only the efficiency of data inspection can be improved, but also the pressure on the database can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data storage method, device, equipment and medium. Background Art

[0002] In real life, banks and other financial institutions often involve massive amounts of data, and these data need to be quality checked by relevant regulatory agencies to ensure the correctness of the data. However, in the prior art, the data quality is generally checked after it is stored in the database. However, due to the large amount of data involved and the uneven data quality, it is easy to repeatedly check the data quality during the inspection process, which not only reduces the efficiency of data quality inspection, but also causes excessive pressure on the database. Summary of the invention

[0003] The embodiments of the present application provide a data storage method, device, equipment and medium, which can improve the efficiency of data quality inspection and reduce database pressure.

[0004] In a first aspect, an embodiment of the present application provides a data storage method, the method comprising: obtaining the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition;

[0005] Based on the preset checking rules, the Spark engine is used to perform a quality check on the first data to generate a quality check report.

[0006] When the data verification result included in the quality inspection report meets the second preset condition, the dedicated network device is controlled to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database reads and stores the first data from the formal area of ​​the dedicated network device.

[0007] In an optional implementation of the first aspect, the method further includes:

[0008] When the data verification result included in the quality inspection report does not meet the second preset condition, the dedicated network device is controlled to store the first data and the quality inspection report in a temporary area of ​​the dedicated network device, so that the first device obtains the first data and the quality inspection report from the temporary area of ​​the dedicated network device to correct the first data according to the quality inspection report.

[0009] In an optional implementation of the first aspect, the first data stored in the temporary area of ​​the dedicated network device is data transmitted by the first device to the temporary area of ​​the dedicated network device through a file transfer protocol.

[0010] In an optional implementation of the first aspect, the first preset condition is that the data size of the first data is consistent with the data size obtained by the dedicated network device from parsing an extensible markup language XML file corresponding to the first data.

[0011] In an optional implementation of the first aspect, the preset check rule includes that all data in the first data satisfies a first condition, and a preset proportion of data in the first data satisfies a second condition.

[0012] In an optional implementation of the first aspect, the preset checking rules are configured in an EXCEL table using an automated generation tool.

[0013] In a second aspect, an embodiment of the present application provides a data storage device, the device comprising:

[0014] An acquisition module, configured to acquire the first data if the first data stored in the temporary area of ​​the dedicated network device meets a first preset condition;

[0015] A quality inspection module, configured to perform a quality inspection on the first data using a Spark engine based on preset inspection rules to generate a quality inspection report;

[0016] The transfer module is used to control the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device when the data verification result included in the quality inspection report meets the second preset condition, so that the database reads and stores the first data from the formal area of ​​the dedicated network device.

[0017] In a third aspect, a data warehousing device is provided, comprising: a memory for storing computer program instructions; and a processor for reading and running the computer program instructions stored in the memory to execute the data warehousing method provided by any optional implementation of the first aspect and the second aspect.

[0018] In a fourth aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, a resource space method provided by any optional implementation of the first aspect and the second aspect is implemented.

[0019] In a fifth aspect, a computer program product is provided. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data warehousing method provided by any optional implementation of the first aspect and the second aspect.

[0020] The embodiments of the present application provide a data warehousing method, apparatus, device and medium, which obtains the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition, and performs a quality check on the first data based on the preset verification rules and the Spark engine to generate a quality inspection report, and controls the dedicated network device to transfer the first data originally stored in the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device when the data verification result included in the quality inspection report meets the second preset condition, thereby facilitating the database to directly read and store the first data from the formal area of ​​the dedicated network device. In this way, not only the efficiency of data inspection can be improved, but also the pressure on the database can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is an architecture diagram of a data storage method provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a data storage method provided in an embodiment of the present application;

[0024] Figure 3 It is a flowchart of another data storage method provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of a data storage device provided in an embodiment of the present application;

[0026] Figure 5 It is a structural schematic diagram of a data storage device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0028] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0029] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0030] In order to solve the problem of excessive pressure on the database caused by the uneven quality of the data entering the database in the prior art, the embodiment of the present application provides a data entry method, device, equipment and medium. The method mainly obtains the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition, and performs a quality check on the first data based on the preset verification rules and the Spark engine to generate a quality inspection report, and when the data verification result included in the quality inspection report meets the second preset condition, the dedicated network device is controlled to transfer the first data originally stored in the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, thereby facilitating the database to directly read and store the first data from the formal area of ​​the dedicated network device. In this way, not only the efficiency of data inspection can be improved, but also the pressure on the database can be effectively reduced.

[0031] The data storage method, device, equipment and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments.

[0032] It should be noted that the data storage method provided in the embodiment of the present application can be applied to Figure 1 In the architecture shown, the specific combination Figure 1 Provide detailed explanation.

[0033] Figure 1 This is an architectural diagram of a data storage method provided in an embodiment of the present application.

[0034] like Figure 1As shown, the architecture diagram may include a first device 10, a dedicated network device 20, an inspection device 30, and a database 40. There is information interaction between the first device 10 and the dedicated network device 20, between the dedicated network device 20 and the inspection device 30, and between the dedicated network device 20 and the database 40, so as to realize data storage.

[0035] Based on the above architecture, before storing the first data in the database, Figure 2 As shown, the first device sends the first data and the Extensible Markup Language (XML) file of the first data to the dedicated network device by means of the file transfer protocol, and stores the first data and the XML file of the first data in the temporary area of ​​the dedicated network device, and then the dedicated network device parses the XML file of the first data, and judges whether the first data is uploaded completely according to the data size obtained by the parsing, and when the first data is uploaded completely, the inspection device obtains the first data, and based on the preset verification rules, uses the Spark engine to perform a quality inspection on the first data to generate a quality inspection report for the first data, and then, when the data verification result included in the quality inspection report is qualified, the dedicated network device is controlled to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database can read and store the first data from the temporary area of ​​the dedicated network device. In this way, the quality inspection of the data can be completed before the data is stored, avoiding the situation of repeatedly checking the data in the database, thereby achieving the improvement of the efficiency of data inspection and reducing the pressure on the database.

[0036] According to the above architecture, the following Figure 3 The method for data storage provided in the embodiment of the present application is described in detail. The embodiment of the present application takes the execution subject as an inspection device as an example to describe the method for data storage provided in the embodiment of the present application in detail.

[0037] Figure 3 It is a flowchart of a data storage method provided in an embodiment of the present application.

[0038] like Figure 3 As shown, the data storage method may specifically include the following steps:

[0039] S110, when the first data stored in the temporary area of ​​the dedicated network device meets a first preset condition, acquiring the first data.

[0040] Among them, the dedicated network device is used to connect the first device and the inspection device. The dedicated network device can be divided into a temporary area and a formal area to facilitate the storage of relevant data according to actual conditions. It should be noted that the first device and the inspection device can modify the data stored in the temporary area of ​​the dedicated network device, but cannot modify the data stored in the formal area of ​​the dedicated network device. The first device can be a server or terminal of an institution such as a bank, etc., which can involve massive data. The inspection device is used to perform quality inspection on the massive data involved in the first device. The first data stored in the temporary area of ​​the dedicated network device is the data transmitted by the first device to the temporary area of ​​the dedicated network device through the file transfer protocol. For example, it can be the data of the on-site inspection and analysis system (EAST), or it can be the deposit data, loan data, etc. involved in the bank, which are not specifically limited here. Among them, the file transfer protocol is a set of standard protocols for file transfer on the network, allowing users to transfer data with the inspection device in the form of file operations, such as file addition, deletion, modification, and query operations. The first preset condition is that the data size of the first data is consistent with the data size obtained by the dedicated network device from the XML file corresponding to the first data.

[0041] Specifically, when the data size of the first data stored in the temporary area of ​​the dedicated network device is consistent with the data size obtained by the dedicated network device from parsing the XML file corresponding to the first data, the inspection device will directly obtain the first data from the temporary area of ​​the dedicated network device. In this way, based on the first data in the temporary area of ​​the dedicated network transmitted in the standard reporting method of the file transfer protocol, the XML file corresponding to the first data can be parsed to determine whether the first data is uploaded completely, which not only solves the problem of low quality of reported data to a certain extent, but also ensures that the inspection device obtains complete first data for quality inspection.

[0042] S120, based on preset checking rules, using the Spark engine to perform a quality check on the first data to generate a quality check report.

[0043] Among them, the preset check rule is at least one rule preset based on actual experience and used to check whether the quality inspection report of the data is qualified. The preset check rule includes a first condition satisfied by all the data in the first data, and a second condition satisfied by a preset proportion of the data in the first data. The first condition and the second condition are preset based on actual needs and are not specifically limited here. That is to say, the preset check rules in the embodiment of the present application are divided into two categories, one is the first condition satisfied by all the data in the first data, that is, a mandatory condition. The other is the second data satisfied by a preset proportion of the data in the first data, that is, a non-mandatory condition. In addition, it should be known that the preset check rules can be configured in an EXCEL table using an automated tool to quickly respond to the continuous update of the preset check rules. The Spark engine is a fast and general computing engine specifically designed for large-scale data processing to improve the efficiency of data verification. The quality inspection report is a visual result report generated by checking the first data based on the preset check rules.

[0044] Specifically, after the inspection device acquires the first data, the inspection device will use the Spark engine to perform a quality inspection on the first data based on the first condition that all data in the first data included in the preset inspection rule meet, and the second condition that a preset proportion of data in the first data meet, so as to generate a quality inspection report for the first data. In this way, the quality inspection of the first data can be performed by the preset inspection rule and the Spark engine, which can not only ensure the data quality of the first data, but also improve the efficiency of the data quality inspection.

[0045] In addition, in some embodiments, the preset verification report may include internal data conflict categories, such as uniqueness check; data missing categories, such as non-null check and completeness check; violation of coding specifications categories, such as date range check, time and date format check, special character check; violation of technical specifications categories, such as range code value check, numerical check, length check; business logic categories, such as numerical comparison check, total score check, etc., which are not limited here.

[0046] S130, when the data verification result included in the quality inspection report meets the second preset condition, control the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database reads and stores the first data from the formal area of ​​the dedicated network device.

[0047] The second preset condition is a condition pre-set based on actual experience and used to determine whether the quality inspection report is qualified. In the embodiment of the present application, the second preset condition is set based on a preset verification rule. The data verification result is the result of quality inspection of each data in the first data based on the preset verification rule.

[0048] Specifically, after generating the quality inspection report of the first data, when the inspection device determines that the data verification result included in the quality inspection report meets the second preset condition, that is, when the quality inspection report of the first data is qualified, the inspection device can control the dedicated network device to transfer the temporary area of ​​the dedicated network device of the first data to the formal area of ​​the dedicated network device by means of signals or instructions, so that the database can directly read and store the first data from the formal area of ​​the dedicated network device. In this way, the situation of low data quality caused by the first device or the inspection device changing the first data stored in the temporary area of ​​the dedicated network device is avoided, thereby improving the accuracy of the data and further alleviating the pressure on the database.

[0049] In one example, it is assumed that the first condition included in the preset verification rule is that the data date of the data is between December 12 and December 30, and the second condition included in the preset verification rule is that the transaction amount involved in 90% of the data is higher than X yuan. If the data verification result included in the quality inspection report generated by the inspection device shows that the data date of all the data involved in the first data is between December 12 and December 30, and the transaction amount involved in 90% of the data in the first data is higher than X yuan, it means that the data verification result involved in the quality inspection report of the first data meets the second preset condition, and then the dedicated network device can be controlled to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database can directly read and store the first data from the formal area of ​​the dedicated network device. It should be noted that the first and second conditions involved above are an example for more clearly describing an embodiment of the present application, and will not limit the substantive content of the first and second conditions.

[0050] The embodiment of the present application can obtain the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition, and perform a quality check on the first data based on the preset verification rules and the Spark engine to generate a quality inspection report, and when the data verification result included in the quality inspection report meets the second preset condition, control the dedicated network device to transfer the first data originally stored in the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, thereby facilitating the database to directly read and store the first data from the formal area of ​​the dedicated network device. In this way, not only the efficiency of data inspection can be improved, but also the pressure on the database can be effectively reduced.

[0051] In order to more comprehensively describe the data storage method provided in the embodiment of the present application, in one embodiment, the data storage method mentioned above may also include:

[0052] When the data verification result included in the quality inspection report does not meet the second preset condition, the first data and the quality inspection report are stored in a temporary area of ​​the dedicated network device, so that the first device can obtain the first data and the quality inspection report from the temporary area of ​​the dedicated network device to correct the first data according to the quality inspection report.

[0053] In this embodiment, when the data verification result of the quality inspection report of the first data does not meet the second preset condition, that is, it indicates that the quality inspection report of the first data is unqualified, the inspection device can store the first data and the quality inspection report in the temporary area of ​​the dedicated network device, so that the first device actively obtains the first data and the corresponding quality inspection report from the temporary area of ​​the dedicated network device, and then corrects the first data according to the quality inspection report. In this way, the verification result of the first data can be fed back to the first device, so as to timely correct the problems and risks in the first data, effectively reduce the risks, and facilitate the subsequent inspection of the corrected first data to ensure the quality of the data entered into the warehouse, thereby effectively reducing the pressure on the database.

[0054] In one embodiment, when the data verification result included in the quality inspection report does not meet the second preset condition, the first data and the verification log of the first data can be stored in a temporary area of ​​a dedicated network device, wherein the verification log includes the quality inspection report and the verification process of the first data, thereby facilitating the first device to read the first data and the verification log of the first data, and to correct the first data according to the verification report of the first data, so as to continue to check the corrected first data to ensure the quality of the data entered into the database, thereby effectively reducing the pressure on the database.

[0055] Based on the same inventive concept, the embodiment of the present application also provides a data storage device. Figure 4 Explanation

[0056] Figure 4 It is a structural schematic diagram of a data storage device provided in an embodiment of the present application.

[0057] like Figure 4 As shown, the data storage device 400 may include: an acquisition module 410 , a quality inspection module 420 and a control module 430 .

[0058] An acquisition module 410 is configured to acquire the first data if the first data stored in the temporary area of ​​the dedicated network device meets a first preset condition;

[0059] The quality inspection module 420 is used to perform a quality inspection on the first data using a Spark engine based on a preset inspection rule to generate a quality inspection report;

[0060] The control module 430 is used to control the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device when the data verification result included in the quality inspection report meets the second preset condition, so that the database reads and stores the first data from the formal area of ​​the dedicated network device.

[0061] In one embodiment, the above-mentioned data storage device may further include: a storage module.

[0062] A storage module is used to store the first data and the quality inspection report in a temporary area of ​​a dedicated network device when the data verification result included in the quality inspection report does not meet a second preset condition, so that the first device can obtain the first data and the quality inspection report from the temporary area of ​​the dedicated network device to correct the first data according to the quality inspection report.

[0063] In one embodiment, the first data stored in the temporary area of ​​the dedicated network device is data transmitted by the first device to the temporary area of ​​the dedicated network device through a file transfer protocol.

[0064] In one embodiment, the first preset condition is that the data size of the first data is consistent with the data size obtained by the dedicated network device from parsing the XML file corresponding to the first data.

[0065] In one embodiment, the preset checking rule includes that all data in the first data meets a first condition, and a preset proportion of data in the first data meets a second condition.

[0066] In one embodiment, the preset checking rules are configured in an EXCEL table using an automated generation tool.

[0067] The embodiment of the present application is to obtain the first data when the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition, and perform a quality check on the first data based on the preset verification rules and the Spark engine to generate a quality inspection report, and when the data verification result included in the quality inspection report meets the second preset condition, the first data originally stored in the temporary area of ​​the dedicated network device is transferred to the formal area of ​​the dedicated network device, thereby facilitating the database to directly read and store the first data from the formal area of ​​the dedicated network device. In this way, not only the efficiency of data inspection can be improved, but also the pressure on the database can be effectively reduced.

[0068] Each module in the data storage device provided in the embodiment of the present application can be implemented Figure 2 or Figure 3 The method steps of the illustrated embodiment can achieve the corresponding technical effects, and for the sake of brevity, they will not be repeated here.

[0069] Figure 5 It is a structural schematic diagram of a data storage device provided in an embodiment of the present application.

[0070] like Figure 5 As shown, the data storage device 500 in this embodiment includes an input device 501, an input interface 502, a central processing unit 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processing unit 503, the memory 504, and the output interface 505 are connected to each other through a bus 510, and the input device 501 and the output device 506 are connected to the bus 510 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the data storage device 500.

[0071] Specifically, the input device 501 receives input information from the outside and transmits the input information to the central processing unit 503 through the input interface 502; the central processing unit 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the data storage device 500 for user use.

[0072] In one embodiment, Figure 5 The data storage device 500 shown includes: a memory 504 for storing programs; a processor 503 for running the programs stored in the memory to execute the embodiments of the present application. Figure 2 or Figure 3 The method of the illustrated embodiment.

[0073] The present application also provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by a processor, the present application also provides a computer-readable storage medium on which computer program instructions are stored; Figure 2 or Figure 3 The method of the illustrated embodiment.

[0074] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0075] The functional blocks shown in the above structural block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0076] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0077] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data storage method, characterized in that: The method comprises: When the first data stored in the temporary area of ​​the dedicated network device meets the first preset condition, acquiring the first data; Based on preset checking rules, using the Spark engine to perform a quality check on the first data to generate a quality check report; When the data verification result included in the quality inspection report meets the second preset condition, control the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device, so that the database reads and stores the first data from the formal area of ​​the dedicated network device; The first preset condition is that the data size of the first data is consistent with the data size obtained by the dedicated network device from parsing the extensible markup language XML file corresponding to the first data; The preset check rule includes a first condition that all data in the first data meet, and a second condition that a preset proportion of data in the first data meet; The second preset condition includes a condition determined based on the preset checking rule and used to determine whether the quality inspection report is qualified.

2. The method according to claim 1, characterized in that: The method further comprises: When the data verification result included in the quality inspection report does not meet the second preset condition, the dedicated network device is controlled to store the first data and the quality inspection report in a temporary area of ​​the dedicated network device, so that the first device obtains the first data and the quality inspection report from the temporary area of ​​the dedicated network device to correct the first data according to the quality inspection report.

3. The method according to claim 1, characterized in that The first data stored in the temporary area of ​​the dedicated network device is data transmitted by the first device to the temporary area of ​​the dedicated network device through a file transfer protocol.

4. The method according to claim 1, characterized in that: The preset checking rules are configured in an EXCEL table using an automated generation tool.

5. A data storage device, characterized in that: The device also includes: An acquisition module, configured to acquire the first data if the first data stored in the temporary area of ​​the dedicated network device meets a first preset condition; A quality inspection module, configured to perform a quality inspection on the first data using a Spark engine based on preset inspection rules to generate a quality inspection report; A transfer module, used for controlling the dedicated network device to transfer the first data from the temporary area of ​​the dedicated network device to the formal area of ​​the dedicated network device when the data verification result included in the quality inspection report meets the second preset condition, so that the database reads and stores the first data from the formal area of ​​the dedicated network device; The first preset condition is that the data size of the first data is consistent with the data size obtained by the dedicated network device from parsing the extensible markup language XML file corresponding to the first data; The preset check rule includes a first condition that all data in the first data meet, and a second condition that a preset proportion of data in the first data meet; The second preset condition includes a condition determined based on the preset checking rule and used to determine whether the quality inspection report is qualified.

6. An information processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the data warehousing method as described in any one of claims 1-4.

7. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data warehousing method according to any one of claims 1 to 4 is implemented.

8. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data warehousing method as described in any one of claims 1 to 4.

Images