Test data deployment method, device, equipment, readable storage medium and product

Abstract

This disclosure provides a test data deployment method, apparatus, device, readable storage medium, and product. The method includes: obtaining a test data deployment request, wherein the test data deployment request includes the machine IP address to which the test data is to be deployed, the database type, the data suffix, and the storage directory; according to the test data deployment request, simulating a distributed shared file system pre-stored with the test data as the local hard drive of the database to which the test data belongs; and according to the database type, selecting a target data import method corresponding to the data type to perform a data import operation on the test data in the local hard drive storage directory. This enables automatic deployment of various types of database data while reducing data transmission and improving data transmission efficiency, without requiring users to have high levels of professional skills, and has strong applicability.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more particularly to a test data deployment method, apparatus, device, readable storage medium, and product. Background Technology

[0002] Due to continuous technological innovation and historical reasons, banks often use multiple databases to store data in their production environments, such as Oracle, Sybase, DB2, Gbase, and MySQL. Simultaneously, multiple types of database testing environments also need to be built concurrently in the testing environment.

[0003] Existing test data deployment methods typically involve developers first obtaining the test data to the database server using commands such as scp, then gaining server privileges, logging into the database server, and manually executing deployment commands or scripts.

[0004] However, since test data needs to be stored on the database server, there are security risks. Furthermore, data transfer of large amounts of data is time-consuming and inefficient. In addition, deployment operations often vary significantly due to different database types, and the complex steps and numerous parameters make deployment skills difficult to master, especially for new employees. Mastering these skills requires continuous learning and practice, which is time-consuming. Summary of the Invention

[0005] This disclosure provides a test data deployment method, apparatus, device, readable storage medium, and product to solve the technical problems of existing data deployment operations requiring high user expertise and involving long data transmission times and low efficiency.

[0006] The first aspect of this disclosure is to provide a method for deploying test data, including:

[0007] Obtain a test data deployment request, wherein the test data deployment request includes the machine IP address to which the test data is to be deployed, the database type, the data suffix, and the storage directory;

[0008] Based on the test data deployment request, the distributed shared file system containing the pre-stored test data is simulated as the local hard drive of the database to which the test data belongs;

[0009] Based on the database type, select the target data import method corresponding to the data type to import the test data in the local hard disk storage directory.

[0010] A second aspect of this disclosure is to provide a test data deployment apparatus, comprising:

[0011] The acquisition module is used to acquire test data deployment requests, wherein the test data deployment requests include the machine IP address to which the test data is to be deployed, the database type, the data suffix, and the storage directory;

[0012] The simulation module is used to simulate the distributed shared file system containing the test data as the local hard disk of the database to which the test data belongs, according to the test data deployment request.

[0013] The transmission module is used to select the target data import method corresponding to the data type according to the database type and perform data import operation on the test data in the local hard disk storage directory.

[0014] A third aspect of this disclosure is to provide an electronic device, including: a memory and a processor;

[0015] Memory; memory for storing executable instructions of the processor;

[0016] The processor is used to invoke program instructions in the memory to execute the test data deployment method as described in the first aspect.

[0017] A fourth aspect of this disclosure is to provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the test data deployment method as described in the first aspect.

[0018] A fifth aspect of this disclosure is to provide a computer program product, including a computer program that, when executed by a processor, implements the test data deployment method as described in the first aspect.

[0019] The test data deployment method, apparatus, equipment, readable storage medium, and product disclosed herein simulate the shared SAN storage containing the test data as a local hard disk of the database after receiving a data deployment request, and adopts corresponding data import methods according to different database types. This enables automatic deployment of various types of database data while reducing data transmission and improving data transmission efficiency. It does not require users to have high professional skills and has strong applicability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0021] Figure 1A flowchart illustrating the test data deployment method provided in Embodiment 1 of this disclosure;

[0022] Figure 2 A flowchart illustrating the test data deployment method provided in Embodiment 2 of this disclosure;

[0023] Figure 3 A flowchart illustrating the test data deployment method provided in Embodiment 3 of this disclosure;

[0024] Figure 4 This is a schematic diagram of the test data deployment device provided in Embodiment 4 of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained based on the embodiments of this disclosure are within the scope of protection of this disclosure.

[0027] Definitions:

[0028] Desensitization refers to the process of deleting, replacing, or simulating data containing sensitive information.

[0029] Test data refers to production data that has been anonymized and used in the development and testing process. Addressing the aforementioned technical problems of existing data deployment operations requiring high levels of user expertise, long data transmission times, and low efficiency, this disclosure provides a test data deployment method, apparatus, equipment, readable storage medium, and product.

[0030] Mounting is a process by which the operating system makes computer files and directories on a storage device (such as a hard drive, CD-ROM, or shared resource) accessible to users through the computer's file system.

[0031] It should be noted that the test data deployment methods, apparatus, equipment, readable storage media, and products provided in this disclosure can be used in data deployment scenarios with various types of databases.

[0032] With continuous technological innovation and historical reasons, banks often use multiple databases to store data in their production environments, while simultaneously building various types of database testing environments. Due to security access requirements and personnel skill limitations, developers are often unable to efficiently and independently import test data into the databases. Currently, data deployment is handled by the project team after test data delivery. During deployment, some developers and testers, unfamiliar with the process or lacking sufficient permissions, experience high communication costs and long deployment times, extending the test preparation cycle and consequently impacting the testing schedule.

[0033] In the process of solving the above-mentioned technical problems, the inventors discovered through research that the de-identified data can be stored in a shared SAN storage. The SAN storage can be directly mounted to the database server as a local disk using disk mounting technology. Then, the database commands can be executed in the simulation of the server's operation, enabling test data import operations on file types supported by the database, such as text files, dmp files, and bcp files.

[0034] Figure 1 This is a flowchart illustrating the test data deployment method provided in Embodiment 1 of this disclosure, as follows: Figure 1 As shown, the method includes:

[0035] Step 101: Obtain test data deployment request, wherein the test data deployment request includes the machine IP address to which the test data is to be deployed, the database type, the data suffix, and the storage directory.

[0036] The execution entity in this embodiment is a test data deployment device, which can be coupled to a server, and the server can communicate and connect with multiple databases.

[0037] In this embodiment, the anonymized data can be pre-stored in a distributed shared file system. Researchers can determine the data deployment information based on actual needs and issue a test data deployment request. Correspondingly, the test data deployment device can obtain this request, which includes the machine IP address where the test data is to be deployed, the database type, the data suffix, and the storage directory.

[0038] Step 102: Based on the test data deployment request, simulate the distributed shared file system containing the pre-stored test data as the local hard disk of the database to which the test data belongs.

[0039] In this embodiment, in order to reduce data transmission, the distributed shared file system containing the test data can be simulated as the local hard disk of the database to which the test data belongs, according to the test data deployment request.

[0040] Specifically, this distributed shared file system can be a shared SAN storage.

[0041] Step 103: Based on the database type, select the target data import method corresponding to the data type to import the test data in the local hard disk storage directory.

[0042] In this embodiment, since there are many types of databases, the correspondence between database types and import methods can be pre-configured to enable data transfer operations across multiple databases. Therefore, after simulating the distributed shared file system as the local hard drive of the database to which the test data belongs, the database can be logged into, and the test data in the local hard drive storage directory can be imported using the target data import method corresponding to the data type, based on the database type.

[0043] Furthermore, based on Embodiment 1, step 102 includes:

[0044] The distributed shared file system containing the test data is simulated as the local hard drive of the database to which the test data belongs by means of disk mounting technology.

[0045] In this embodiment, disk mounting can be used to simulate operations on the local hard drive of the database. Specifically, disk mounting technology can be used to simulate a distributed shared file system containing the test data as the local hard drive of the database to which the test data belongs.

[0046] Furthermore, based on Embodiment 1, after step 103, the following steps are also included:

[0047] Perform an unload operation on the distributed shared file system.

[0048] In this embodiment, after completing the deployment of the current database test data, in order to ensure that test data for other databases can be deployed, the distributed shared file system can be unloaded.

[0049] The test data deployment method provided in this embodiment simulates the shared SAN storage where the test data is located as a local hard disk of the database after obtaining the data deployment request, and adopts the corresponding data import method according to different database types. In this way, it can realize the automatic deployment of data of various types of databases while reducing data transmission and improving data transmission efficiency. It does not require users to have high professional skills and has strong applicability.

[0050] Figure 2 This is a flowchart illustrating the test data deployment method provided in Embodiment 2 of this disclosure. Based on Embodiment 1, as follows... Figure 2 As shown, step 103 includes:

[0051] Step 201: Based on the database type, determine at least one data import method corresponding to the database type.

[0052] Step 202: Determine the data import method corresponding to the data suffix among the at least one data import methods as the target data import method.

[0053] In this embodiment, a database type can correspond to at least one data import method. To accurately determine the data import method, it is also necessary to further determine the data suffix of the test data. Specifically, at least one data import method corresponding to the database type can be determined based on the database type. The data import method corresponding to the data suffix among the at least one data import method is determined as the target data import method.

[0054] For example, in practical applications, when the database type is Oracle, there are two import methods: IMTDT and SQLLoad. Furthermore, if the data extension indicates that the test data is a text file, the SQLLoad method can be used for data deployment; if the data extension indicates that the test data is a database file, the IMTDT method can be used.

[0055] The test data deployment method provided in this embodiment determines the target data import method based on the database type and data suffix, thereby enabling precise data deployment operations for different data in different database types.

[0056] Figure 3 This is a flowchart illustrating the test data deployment method provided in Embodiment 3 of this disclosure. Based on any of the above embodiments, such as... Figure 3 As shown, before step 101, the following steps are also included:

[0057] Step 301: Obtain the production data to be processed.

[0058] Step 302: De-identify the production data to be processed to obtain the test data.

[0059] In this embodiment, before data deployment, it is necessary to obtain anonymized test data. Specifically, the production data to be processed can be obtained. The production data to be processed is then anonymized to obtain the test data.

[0060] Furthermore, based on any of the above embodiments, step 302 specifically includes:

[0061] Determine the data type corresponding to the production data to be processed.

[0062] The production data to be processed is desensitized using a desensitization method corresponding to the data type to obtain the test data.

[0063] In this embodiment, the production data may include various different data types. For example, the production data may include user identification data, user communication method data, user address data, etc. To perform de-identification on the production data, it is first necessary to determine the data type of the production data. For different data types, a de-identification method corresponding to the data type is used to perform de-identification on the production data to be processed, thereby obtaining test data.

[0064] For example, de-identification operations include, but are not limited to, hiding some bytes, replacing some bytes, and deleting some bytes.

[0065] The test data deployment method provided in this embodiment uses a desensitization method corresponding to different data types to perform desensitization operations on the production data to be processed, thereby obtaining test data. This enables targeted and precise desensitization operations on production data of different data types, providing a foundation for subsequent data testing.

[0066] Figure 4 This is a schematic diagram of the test data deployment device provided in Embodiment 4 of this disclosure, as shown below. Figure 4 As shown, the device includes: an acquisition module 41, a simulation module 42, and a transmission module 43. The acquisition module 41 is used to acquire a test data deployment request, wherein the test data deployment request includes the IP address of the machine to which the test data is to be deployed, the database type, the data suffix, and the storage directory. The simulation module 42 is used to simulate a distributed shared file system containing the pre-stored test data as the local hard drive of the database to which the test data belongs, according to the test data deployment request. The transmission module 43 is used to select a target data import method corresponding to the data type to perform a data import operation on the test data in the local hard drive storage directory, based on the database type.

[0067] Furthermore, based on Embodiment 4, the simulation module is used to: simulate the distributed shared file system containing the test data as the local hard disk of the database to which the test data belongs through disk mounting technology.

[0068] Furthermore, based on Embodiment 4, the device further includes an unloading module for performing an unloading operation on the distributed shared file system.

[0069] Furthermore, based on Embodiment 4, the transmission module is configured to: determine at least one data import method corresponding to the database type according to the database type; and determine the data import method corresponding to the data suffix among the at least one data import methods as the target data import method.

[0070] Furthermore, based on any of the above embodiments, the device further includes: an acquisition module, which is also used to acquire production data to be processed; and a processing module, which is used to perform a desensitization operation on the production data to be processed to obtain the test data.

[0071] Furthermore, based on any of the above embodiments, the processing module is used to: determine the data type corresponding to the production data to be processed; and perform a desensitization operation on the production data to be processed using a desensitization method corresponding to the data type to obtain the test data.

[0072] Another embodiment of this disclosure also provides an electronic device, including: a memory and a processor;

[0073] Memory; memory for storing executable instructions of the processor;

[0074] The processor is used to invoke program instructions in the memory to execute the test data deployment method as described in any of the above embodiments.

[0075] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0076] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0077] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0078] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0079] Another embodiment of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the test data deployment method as described in any of the above embodiments.

[0080] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0081] Another embodiment of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the test data deployment method as described in any of the above embodiments.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0083] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method of testing data deployment, characterized by, The method comprises the following steps: obtaining a test data deployment request, wherein the test data deployment request comprises a machine IP address to which the test data is to be deployed, a database type to which the test data belongs, a data suffix, and a storage directory; according to the test data deployment request, simulating a distributed shared file system pre-stored with desensitized test data into a local hard disk of a database to which the test data belongs through a disk mounting technology; the distributed shared file system is a SAN storage; according to the database type, selecting a target data import mode corresponding to the data type to perform a data import operation on the test data in the local hard disk storage directory, wherein the target data import mode is determined according to the database type and the data suffix; the step of selecting a target data import mode corresponding to the data type to perform a data import operation on the test data in the local hard disk according to the database type comprises the following steps: determining at least one data import mode corresponding to the database type according to the database type; determining a data import mode corresponding to the data suffix in the at least one data import mode as the target data import mode; when the database type is an Oracle database and the data suffix represents that the test data is text file data, performing a data deployment operation by using a sqlload import mode; when the database type is an Oracle database and the data suffix represents that the test data is database file data, performing a data deployment operation by using an imtdt import mode.

2. The method of claim 1, wherein, after the step of selecting a target data import mode corresponding to the data type to perform a data import operation on the test data in the local hard disk storage directory according to the database type, the method further comprises the following step: performing an uninstall operation on the distributed shared file system.

3. The method according to any of claims 1-2, characterized in that, before the step of obtaining a test data deployment request, the method further comprises the following step: obtaining production data to be processed; performing a desensitization operation on the production data to be processed to obtain the test data.

4. The method of claim 3, wherein, the step of performing a desensitization operation on the production data to be processed to obtain the test data comprises the following steps: determining a data type corresponding to the production data to be processed; performing a desensitization operation on the production data to be processed by using a desensitization mode corresponding to the data type to obtain the test data.

5. A test data deployment apparatus, characterized by, The method comprises the following steps: an obtaining module is configured to obtain a test data deployment request, wherein the test data deployment request comprises a machine IP address to which the test data is to be deployed, a database type to which the test data belongs, a data suffix, and a storage directory; a simulation module is configured to simulate a distributed shared file system pre-stored with desensitized test data into a local hard disk of a database to which the test data belongs through a disk mounting technology according to the test data deployment request; the distributed shared file system is a SAN storage; a transmission module is configured to select a target data import mode corresponding to the data type to perform a data import operation on the test data in the local hard disk storage directory according to the database type, wherein the target data import mode is determined according to the database type and the data suffix. The transmission module is specifically configured to determine at least one data import mode corresponding to the database type according to the database type; The data import mode corresponding to the data suffix in the at least one data import mode is determined as the target data import mode; When the database type is an Oracle database and the data suffix represents that the test data is text file data, a sqlload import mode is used for data deployment operation; when the database type is an Oracle database and the data suffix represents that the test data is a database file, an imtdt import mode is used for data deployment operation.

6. An electronic device, comprising: Comprise: Memory, processor; Memory; Memory for storing the processor executable instructions; The processor is used to call the program instruction in the memory to execute the test data deployment method in any one of claims 1-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the test data deployment method in any one of claims 1-4.

8. A computer program product, characterised in that, The computer program is executed by the processor to implement the test data deployment method in any one of claims 1-4.

Images