Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method relates to the technical field of financial science and technology, and comprises the following steps: obtaining a to-be-migrated target data range in a first database, and migrating first data information from the first database to a target platform according to the target data range; checking the first data information based on a preset data rule through the target platform to obtain a target checking result, and sending the first data information to a target server through the target platform after the target checking result represents that the checking is passed; and reading the first data information from the target server through a second database to complete data migration. Through the method and the device, the problem of relatively poor heterogeneous data migration safety in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, in particular to a data processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] At present, data migration technology mostly relies on manual script writing or the use of general data migration tools, but these methods are often difficult to deal with complex data structures and massive data volumes, especially in the case of significant differences between data sources and target systems, the consistency of data is difficult to guarantee, and the data security protection measures in the migration process are insufficient. In the face of the challenge of heterogeneous data migration, the existing technology generally lacks effective solutions for data security. During the migration process, data often needs to be processed and converted through multiple links, and each link may become a weak point of data security.

[0003] In view of the poor security of heterogeneous data migration in the related art, no effective solution has been proposed so far. SUMMARY

[0004] The main purpose of the present application is to provide a data processing method and device, a storage medium and an electronic device to solve the problem of poor security of heterogeneous data migration in the related art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method comprises: obtaining a target data range to be migrated in a first database, and migrating first data information from the first database to a target platform according to the target data range; verifying the first data information based on a preset data rule through the target platform to obtain a target verification result, and sending the first data information to a target server through the target platform after the target verification result represents that the verification is passed; reading the first data information from the target server through a second database to complete data migration.

[0006] Further, obtaining the target data range to be migrated in the first database comprises: determining a target business requirement, and determining a target data field necessary for the second database based on the target business requirement; determining the target data range according to the target data field.

[0007] Further, the migrating the first data information from the first database to the target platform according to the target data range comprises: extracting the first data information from the first database by a data pump tool to obtain the first data information in a first data format; obtaining a second data format corresponding to the target platform, and performing format conversion on the first data information in the first data format to obtain the first data information in the second data format; and migrating the first data information in the second data format to the target platform.

[0008] Further, the verifying the first data information based on the preset data rule by the target platform to obtain a target verification result comprises: verifying a total number of data of the first data information to obtain a first verification result; verifying data content of the first data information to obtain a second verification result; verifying data integrity of the first data information to obtain a third verification result; performing abnormal data detection on the first data information to obtain a fourth verification result; and obtaining the target verification result according to the first verification result, the second verification result, the third verification result and the fourth verification result.

[0009] Further, the verifying the data content of the first data information to obtain the second verification result comprises: obtaining a plurality of field information corresponding to the first data information; constructing a target structured query language according to the plurality of field information; and verifying field values corresponding to the plurality of field information according to the target structured query language to obtain the second verification result.

[0010] Further, the verifying the data integrity of the first data information to obtain the third verification result comprises: obtaining a primary key value in the first data information and a foreign key value in the first data information; and verifying the data integrity of the first data information according to the primary key value and the foreign key value to obtain the third verification result.

[0011] Further, the reading the first data information from the target server by the second database to complete the data migration comprises: loading the first data information from the target server to a temporary data table by the second database; performing data mapping on the first data information in the temporary data table according to a data structure of the second database to obtain second data information; and migrating the second data information to a business data table of the second database to complete the data migration.

[0012] In order to achieve the above object, according to another aspect of the present application, a data processing device is provided. The device comprises: an acquisition unit configured to acquire a target data range to be migrated in a first database, and migrate first data information from the first database to a target platform according to the target data range; a verification unit configured to verify the first data information based on a preset data rule by the target platform, obtain a target verification result, and send the first data information to a target server by the target platform when the target verification result indicates that the verification is passed; and a reading unit configured to read the first data information from the target server by a second database, so as to complete data migration.

[0013] Further, the acquisition unit comprises: a first determination module configured to determine a target business requirement, and determine a target data field necessary for the second database based on the target business requirement; and a second determination module configured to determine the target data range according to the target data field.

[0014] Further, the acquisition unit comprises: an extraction module configured to extract the first data information from the first database by a data pump tool, and obtain first data information in a first data format; an acquisition module configured to acquire a second data format corresponding to the target platform, and convert the first data information in the first data format to obtain first data information in a second data format; and a migration module configured to migrate the first data information in the second data format to the target platform.

[0015] Further, the verification unit comprises: a first verification module configured to verify a total number of data of the first data information, and obtain a first verification result; a second verification module configured to verify data content of the first data information, and obtain a second verification result; a third verification module configured to verify data integrity of the first data information, and obtain a third verification result; a fourth verification module configured to detect abnormal data of the first data information, and obtain a fourth verification result; and a processing module configured to obtain the target verification result according to the first verification result, the second verification result, the third verification result and the fourth verification result.

[0016] Further, the second verification module comprises: a first acquisition submodule configured to acquire a plurality of field information corresponding to the first data information; a construction submodule configured to construct a target structured query language according to the plurality of field information; and a first verification submodule configured to verify field values corresponding to the plurality of field information according to the target structured query language, and obtain the second verification result.

[0017] Further, the third checking module comprises: a second acquisition submodule, configured to acquire a primary key value in the first data information and a foreign key value in the first data information; and a second checking submodule, configured to check data integrity of the first data information according to the primary key value and the foreign key value, to obtain the third checking result.

[0018] Further, the reading unit comprises: a loading module, configured to load the first data information from the target server to a temporary data table through the second database; a mapping module, configured to perform data mapping on the first data information in the temporary data table according to a data structure of the second database, to obtain second data information; and a migration module, configured to migrate the second data information to a business data table of the second database, to complete data migration.

[0019] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a memory storing an executable program; and a processor configured to run the program, wherein the program, when running, performs any one of the data processing methods.

[0020] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which stores a program, wherein the program, when running, controls a device where the storage medium is located to perform any one of the data processing methods.

[0021] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising a computer program or instructions, which, when executed by a processor, implement any one of the data processing methods.

[0022] In the embodiments of the present application, the following steps are adopted: acquiring a target data range to be migrated in a first database, and migrating first data information from the first database to a target platform according to the target data range; performing checking on the first data information based on a preset data rule through the target platform, to obtain a target checking result, and sending the first data information to a target server through the target platform when the target checking result indicates that the checking is passed; and reading the first data information from the target server through a second database, to complete data migration, thereby solving the technical problem of poor security of heterogeneous data migration in the related art.

[0023] In the scheme, the target data range to be migrated in the first database is accurately acquired, the blindness in the data migration process is avoided, unnecessary data transmission is reduced, and the risk of data leakage is reduced. According to the target data range, data is extracted from the first database and preliminarily processed, and data inspection is performed with the aid of the target platform, so that data problems can be quickly identified and located, and corrective measures can be taken in time to avoid security risks caused by data quality problems. After the target inspection result shows that the data meets the expected standard, the first data information is safely transmitted to the target server, and finally the second database reads the first data information from the target server to complete the data migration, the target data range is accurately acquired, the data amount in the transmission process is reduced, and the risk of data leakage is reduced. The data inspection is performed on the target platform, which effectively avoids subsequent security risks caused by data quality problems, and thus the technical effects of improving the security and efficiency of heterogeneous data migration are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and are not intended to limit the present application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown;

[0026] Figure 2 A flowchart of a data processing method according to an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram of a data processing device according to an embodiment of the present application is shown;

[0028] Figure 4 A structure block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the collected information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and related users or institutions are provided with an interface to provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0032] Embodiment 1

[0033] According to the embodiments of the present application, a method for data processing is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0034] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the data processing method is shown. As Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0038] The display can be a touch screen liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] In the above operating environment, the present application provides a data processing method as shown in Figure 2 Figure 2 is a flowchart of the data processing method according to an embodiment of the present application. The data processing method includes:

[0040] In step S201, a target data range to be migrated in the first database is obtained, and first data information is migrated from the first database to the target platform according to the target data range.

[0041] Optionally, through business requirement analysis, it is determined which data are key data required by the business migration, including but not limited to customer information, transaction records, account data, etc., i.e., the target data range to be migrated in the first database is determined. According to the determined target data range, the required data is extracted from the first database to obtain the first data information.

[0042] In an optional embodiment, data extraction can be performed through a data extraction tool to ensure the integrity and consistency of the data.

[0043] In an optional embodiment, the data can also be preliminarily processed, such as data cleaning, data conversion and data desensitization, to adapt to the data format and security requirements of the target platform.

[0044] In step S202, the first data information is verified by the target platform based on a preset data rule to obtain a target verification result, and the first data information is sent to the target server by the target platform when the target verification result indicates that the verification is passed.

[0045] ​Optionally, before data migration begins, data rules can be preset based on business needs and the characteristics of the target system. Data rules can include aspects such as data integrity, data consistency, data format, and data range. For example, checking whether the data is missing key fields, whether it meets the expected numerical range, and whether it is consistent with business logic.

[0046] The target platform's data processing engine can be used to verify the initial data information extracted from the first database according to data rules, thereby obtaining the target verification result. The parallel processing capability of the data platform can significantly improve the speed of data verification, enabling the verification work to be completed in a short time even when dealing with large-scale data.

[0047] After the data verification is completed, a target verification result is generated, which includes whether the data meets each rule. If the data fully complies with all preset rules, i.e., the target verification result indicates that the data verification has passed, the next step of data migration can proceed; otherwise, if the verification result finds data problems, the abnormal data is marked for subsequent data cleaning or repair.

[0048] After the target verification result characterization test is passed, the target platform will transmit the first data information of the passed test to the target server through a secure communication mechanism. In order to protect data security, the transmission can be encrypted to ensure that the data is not intercepted or tampered with during the transmission process.

[0049] Step S203: Read the first data information from the target server through the second database to complete the data migration.

[0050] Optionally, the second database reads the first data information from the target server to complete the data migration.

[0051] In an optional embodiment, since the first database and the second database may have differences in data structure, data structure adaptation work can be performed when importing data, such as field type conversion, data format adjustment, etc., to ensure that the data can be correctly stored and used in the target database.

[0052] After the data is imported into the second database, data verification and checking can be performed to confirm the integrity and accuracy of the data migration. This typically includes data volume verification, data content checking, and data relationship verification to ensure that all migrated data meets business requirements and the structural requirements of the target database.

[0053] In summary, accurately identifying the target data range to be migrated from the first database avoids blind migration, reduces unnecessary data transmission, and lowers the risk of data leakage. Based on the target data range, data is extracted from the first database and preliminarily processed. Data verification using the target platform quickly identifies and locates data problems, allowing for timely corrective measures and preventing security risks caused by data quality issues. After the target verification results indicate that the data meets the expected standards, the first data information is securely transmitted to the target server. Finally, the second database reads the first data information from the target server to complete the data migration. Accurately identifying the target data range reduces the amount of data transmitted, thereby lowering the risk of data leakage. Data verification is performed on the target platform, effectively avoiding subsequent security risks caused by data quality issues, thus achieving the technical effect of improving the security and efficiency of heterogeneous data migration.

[0054] Optionally, in the data processing method provided in this application embodiment, obtaining the target data range to be migrated in the first database includes: determining the target business requirements, and determining the target data fields necessary for the second database based on the target business requirements; and determining the target data range based on the target data fields.

[0055] In an optional embodiment, target business requirements are obtained and analyzed, including analyzing business processes, identifying key business metrics, and evaluating data usage scenarios. The aim is to ensure that data migration not only meets current business needs but also fully considers the possibilities for future business development. For example, if the business requirements involve real-time analysis of customer data, then historical transaction records and basic customer information become critical data that must be migrated.

[0056] Once the business requirements are understood, it's necessary to refine the analysis to the specific database field level, determining the minimum set of data fields required for the second database to function. Eliminate unnecessary fields and focus on key fields that directly impact the business. For example, if the business requirement is to generate sales reports, then fields such as product ID, sales date, sales quantity, and sales location might need to be considered, while certain auxiliary fields used for internal management may not require migration.

[0057] Finally, determine the target data range based on the target data fields, for example, by filtering, sorting, and extracting data using query statements.

[0058] Through the above steps, this solution ensures the targeted and efficient nature of data migration, avoiding the waste of resources and potential risks caused by blindly migrating large amounts of irrelevant data.

[0059] Optionally, in the data processing method provided in this application embodiment, migrating first data information from a first database to a target platform according to the target data range includes: extracting first data information from the first database using a data pump tool to obtain first data information in a first data format; obtaining a second data format corresponding to the target platform, and converting the first data information in the first data format to obtain first data information in the second data format; and migrating the first data information in the second data format to the target platform.

[0060] In an optional embodiment, first data information is extracted from a first database using a data pump tool. Data pump is a high-efficiency data transfer tool used to export data, objects, and SQL scripts into a specific binary format file (DMP file, i.e., the first data information in the aforementioned first data format). At this stage, based on the previously determined target data range, the data pump tool performs the export operation, extracting the first data information that is relevant to business needs and has been filtered from the source database. The advantage of data pump is that it can preserve all metadata and object dependencies, ensuring data integrity and consistency.

[0061] Target platforms typically use data storage formats different from other databases. Therefore, it's necessary to determine the second data format appropriate for the target platform and convert data in DMP format or other first data formats to the second data format accepted by the target platform. After the data format conversion is complete, the converted first data information can be uploaded to the target platform.

[0062] Through the above steps, the data migration process accurately extracted the target data information from the first database and ensured that the data could be successfully imported into the target platform through format conversion, thereby improving the success rate and speed of data migration.

[0063] Optionally, in the data processing method provided in this application embodiment, the target platform performs inspection on the first data information based on preset data rules to obtain the target inspection result, including: inspecting the total number of data entries in the first data information to obtain a first inspection result; inspecting the data content of the first data information to obtain a second inspection result; inspecting the data integrity of the first data information to obtain a third inspection result; performing abnormal data detection on the first data information to obtain a fourth inspection result; and obtaining the target inspection result based on the first inspection result, the second inspection result, the third inspection result, and the fourth inspection result.

[0064] In an optional embodiment, the total number of data entries in the first data information is checked, i.e., to verify whether the data volume is consistent before and after the data migration. By querying the total number of data entries within the target data range in the source database and comparing it with the number of data entries in the first data information on the target platform, it can be verified whether the data has been migrated to the target platform completely. If the total number of data entries matches, the first check result is passed; otherwise, further investigation is needed to determine the cause of data loss.

[0065] In an optional embodiment, the content of the first data information is verified to ensure that the migrated data is completely consistent with the source data in content. The second verification result ensures the accuracy of the data and the consistency of the business logic.

[0066] In an optional embodiment, the data integrity of the first data information is checked, i.e., whether the data structure conforms to the integrity constraints of the target platform or target database, such as checking whether primary keys are duplicated and whether foreign key references are valid. This check is crucial for maintaining the relationships between data and the validity of the data. If the data integrity check passes, the third check is considered passed.

[0067] In an optional embodiment, the first data information undergoes anomaly detection, i.e., identifying errors or unexpected data values, such as null values, out-of-time values, and format errors. Through preset rules and algorithms, the target platform can automatically identify and mark anomaly data, providing guidance for subsequent data cleaning and repair. If the anomaly detection results indicate that the data meets preset quality standards, the fourth verification result is considered passed.

[0068] Finally, the results of the four tests are comprehensively evaluated. Only when all test results are normal can the data migration be considered to have met the quality standards, and the target test result is considered passed. If any test result reveals a data problem, corresponding data cleaning and repair measures need to be taken according to the specific problem type to ensure the accuracy and integrity of the data.

[0069] By implementing the verification process, we ensured the rigor of the data migration process and the high standards of data quality, providing a reliable data foundation for subsequent data integration, analysis, and business applications.

[0070] Optionally, in the data processing method provided in this application embodiment, verifying the data content of the first data information to obtain a second verification result includes: obtaining multiple field information corresponding to the first data information; constructing a target structured query statement based on the multiple field information; and verifying the field values ​​corresponding to the multiple field information based on the target structured query statement to obtain a second verification result.

[0071] In an optional embodiment, after the data is migrated to the target platform, it is first necessary to obtain the field information from all the initial data information to be verified. The field information may include attributes such as field name, field type, and field length, as well as field value rules related to business logic, such as whether the field allows null values, the value range of numeric fields, and the format of date fields.

[0072] Based on the obtained field information, a structured query statement is constructed to validate the data content. The structured query statement can be designed with corresponding validation rules based on the characteristics of different fields. For example, for fields containing key business logic, a complex logical query statement may need to be constructed to check whether the field values ​​meet business requirements; for fields that do not allow null values, a query statement needs to be constructed to check for the existence of NULL values.

[0073] The first data information on the target platform is examined by constructing a structured query statement. Each statement examines whether the field values ​​of a specific field or set of fields meet the expected rules and standards, thus obtaining a second inspection result. If all fields pass the inspection, the second inspection result will indicate that the data content inspection is successful; otherwise, if some field values ​​are found to be inconsistent with the rules, the second inspection result needs to include specific abnormal records so that subsequent data cleaning or repair work can be carried out.

[0074] By conducting verification procedures, the accuracy and compliance of data content can be ensured after data migration to the target platform. Data issues can be identified and addressed in a timely manner, avoiding the impact of data inconsistencies or errors on business operations.

[0075] Optionally, in the data processing method provided in this application embodiment, verifying the data integrity of the first data information to obtain a third verification result includes: obtaining the primary key value and the foreign key value in the first data information; verifying the data integrity of the first data information based on the primary key value and the foreign key value to obtain a third verification result.

[0076] In an optional embodiment, the primary key value and foreign key value in the first data information are obtained. A primary key is a field or combination of fields that uniquely identifies a row of records in a table, while a foreign key is a field used to link two or more tables, typically a reference to the primary key of another table. The data integrity of the first data information is checked based on the primary key and foreign key values ​​to verify whether the primary key values ​​of each data table are unique and without duplication. The uniqueness of the primary key is a fundamental requirement for data integrity. It is also verified that all foreign key values ​​in the first data information can be found in the corresponding primary key values ​​of the linked data tables to ensure the correctness of the reference relationships between the data.

[0077] Data integrity checks can promptly identify and resolve potential problems during data migration, thereby preventing business interruptions and data inconsistencies caused by data issues and ensuring business continuity and data consistency.

[0078] Optionally, in the data processing method provided in this application embodiment, reading first data information from the target server through the second database to complete data migration includes: loading the first data information from the target server into a temporary data table through the second database; mapping the first data information in the temporary data table according to the data structure of the second database to obtain second data information; and migrating the second data information to the business data table of the second database to complete the data migration.

[0079] In an optional embodiment, the verified and transformed initial data information typically exists as files on the target server. The second database needs to read these files from the target server's file system and load the data into a pre-created temporary data table. The temporary data table serves as an intermediate buffer for data import and transformation, facilitating subsequent data matching and mapping.

[0080] After loading the first data information into a temporary data table, data mapping is performed on the first data information in the temporary data table according to the data structure of the second database. Data mapping is the process of adjusting and matching the first data information in the temporary data table according to the table structure of the second database. This includes mapping field names, converting field types, and any possible business logic conversions. The purpose of data mapping is to ensure that the first data information can be accurately adapted to the structure of the second database, generating second data information that meets the requirements of the new environment.

[0081] After data mapping is complete, the second set of data can be formally migrated to the business data tables in the second database, replacing or supplementing existing data. This ensures the atomicity and consistency of the data migration. The migration of data to the business data tables marks the completion of the data migration process, and the new system can begin using the migrated data for business processing.

[0082] Through data loading, mapping, and migration steps, the first data information was safely and accurately migrated to the business data tables of the second database, achieving seamless data transfer and ensuring business continuity and data availability.

[0083] The data processing method provided in this application embodiment obtains the target data range to be migrated from the first database, and migrates the first data information from the first database to the target platform according to the target data range; the target platform verifies the first data information based on preset data rules to obtain the target verification result, and after the target verification result indicates that the verification is passed, the target platform sends the first data information to the target server; the first data information is read from the target server through the second database to complete the data migration, which solves the technical problem of poor security in heterogeneous data migration in related technologies.

[0084] In this solution, accurately identifying the target data range to be migrated from the first database avoids blind migration, reduces unnecessary data transmission, and lowers the risk of data leakage. Based on the target data range, data is extracted from the first database and preliminarily processed. Data verification is performed using the target platform, enabling rapid identification and location of data problems, allowing for timely corrective measures and preventing security risks caused by data quality issues. After the target verification results indicate that the data meets the expected standards, the first data information is securely transmitted to the target server. Finally, the second database reads the first data information from the target server to complete the data migration. Accurately identifying the target data range reduces the amount of data transmitted, thereby lowering the risk of data leakage. Data verification is performed on the target platform, effectively avoiding subsequent security risks caused by data quality issues, thus achieving the technical effect of improving the security and efficiency of heterogeneous data migration.

[0085] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0086] Example 2

[0087] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. The data processing apparatus provided in this application will be described below.

[0088] According to an embodiment of this application, an apparatus for implementing the above-described data processing method is also provided, such as... Figure 3 As shown, the device includes: an acquisition unit 301, an inspection unit 302, and a reading unit 303.

[0089] The acquisition unit 301 is used to acquire the target data range to be migrated in the first database, and migrate the first data information from the first database to the target platform according to the target data range;

[0090] The inspection unit 302 is used to inspect the first data information based on preset data rules through the target platform, obtain the target inspection result, and send the first data information to the target server through the target platform after the target inspection result indicates that the inspection has passed.

[0091] The reading unit 303 is used to read the first data information from the target server through the second database in order to complete the data migration.

[0092] The data processing apparatus provided in this application embodiment acquires the target data range to be migrated from the first database through the acquisition unit 301, and migrates the first data information from the first database to the target platform according to the target data range; the verification unit 302 verifies the first data information based on preset data rules through the target platform, obtains the target verification result, and sends the first data information to the target server through the target platform after the target verification result indicates that the verification has passed; the reading unit 303 reads the first data information from the target server through the second database to complete the data migration, thus solving the technical problem of poor security in heterogeneous data migration in related technologies.

[0093] In this solution, accurately identifying the target data range to be migrated from the first database avoids blind migration, reduces unnecessary data transmission, and lowers the risk of data leakage. Based on the target data range, data is extracted from the first database and preliminarily processed. Data verification is performed using the target platform, enabling rapid identification and location of data problems, allowing for timely corrective measures and preventing security risks caused by data quality issues. After the target verification results indicate that the data meets the expected standards, the first data information is securely transmitted to the target server. Finally, the second database reads the first data information from the target server to complete the data migration. Accurately identifying the target data range reduces the amount of data transmitted, thereby lowering the risk of data leakage. Data verification is performed on the target platform, effectively avoiding subsequent security risks caused by data quality issues, thus achieving the technical effect of improving the security and efficiency of heterogeneous data migration.

[0094] Optionally, in the data processing apparatus provided in this application embodiment, the acquisition unit includes: a first determining module, configured to determine the target business requirements and determine the target data fields necessary for the second database based on the target business requirements; and a second determining module, configured to determine the target data range based on the target data fields.

[0095] Optionally, in the data processing apparatus provided in this application embodiment, the acquisition unit includes: an extraction module, used to extract first data information from a first database using a data pump tool to obtain first data information in a first data format; an acquisition module, used to acquire a second data format corresponding to a target platform, and perform format conversion on the first data information in the first data format to obtain first data information in the second data format; and a migration module, used to migrate the first data information in the second data format to the target platform.

[0096] Optionally, in the data processing apparatus provided in this application embodiment, the verification unit includes: a first verification module, used to verify the total number of data entries of the first data information to obtain a first verification result; a second verification module, used to verify the data content of the first data information to obtain a second verification result; a third verification module, used to verify the data integrity of the first data information to obtain a third verification result; a fourth verification module, used to perform abnormal data detection on the first data information to obtain a fourth verification result; and a processing module, used to obtain a target verification result based on the first verification result, the second verification result, the third verification result, and the fourth verification result.

[0097] Optionally, in the data processing apparatus provided in this application embodiment, the second verification module includes: a first acquisition submodule, used to acquire multiple field information corresponding to the first data information; a construction submodule, used to construct a target structured query statement based on the multiple field information; and a first verification submodule, used to verify the field values ​​corresponding to the multiple field information based on the target structured query statement to obtain a second verification result.

[0098] Optionally, in the data processing apparatus provided in this application embodiment, the third verification module includes: a second acquisition submodule, used to acquire the primary key value and the foreign key value in the first data information; and a second verification submodule, used to verify the data integrity of the first data information based on the primary key value and the foreign key value, and obtain a third verification result.

[0099] Optionally, in the data processing apparatus provided in this application embodiment, the reading unit includes: a loading module, used to load first data information from the target server into a temporary data table through a second database; a mapping module, used to perform data mapping on the first data information in the temporary data table according to the data structure of the second database to obtain second data information; and a migration module, used to migrate the second data information to a business data table in the second database to complete the data migration.

[0100] It should be noted that the acquisition unit 301, verification unit 302, and reading unit 303 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0101] Example 3

[0102] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0103] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: obtain the target data range to be migrated from the first database, and migrate the first data information from the first database to the target platform according to the target data range; verify the first data information through the target platform based on preset data rules, obtain the target verification result, and after the target verification result indicates that the verification has passed, send the first data information to the target server through the target platform; and read the first data information from the target server through the second database to complete the data migration.

[0105] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: obtaining the target data range to be migrated in the first database includes: determining the target business requirements, and determining the target data fields necessary for the second database based on the target business requirements; and determining the target data range based on the target data fields.

[0106] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: migrating first data information from the first database to the target platform according to the target data range includes: extracting first data information from the first database through a data pump tool to obtain first data information in a first data format; obtaining a second data format corresponding to the target platform, and converting the first data information in the first data format to obtain first data information in the second data format; and migrating the first data information in the second data format to the target platform.

[0107] The processor can access the information and application programs stored in the memory via the transmission device to execute the following steps: The target platform verifies the first data information based on preset data rules to obtain a target verification result, including: verifying the total number of data entries in the first data information to obtain a first verification result; verifying the data content of the first data information to obtain a second verification result; verifying the data integrity of the first data information to obtain a third verification result; detecting abnormal data in the first data information to obtain a fourth verification result; and obtaining the target verification result based on the first verification result, the second verification result, the third verification result, and the fourth verification result.

[0108] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: verifying the data content of the first data information to obtain a second verification result, including: obtaining multiple field information corresponding to the first data information; constructing a target structured query statement based on the multiple field information; and verifying the field values ​​corresponding to the multiple field information based on the target structured query statement to obtain a second verification result.

[0109] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: verifying the data integrity of the first data information to obtain a third verification result, including: obtaining the primary key value and the foreign key value in the first data information; verifying the data integrity of the first data information based on the primary key value and the foreign key value to obtain a third verification result.

[0110] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: reading first data information from the target server through the second database to complete the data migration, including: loading the first data information from the target server into a temporary data table through the second database; mapping the first data information in the temporary data table according to the data structure of the second database to obtain the second data information; and migrating the second data information to the business data table of the second database to complete the data migration.

[0111] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0112] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0113] Example 4

[0114] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.

[0115] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0116] This application also provides a computer program product, which, when executed on a data processing device, is a program adapted to perform data processing method steps.

[0117] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0118] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0123] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the target data range to be migrated from the first database, and migrate the first data information from the first database to the target platform according to the target data range; The target platform verifies the first data information based on preset data rules to obtain a target verification result. After the target verification result indicates that the verification has passed, the first data information is sent to the target server through the target platform. The data migration is completed by reading the first data information from the target server through a second database.

2. The method according to claim 1, characterized in that, The scope of the target data to be migrated in the first database includes: Determine the target business requirements, and based on the target business requirements, determine the target data fields necessary for the second database; The target data range is determined based on the target data field.

3. The method according to claim 1, characterized in that, Migrating first data information from the first database to the target platform according to the target data range includes: The first data information is extracted from the first database using the data pump tool to obtain the first data information in the first data format; Obtain the second data format corresponding to the target platform, and convert the first data information of the first data format to obtain the first data information of the second data format; The first data information in the second data format is migrated to the target platform.

4. The method according to claim 1, characterized in that, The target platform verifies the first data information based on preset data rules, and the resulting verification results include: The total number of data entries in the first data information is checked to obtain the first check result; The data content of the first data information is examined to obtain a second examination result; The integrity of the first data information is verified to obtain a third verification result. Anomaly detection is performed on the first data information to obtain the fourth inspection result; The target test result is obtained based on the first test result, the second test result, the third test result, and the fourth test result.

5. The method according to claim 4, characterized in that, The data content of the first data information is examined to obtain the second examination result, which includes: Obtain multiple field information corresponding to the first data information; Based on the information from the multiple fields, construct the target structured query statement; Based on the target structured query statement, the field values ​​corresponding to the multiple field information are examined to obtain the second examination result.

6. The method according to claim 4, characterized in that, The data integrity of the first data information is checked, and the third check result includes: Obtain the primary key value and the foreign key value from the first data information; The data integrity of the first data information is checked based on the primary key value and the foreign key value to obtain the third check result.

7. The method according to claim 1, characterized in that, The data migration is completed by reading the first data information from the target server through a second database, including: The first data information is loaded from the target server into a temporary data table through the second database; Based on the data structure of the second database, the first data information in the temporary data table is mapped to obtain the second data information; The second data information is migrated to the business data table of the second database to complete the data migration.

8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the target data range to be migrated in the first database, and migrate the first data information from the first database to the target platform according to the target data range; The verification unit is used to verify the first data information based on preset data rules through the target platform, obtain the target verification result, and send the first data information to the target server through the target platform after the target verification result indicates that the verification has passed. The reading unit is used to read the first data information from the target server through the second database to complete the data migration.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 7.

Images