Data migration method and apparatus, related device and computer program product

Abstract

The present disclosure provides a data migration method, device, related equipment and computer program product, and relates to the technical field of computers and Internet. The data migration method comprises: inquiring the requirements of a data migration task from a target object according to a problem list; generating data migration requirement text information of the data migration task according to the reply of the target object, wherein the data migration requirement text information comprises a source database and a target database of the data migration task; converting the data migration requirement text information into a configuration file supported by a data migration component; determining that the data migration component cannot normally run the configuration file, and then determining that the data migration component does not support the source database or the target database; generating data migration code according to the configuration file; and running the data migration code to migrate the data in the source database to the target database. The present disclosure can simply and automatically complete the migration of data.

Description

Technical Field

[0001] This disclosure relates to the field of computer and Internet technology, and in particular to a data migration method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Data migration is a very common functional scenario. For example, when machines are replaced, the system's data needs to be migrated from the old machine to the new machine; or various order systems synchronize order information to a big data system.

[0004] Currently, there are many mature structured data migration methods and systems in the industry. The typical process is: manually setting up a data migration configuration file, building the target database table, connecting the source database and the target database, and migrating the data to the target database.

[0005] However, traditional data migration methods require human intervention, resulting in low data migration efficiency and complex operation. Summary of the Invention

[0006] The purpose of this disclosure is to provide a data migration method, apparatus, electronic device, computer-readable storage medium, and computer program product that can complete data migration in a relatively simple and efficient manner.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] This disclosure provides a data migration method, comprising: querying a target object for data migration task requirements based on a list of questions; generating data migration requirement text information for the data migration task based on the target object's response, the data migration requirement text information including a source database and a target database for the data migration task; converting the data migration requirement text information into a configuration file supported by a data migration component; determining that the data migration component cannot run the configuration file correctly, and thus determining that the data migration component does not support the source database or the target database; generating data migration code based on the configuration file; and running the data migration code to migrate data from the source database to the target database.

[0009] In some embodiments, the method further includes: acquiring data migration requirement training text information and data migration training configuration file of the data migration component, wherein the data migration component can run the data migration training configuration file normally; acquiring a first large model that has completed natural language pre-training; fine-tuning the first large model using the data migration requirement training text information and the data migration training configuration file; wherein converting the data migration requirement text information into a configuration file supported by the data migration component includes: converting the data migration requirement text information into the configuration file supported by the data migration component using the first large model.

[0010] In some embodiments, the data migration component includes a first data migration component, and the configuration file includes a first configuration file; wherein, converting the data migration requirement text information into a configuration file supported by the data migration component includes: converting the data migration requirement text information into the first configuration file supported by the first data migration component; wherein, the method further includes: if it is determined that the first data migration component can run the configuration file normally, then running the first configuration file through the first data migration component to migrate data from the source database to the target database.

[0011] In some embodiments, the data migration component further includes a second data migration component, and the configuration file further includes a second configuration file; wherein the method further includes: if it is determined that the first data migration component cannot run the first configuration file normally, then converting the data migration requirement text information into a second configuration file supported by the second data migration component; if it is determined that the second data migration component can run the second configuration file normally, running the second configuration file through the second data migration component to migrate data from the source database to the target database; if it is determined that the second data migration component cannot run the second configuration file normally, then determining that the data migration component does not support the source database or the target database.

[0012] In some embodiments, the method involves querying a target object for data migration task requirements based on a list of questions. The method further includes: obtaining a list of questions for the data migration task; querying the target object using the list of questions; collecting the target object's response information, which includes target terms; and generating data migration requirement text information for the data migration task based on the target object's response, which includes: correcting the target terms in the response content using preset terms from a preset database; and generating the data migration requirement text information based on the corrected response information.

[0013] In some embodiments, generating data migration code based on the configuration file includes: parsing the configuration file to obtain a source database, a source data table, a target database, and a target data table for the data migration task; generating a first prompt word for connecting to the source database, a second prompt word for reading data from the source data table in the source database, a third prompt word for connecting to the target database, and a fourth prompt word for writing data to the target data table in the target database; inputting the first prompt word, the second prompt word, the third prompt word, and the fourth prompt word into a second large model, so that the second large model generates the first code for connecting to the source database, the second code for reading data from the source data table in the source database, the third code for connecting to the target database, and the fourth code for writing data to the target data table in the target database; and generating the data migration code based on the first code, the second code, the third code, and the fourth code.

[0014] This disclosure provides a data migration apparatus, including: an inquiry module, a text information generation module, a configuration file generation module, a first running module, a migration code generation module, and a code running module.

[0015] The query module is used to inquire about the requirements of the data migration task from the target object based on a list of questions; the text information generation module can be used to generate data migration requirement text information for the data migration task based on the response from the target object, the data migration requirement text information including the source database and the target database of the data migration task; the configuration file generation module can be used to convert the data migration requirement text information into a configuration file supported by the data migration component; the first running module can be used to determine that the data migration component does not support the source database or the target database if it determines that the data migration component cannot run the configuration file normally; the migration code generation module can be used to generate data migration code based on the configuration file; the code running module can be used to run the data migration code to migrate the data from the source database to the target database.

[0016] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the data migration method described above.

[0017] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the data migration method as described in any of the preceding embodiments.

[0018] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the aforementioned data migration method.

[0019] The data migration method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure can automatically generate configuration files or migration code based on the target object's response information when only asking questions from the target object's question list, so as to complete the data migration automatically and efficiently.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A schematic diagram of a scenario that can be applied to the data migration method or data migration apparatus of the embodiments of this disclosure is shown.

[0023] Figure 2 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0025] Figure 4 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0026] Figure 5 This is a flowchart illustrating a method for generating text information about data migration requirements according to an exemplary embodiment.

[0027] Figure 6 This is a flowchart illustrating a data migration code generation method according to an exemplary embodiment.

[0028] Figure 7 This is a schematic diagram illustrating the structure of a data migration method according to an exemplary embodiment.

[0029] Figure 8This is a schematic diagram illustrating a configuration file generation method according to an exemplary embodiment.

[0030] Figure 9 This is a time-series interaction diagram illustrating a data migration requirement data extraction according to an exemplary embodiment.

[0031] Figure 10 This is a timing interaction diagram illustrating data migration code generation according to an exemplary embodiment.

[0032] Figure 11 This is a block diagram illustrating a data migration apparatus according to an exemplary embodiment.

[0033] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0035] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0036] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0037] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0040] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0041] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0042] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0043] Mature structured data migration methods and systems typically involve business users manually configuring settings on a webpage to generate the configuration file, or IT personnel directly writing the configuration file. However, these methods require business users to possess specialized data migration knowledge or programming skills, placing a high demand on them. This application proposes a more interactive and user-friendly method for setting up configuration files, which involves multiple rounds of queries to business users and automatically generates the configuration file based on their needs.

[0044] Furthermore, mature structured data migration methods and systems can only handle source and target databases where system functions are already implemented. This application proposes a method for handling unknown databases. Based on requirements obtained through multiple rounds of queries, if the source or target database is unknown, code for reading source data and code for writing to the target database can be generated and then submitted to the system to implement data migration.

[0045] Figure 1 A schematic diagram of a scenario that can be applied to the data migration method or data migration apparatus of the embodiments of this disclosure is shown.

[0046] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0047] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0048] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0049] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0050] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0051] Server 105 may, for example, query the target object for the requirements of the data migration task based on a list of questions; Server 105 may, for example, generate data migration requirement text information for the data migration task based on the target object's response, the data migration requirement text information including the source database and target database of the data migration task; Server 105 may, for example, convert the data migration requirement text information into a configuration file supported by the data migration component; Server 105 may, for example, determine that the data migration component cannot run the configuration file properly, and thus determine that the data migration component does not support the source database or the target database; Server 105 may, for example, generate data migration code based on the configuration file; Server 105 may, for example, run the data migration code to migrate data from the source database to the target database.

[0052] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0053] The following section will first explain some of the terms used in the embodiments of this disclosure so that those skilled in the art can understand them.

[0054] General-purpose LLM (Large Language Model): Large language models that can be used to develop applications or perform secondary training.

[0055] src: The source text in the corpus dataset of the large language model.

[0056] tgt: Target text in the corpus dataset of a large language model.

[0057] The preceding text introduced some terms and concepts involved in the embodiments of this disclosure. The following text introduces the technical features involved in the embodiments of this disclosure.

[0058] Figure 2 This is a flowchart illustrating a data migration method according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be implemented by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0059] Reference Figure 2 The data migration method provided in this disclosure may include the following steps.

[0060] Step S202: Based on the problem list, inquire with the target object about the requirements of the data migration task.

[0061] In some embodiments, a list of questions may be pre-set, which may include, for example, questions related to data migration such as: what is the source database, what is the target database, what is the source table, what is the target table, and what is the data migration method.

[0062] These questions allow you to ask the target audience about their requirements for the data migration task.

[0063] Step S204: Generate data migration requirement text information for the data migration task based on the response from the target object. The data migration requirement text information includes the source database and the target database of the data migration task.

[0064] In some embodiments, the target's response may be a text response or a voice response, etc., and this application does not limit this.

[0065] In some embodiments, the target object's response can be converted into text information indicating a data migration requirement. For example, assuming the target object provides a voice response, the voice response can be converted into text information to serve as the migration requirement text information. This application does not limit the format or content of the data migration requirement text information.

[0066] In some embodiments, since the target object may not be clear about the standard names of certain terms involved in the transfer task, or the description is inaccurate or the speech conversion is inaccurate, it may be necessary to correct the target object's response. For example, the target object's response can be corrected by preset standard terms, or by a pre-trained neural network model, thereby making the target object's response more standard.

[0067] Specifically, if the source library uses an abbreviation in the target object's response, the full name of the abbreviation can be determined by traversing the preset standard names. Then, the target object can be queried to see if the full name of the source library is XXXX, thus making the data migration more accurate.

[0068] In some embodiments, if the target object responds via text, the target object's response text can be converted into a data migration requirement text.

[0069] Step S206: Convert the data migration requirement text information into a configuration file supported by the data migration component.

[0070] In some embodiments, a neural network model can be pre-trained to convert textual information about data migration requirements into a configuration file supported by the data migration component.

[0071] In some embodiments, a neural network model can be trained using the following method: First, a large model is pre-trained using business text within a preset business scenario to give it a basic understanding of that scenario; then, the large model is fine-tuned using known data migration requirement text; finally, the large model is fine-tuned again using known data migration configuration files of certain data migration components. Through this method, the large model gains a basic understanding of the preset business scenario, the data migration requirement text, and the data migration configuration files corresponding to certain data migration components within that scenario. Based on this understanding, the large model can accurately convert the data migration requirement text into a configuration file for a specific data migration component.

[0072] Large models, also known as mega-models or ultra-large-scale models, refer to deep learning models with a large number of parameters and computational complexity. These models typically consist of billions or even hundreds of billions of parameters and can process massive amounts of data and produce complex outputs.

[0073] It is important to note that different data migration components may correspond to different types of data migration configuration files (different formats or different content).

[0074] In some embodiments, after the large model training is completed, the aforementioned data migration requirement text information can be combined with specific prompts (such as generating a configuration file corresponding to a data migration component based on the data migration requirement text information) to generate a configuration file supported by the data migration component. These prompts can be used to indicate the specific type of data migration component configuration file to generate.

[0075] The aforementioned data migration components are a set of tools or modules used to migrate data from one location or system to another. They provide functions such as data selection, preparation, transformation, and transfer to ensure data consistency and integrity. These data migration components are pre-built tools with relatively complete functions.

[0076] If step S208 determines that the data migration component cannot run the configuration file correctly, then it is determined that the data migration component does not support the source database or the target database.

[0077] In some embodiments, if the data migration component can run the above configuration file normally, it is determined that the data migration component supports the source database or the target database. In this case, the data migration component is directly allowed to run the configuration file normally to realize the data migration task from the source database to the target database.

[0078] In some embodiments, if the data migration component cannot run the above configuration file properly (e.g., an error occurs during operation), it is determined that the data migration component does not support the source database or the target database.

[0079] Step S210: Generate data migration code based on the configuration file.

[0080] In some embodiments, if it is determined that the data migration component cannot function properly according to the above configuration file, data migration code can be generated based on the above configuration file or the above data migration requirement text information.

[0081] Specifically, another large model can be pre-trained to generate data migration code based on the aforementioned configuration file or data migration requirement text information.

[0082] Step S212: Run the data migration code to migrate data from the source database to the target database.

[0083] In some embodiments, data migration code can be run to migrate data from a source database to a target database; or to migrate data from a source table in a source database to a target table in a target database.

[0084] The above embodiments can automatically generate configuration files or migration code based on the target object's responses when only questions from the target object's question list are asked, so as to complete data migration automatically and efficiently.

[0085] Figure 3 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0086] refer to Figure 3 The above data migration method may include the following steps.

[0087] Step S302: Obtain the data migration requirement training text information and data migration training configuration file of the data migration component. The data migration component can run the data migration training configuration file normally.

[0088] The aforementioned data migration requirement training text information may refer to the preset text information used to generate configuration files. This application does not impose any restrictions on the content, format, etc. of the text information, as long as the text information can describe the data migration task.

[0089] In some embodiments, the aforementioned data migration training configuration file may refer to the configuration file corresponding to the data migration component, that is, the preset data migration component can run the data migration configuration file.

[0090] In some embodiments, the data migration components described above may include one, two, or more, and this application does not impose any restrictions on this.

[0091] Step S304: Obtain the first large model that has completed natural language pre-training.

[0092] In some embodiments, the first model can be trained in advance using text content from general scenarios to enable it to have a basic understanding of language content in most scenarios.

[0093] Step S306: Fine-tune the first large model by using the data migration requirement training text information and the data migration training configuration file.

[0094] In some embodiments, after the pre-training of the large model is completed, the training text information on data migration requirements and the training text information on data migration configuration can be fed to the first large model to fine-tune the training of the large model. This allows the large model to have a basic understanding and awareness of how to generate the data migration sequence configuration file corresponding to the specified data migration component based on the training text information on data migration requirements.

[0095] Step S308: The data migration requirement text information is converted into a configuration file supported by the data migration component through the first major model.

[0096] After training the first major model, the data migration requirement text information can be converted into a configuration file supported by the data migration component.

[0097] The above method allows for a simple and convenient conversion of textual information about data migration requirements into configuration files supported by the data migration component.

[0098] Figure 4 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0099] In some embodiments, the data migration component described above may include at least one data migration component, which may include a first data migration component and a second data migration component.

[0100] In some embodiments, the configuration file generated based on the data migration requirement text information may include at least one configuration file, which may include a first configuration file and a second configuration file.

[0101] refer to Figure 4 The above data migration method may include the following steps.

[0102] Step S402: Based on the problem list, inquire with the target object about the requirements of the data migration task.

[0103] Step S404: Generate data migration requirement text information for the data migration task based on the response from the target object. The data migration requirement text information includes the source database and the target database of the data migration task.

[0104] Step S406: Convert the data migration requirement text information into a first configuration file supported by the first data migration component.

[0105] Step S408: If it is determined that the first data migration component can run the configuration file normally, then the first configuration file is run through the first data migration component to migrate the data in the source database to the target database.

[0106] Step S410: If it is determined that the first data migration component cannot run the first configuration file properly, the data migration requirement text information is converted into a second configuration file supported by the second data migration component.

[0107] Step S412: If it is determined that the second data migration component can run the second configuration file normally, the second configuration file is run through the second data migration component to migrate the data in the source database to the target database.

[0108] Step S414: If it is determined that the second data migration component cannot run the second configuration file properly, then it is determined that the data migration component does not support the source database or the target database.

[0109] In some embodiments, if after traversing at least one of the above data migration components it is found that none of the data migration components support the source database or target database of the data migration requirement text information center, then it is determined that the existing data migration components do not support the source database or target database.

[0110] Step S416: If it is determined that the data migration component does not support the source database or the target database, generate data migration code according to the configuration file.

[0111] Step S418: Run the data migration code to migrate data from the source database to the target database.

[0112] Using the above method, it is possible to accurately determine whether the existing data migration component can support the source database or the target database. If the existing data migration component cannot support the source database or the target database, data migration code is generated based on the data migration requirement text information or configuration file, so as to complete the task of migrating data from the source database to the target database based on the data migration code.

[0113] Figure 5 This is a flowchart illustrating a method for generating text information about data migration requirements according to an exemplary embodiment.

[0114] refer to Figure 5 The above-mentioned method for generating text information on data migration requirements may include the following steps.

[0115] Step S502: Obtain the problem list for the data migration task.

[0116] Step S504: Ask the target object through the question list.

[0117] Step S506: Collect the response information from the target object, which includes the target noun.

[0118] Step S508: Modify the target nouns in the response content using preset nouns from the preset database.

[0119] Step S510: Generate data migration requirement text information based on the revised response information.

[0120] The above methods can be used to repair the response information of the target object, making the response more standardized, thereby making the requirement information of the data migration task more accurate, and thus ensuring the normal progress of the subsequent data migration task.

[0121] Figure 6 This is a flowchart illustrating a data migration code generation method according to an exemplary embodiment.

[0122] refer to Figure 6 The above data migration code generation method may include the following steps.

[0123] Step S602: Parse the configuration file to obtain the source database, source data table, target database, and target data table for the data migration task.

[0124] Step S604: Generate a first prompt word for connecting to the source database, a second prompt word for reading data from the source data table of the source database, a third prompt word for connecting to the target database, and a fourth prompt word for writing data to the target data table of the target database.

[0125] The aforementioned first prompt word can be used to assist in prompting the large model to generate code that connects to the source database. For example, "generate code that connects to the XX database" would be the aforementioned first prompt word.

[0126] The aforementioned second prompt can be used to assist in prompting the large model to generate code that reads data from the source data table in the source database. For example, "generate code that reads data from table XX in database XX" would be the aforementioned second prompt.

[0127] Similarly, the third and fourth prompt words can be understood, and this application does not impose any restrictions on them.

[0128] It is understood that those skilled in the art can add, delete, and modify the aforementioned first prompt word, second prompt word, third prompt word, and fourth prompt word according to their own needs, and this application does not impose any restrictions on this.

[0129] Step S606: Input the first prompt word, the second prompt word, the third prompt word, and the fourth prompt word into the second large model, so that the second large model generates the first code for connecting to the source database, the second code for reading data from the source data table of the source database, the third code for connecting to the target database, and the fourth code for writing data to the target data table of the target database.

[0130] The second major model mentioned above can be a pre-trained model that can generate data migration code based on some prompt words and database.

[0131] Step S608: Generate data migration code based on the first code, the second code, the third code, and the fourth code.

[0132] Below, this application will explain and illustrate the data migration method in conjunction with specific application scenarios.

[0133] The new generation of cloud network operation business system consists of multiple IT systems, each of which continuously produces and accumulates a large amount of data, making data migration a very common functional scenario. For example, when machines are replaced, data from IT systems is migrated from the old machines to the new machines; and order systems synchronize order information to the big data system, etc.

[0134] The data in the new generation of cloud network operation business systems is mostly structured data. Therefore, the data migration scenario discussed in this application can be the migration of data from structured data storage to structured data storage.

[0135] Currently, there are many mature structured data migration methods and systems in the industry. The typical process is: manually setting up a data migration configuration file, building the target database table, connecting the source database and the target database, and migrating the data to the target database.

[0136] Mature structured data migration methods and systems typically involve business users manually configuring settings on a webpage to generate the configuration file, or IT personnel directly writing the configuration file. This application proposes a more interactive and user-friendly method for setting up configuration files, which involves multiple rounds of queries to business users and automatically generating the configuration file based on their needs.

[0137] Mature structured data migration methods and systems can only handle source and target databases whose system functions (such as data migration components) have already been implemented. This application proposes a method for handling unknown databases. Based on the requirements obtained from multiple rounds of queries, if the source or target database is unknown, code for reading source data and code for writing to the target database will be generated and then submitted to the system to implement data migration.

[0138] This patent proposes a user-friendly data migration method and system based on a large AI model, which improves the intelligence of data migration configuration and addresses the inability of traditional methods to handle unknown data sources.

[0139] The execution process of a data migration task is roughly as follows: Figure 7 As shown.

[0140] like Figure 7 As shown, firstly, the requirement inquiry module 701, based on the question list, inquires about the requirements of the data migration task from the business personnel through multiple rounds of dialogue; then, the requirement inquiry module 701 extracts the requirements from the multi-round dialogue text and generates a configuration file; finally, the data migration module 702 reads and parses the configuration file. If both the source database and the target database are known, the data migration task is started directly; otherwise, code for reading and writing data is automatically generated and executed to achieve data migration.

[0141] The main functions of the demand inquiry module can be found in the following reference. Figure 8 The illustrated embodiment.

[0142] like Figure 8 As shown, the requirement query module can extract requirement information and generate configuration files.

[0143] For details on extracting demand information, please refer to [link / reference]. Figure 9 The illustrated embodiment.

[0144] When business personnel enter the system, the requirements inquiry module will proactively consult with them about their data migration task needs based on a pre-prepared question list template. Through multiple rounds of dialogue, it will collect the text information needed to generate the configuration file. Then, based on the business personnel's answers, it will retrieve more precise information from the enterprise knowledge database and extract it as the requirements information.

[0145] For example, when a business user enters the system, the requirements inquiry module asks, "Which is the source database?" The business user answers, "XXX production environment database." The requirements inquiry module then searches the enterprise knowledge database and continues to ask, "Please confirm if it is the following database? Database type: xxx, database IP: xxx, database port: xxx, username: xxx, password: xxx..." The business user answers, "No, it should be xxx..." The requirements inquiry module asks the business user questions one by one based on a pre-prepared list of questions, and obtains more precise information by combining it with the enterprise knowledge database. Then, it extracts the real requirements information from the text information of the multi-round dialogue (such as information about the source database, source tables, target database, target tables, the correspondence between source and target tables, data read / write throughput requirements, data filtering rules, which tables to be operated on, which fields to be operated on, special transformation operations, and whether a single operation may involve dozens of tables, etc.).

[0146] The generated configuration file may include the following content.

[0147] Regularly collect datasets related to data migration configurations to serve as the `src` and `tgt` text files. The `src` file contains well-structured requirements information, including source database information, source table information, target database information, target table information, the correspondence between source and target tables, data read / write throughput requirements, and data filtering rules. The `tgt` file is a directly executable JSON file containing the data migration configuration. Regularly read the latest data migration configuration dataset and train a large configuration model using a fine-tuning mode based on a general large language model. When the requirements query module extracts requirements information based on business personnel input, it inputs the configuration to generate the large model and automatically generates the data migration configuration file.

[0148] In some embodiments, if the migration component is upgraded, the configuration file is also upgraded, but the large model does not need to be upgraded; only the prompt words need to be changed.

[0149] In some embodiments, the core functions of the data migration module may include: a data migration component and data migration code generation. The data migration component integrates various data read / write methods and data migration tools, offering relatively comprehensive functionality and stable performance. Data migration code generation automatically generates data migration code based on the data migration configuration file, which can be submitted for execution after user review.

[0150] The data migration module receives the data migration configuration file and first analyzes whether the source and target databases exist in the data migration component. If both exist, the data migration task is submitted for execution through the component. If either the source or target database is missing, data migration code is generated using the data migration code generation function, reviewed by the user, and then submitted for execution. Generally, if the data migration component encounters an error while running the configuration file, it can be assumed that the component does not support either the source or target database.

[0151] The data migration component is a traditional data migration system, which will not be described in this application. The data migration code generation is a code generation tool based on a large AI model. The process of generating data migration code (read and write code) is as follows: Figure 10 As shown.

[0152] like Figure 10 As shown, the data migration configuration parsing module first parses the source database and generates prompts for connecting to the source database using a prompt word toolkit. Inputting these prompts into the Large LLM (Limited Model) generates the code for connecting to the source database. Similarly, the code for reading from the source database, connecting to the target database, and writing data to the target database can be generated sequentially.

[0153] Understandably, the final generated data migration code is more rudimentary than the data migration component, for example, it lacks circuit breaker functionality. However, this data migration code can complete the most basic data migration operations.

[0154] This application proposes a multi-round query and retrieval method to analyze and extract requirements for data migration tasks. Compared to traditional data migration methods, business personnel do not need to learn many complex operations; they only need to input their requirements based on the system's queries. Furthermore, business personnel do not need to learn much about the underlying system design and internal data design; the system retrieves more precise requirement information based on the vague requirements.

[0155] This application proposes a method for fine-tuning a general large model using a data migration configuration dataset to obtain a large production model with configuration settings, thereby automatically generating data migration configuration files based on demand information. Compared to traditional data migration methods, this method eliminates the steps of manipulating configuration items on a webpage or manually writing configuration files, thus improving the automation and intelligence of data migration configuration.

[0156] This application proposes a method for generating data read / write code based on a data migration configuration file using a general-purpose LLM, thereby overcoming the shortcomings of traditional data migration methods in handling unknown data sources.

[0157] It should be particularly noted that the steps in the various embodiments of the above data migration method can be overlapped, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of data migration methods should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the embodiments.

[0158] Based on the same inventive concept, this disclosure also provides a data migration apparatus, as shown in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be described again.

[0159] Figure 11 This is a block diagram illustrating a data migration apparatus according to an exemplary embodiment. (Refer to...) Figure 11 The data migration device 1100 provided in this embodiment may include: an inquiry module 1101, a text information generation module 1102, a configuration file generation module 1103, a first running module 1104, a migration code generation module 1105, and a code running module 1106.

[0160] The system includes the following modules: Inquiry module 1101, which queries the target object for the requirements of the data migration task based on a list of questions; Text information generation module 1102, which generates data migration requirement text information for the data migration task based on the target object's response, including the source database and target database; Configuration file generation module 1103, which converts the data migration requirement text information into a configuration file supported by the data migration component; First execution module 1104, which determines that the data migration component does not support the source database or target database if it cannot run the configuration file correctly; Migration code generation module 1105, which generates data migration code based on the configuration file; and Code execution module 1106, which runs the data migration code to migrate data from the source database to the target database.

[0161] It should be noted that the query module 1101, text information generation module 1102, configuration file generation module 1103, first running module 1104, migration code generation module 1105, and code running module 1106 mentioned above correspond to S202 to S212 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0162] In some embodiments, the data migration device 1100 may further include: a data migration training configuration file acquisition module, a first large model acquisition module, and a fine-tuning training module.

[0163] Among them, the data migration training configuration file acquisition module can be used to obtain the data migration requirement training text information and data migration training configuration file of the data migration component, and the data migration component can run the data migration training configuration file normally; the first major model acquisition module can be used to obtain the first major model that has completed natural language pre-training; the fine-tuning training module can be used to fine-tune the first major model through the data migration requirement training text information and data migration training configuration file.

[0164] The configuration file generation module 1103 may include a large model processing submodule.

[0165] The large model processing submodule can be used to convert the text information of data migration requirements into a configuration file supported by the data migration component through the first large model.

[0166] In some embodiments, the data migration component includes a first data migration component, and the configuration file includes a first configuration file.

[0167] The configuration file generation module 1103 may include: a first configuration file generation submodule.

[0168] The first configuration file generation submodule can be used to convert the text information of data migration requirements into the first configuration file supported by the first data migration component.

[0169] The data migration device 1100 may further include: a first judgment module.

[0170] The first judgment module can be used to run the first configuration file through the first data migration component if it is determined that the first data migration component can run the configuration file normally, so as to migrate the data in the source database to the target database.

[0171] In some embodiments, the data migration component further includes a second data migration component, and the configuration file further includes a second configuration file; wherein, the data migration device 1100 may further include: a second judgment module, a third judgment module, and a fourth judgment module.

[0172] The second judgment module can be used to convert the data migration requirement text information into a second configuration file supported by the second data migration component if it is determined that the first data migration component cannot run the first configuration file normally; the third judgment module can be used to run the second configuration file through the second data migration component to migrate the data in the source database to the target database if it is determined that the second data migration component can run the second configuration file normally; the fourth judgment module can be used to determine that the data migration component does not support the source database or the target database if it is determined that the second data migration component cannot run the second configuration file normally.

[0173] In some embodiments, the data migration device 1100 queries the target object for the requirements of the data migration task based on the problem list; wherein, the data migration device 1100 may further include: a problem list acquisition module, a query module, and an information collection module.

[0174] The problem list acquisition module can be used to obtain a problem list for data migration tasks; the inquiry module can be used to inquire with the target object through the problem list; and the information collection module can be used to collect the response information of the target object, which includes the target noun.

[0175] The text information generation module 1102 may include: a noun correction submodule and a migration requirement text information generation submodule.

[0176] The noun correction submodule can be used to correct the target nouns in the response content using preset nouns in the preset database; the migration requirement text information generation submodule can be used to generate data migration requirement text information based on the corrected response information.

[0177] In some embodiments, the migration code generation module 1105 may include: a parsing submodule, a prompt word generation submodule, a subcode generation submodule, and a data migration code generation submodule.

[0178] The system includes the following components: a parsing submodule for parsing configuration files to obtain the source database, source data table, target database, and target data table for the data migration task; a prompt word generation submodule for generating a first prompt word for connecting to the source database, a second prompt word for reading data from the source data table in the source database, a third prompt word for connecting to the target database, and a fourth prompt word for writing data to the target data table in the target database; a subcode generation submodule for inputting the first, second, third, and fourth prompt words into a second main model to generate a first code for connecting to the source database, a second code for reading data from the source data table in the source database, a third code for connecting to the target database, and a fourth code for writing data to the target data table in the target database; and a data migration code generation submodule for generating data migration code based on the first, second, third, and fourth codes.

[0179] Since the functions of the device 1100 have been described in detail in their respective method embodiments, they will not be repeated here.

[0180] The modules and / or submodules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or submodules can also be located in a processor. The names of these modules and / or submodules do not, in some cases, constitute a limitation on the module and / or submodule itself.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0182] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0183] Figure 12 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 12 The illustrated electronic device 1200 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0184] like Figure 12 As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0185] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0186] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs the functions defined above in the system of this disclosure.

[0187] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0188] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: querying a target object for data migration task requirements based on a list of questions; generating data migration requirement text information for the data migration task based on the target object's response, the data migration requirement text information including the source database and target database of the data migration task; converting the data migration requirement text information into a configuration file supported by the data migration component; determining that the data migration component cannot run the configuration file correctly, and if so, determining that the data migration component does not support the source database or the target database; generating data migration code based on the configuration file; and running the data migration code to migrate data from the source database to the target database.

[0189] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0190] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several computer program instructions to cause an electronic device (such as a server or terminal device, etc.) to execute the method according to the embodiments of this disclosure.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0192] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A data migration method, characterized in that, include: Based on the question list, inquire with the target group about the requirements of the data migration task; Based on the response from the target object, generate data migration requirement text information for the data migration task, which includes the source database and the target database of the data migration task. The data migration requirement text information is converted into a configuration file supported by the data migration component; If it is determined that the data migration component cannot run the configuration file properly, then it is determined that the data migration component does not support the source database or the target database. Generate data migration code based on the configuration file; Run the data migration code to migrate data from the source database to the target database.

2. The method according to claim 1, characterized in that, Its features are, The method further includes: Obtain the data migration requirement training text information and data migration training configuration file of the data migration component, and the data migration component can run the data migration training configuration file normally; Obtain the first major model that has completed natural language pre-training; The first large model is fine-tuned and trained using the data migration requirement training text information and the data migration training configuration file. The process of converting the data migration requirement text information into a configuration file supported by the data migration component includes: The first major model converts the textual information of the data migration requirement into the configuration file supported by the data migration component.

3. The method according to claim 1, characterized in that, The data migration component includes a first data migration component, and the configuration file includes a first configuration file; The process of converting the data migration requirement text information into a configuration file supported by the data migration component includes: The data migration requirement text information is converted into the first configuration file supported by the first data migration component; The method further includes: If it is determined that the first data migration component can run the configuration file normally, then the first configuration file is run through the first data migration component to migrate the data in the source database to the target database.

4. The method according to claim 3, characterized in that, The data migration component further includes a second data migration component, and the configuration file further includes a second configuration file; wherein, the method further includes: If it is determined that the first data migration component cannot run the first configuration file properly, the data migration requirement text information is converted into the second configuration file supported by the second data migration component; If it is determined that the second data migration component can run the second configuration file normally, the second configuration file is run through the second data migration component to migrate the data in the source database to the target database; If it is determined that the second data migration component cannot function properly with the second configuration file, then it is determined that the data migration component does not support the source database or the target database.

5. The method according to claim 1, characterized in that, Based on the question list, ask the target group for the requirements of the data migration task, including: Obtain the problem list for the data migration task; The target object is questioned using the aforementioned list of questions; Collect the response information from the target object, wherein the response information includes the target noun; The process of generating data migration requirement text information for the data migration task based on the response from the target object includes: The target nouns in the response content are corrected using preset nouns from a preset database; The data migration requirement text information is generated based on the revised response information.

6. The method according to claim 1, characterized in that, Generate data migration code based on the configuration file, including: The configuration file is parsed to obtain the source database, source data table, target database, and target data table for the data migration task; Generate a first prompt word for connecting to the source database, a second prompt word for reading data from the source data table of the source database, a third prompt word for connecting to the target database, and a fourth prompt word for writing data to the target data table of the target database; The first prompt word, the second prompt word, the third prompt word, and the fourth prompt word are respectively input into the second large model so that the second large model generates the first code for connecting to the source database, the second code for reading data from the source data table of the source database, the third code for connecting to the target database, and the fourth code for writing data to the target data table of the target database. The data migration code is generated based on the first code, the second code, the third code, and the fourth code.

7. A data migration device, characterized in that, include: The query module is used to ask the target object for the requirements of the data migration task based on the question list; A text information generation module is used to generate data migration requirement text information for the data migration task based on the response of the target object. The data migration requirement text information includes the source database and the target database of the data migration task. The configuration file generation module is used to convert the data migration requirement text information into a configuration file supported by the data migration component; The first running module is used to determine that if the data migration component cannot run the configuration file normally, then the data migration component does not support the source database or the target database. The migration code generation module is used to generate data migration code based on the configuration file; The code execution module is used to run the data migration code to migrate data from the source database to the target database.

8. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being used to execute the data migration method as described in any one of claims 1-6 based on computer program instructions stored in the memory.

9. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the data migration method as described in any one of claims 1-6.

10. A computer program product comprising computer program instructions stored in a computer-readable storage medium, characterized in that, When the computer program instructions are executed by the processor, they implement the method according to any one of claims 1-6.

Images