Data storage scheduling migration method, device, computer equipment and storage medium

Abstract

The present invention discloses a data storage scheduling migration method, device, computer equipment and storage medium, and relates to the field of data processing. The method includes: when the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program in the master node, and the persistent volume declaration is used to declare the storage method and read-write method of the persistent volume; the AI ​​model predicts the current environment prediction parameters; the mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters; the working data is transmitted to the cloud storage server through the mounting service; when the slave node calls the POD program, the working data is obtained from the cloud storage server. In this way, the data synchronization speed and stability can be automatically optimized according to the current environment prediction parameters, effectively reducing the situation of data loss and synchronization failure.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data storage scheduling migration method, device, computer equipment and storage medium. Background Art

[0002] In a cloud-native Kubernetes environment, for convenience and personal preference, users often customize directories directly on their local machines and use them as external storage for Kubernetes applications. This approach can cause a series of problems, especially when POD programs are scheduled to other working nodes, which can easily cause errors. Although automatic storage provisioning solutions are commonly used in the market to solve such problems, such solutions usually make data management complex and unintuitive, resulting in increased maintenance costs. Summary of the invention

[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a data storage scheduling migration method, device, computer equipment and storage medium, which are used to intelligently enable the mounting service of the mounting tool according to the environmental prediction parameters of the current cloud native Kubernetes environment, and use the mounting service to realize data storage and scheduling migration.

[0004] The present invention provides the following technical solutions:

[0005] In a first aspect, the present invention provides a data storage scheduling migration method, comprising:

[0006] When the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage mode and read-write mode of the persistent volume;

[0007] The AI ​​model predicts the current environmental prediction parameters;

[0008] The mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters;

[0009] Transmitting the working data to a cloud storage server through the mounting service;

[0010] When the slave node calls the POD program, the working data is obtained from the cloud storage server.

[0011] In one embodiment, the AI ​​model predicts current environmental prediction parameters, including:

[0012] Get current network performance data;

[0013] The AI ​​model generates the current environment prediction parameters based on the current network performance data.

[0014] In one embodiment, before the AI ​​model generates the current environment prediction parameter according to the current network performance data, it includes:

[0015] Obtain historical network performance data and its corresponding historical mount operation results;

[0016] Based on a decision tree and a reinforcement learning algorithm, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model.

[0017] In one embodiment, the AI ​​model is trained based on the decision tree and reinforcement learning algorithm according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model, including:

[0018] Based on the decision tree, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain historical environment prediction parameters;

[0019] Based on the reinforcement learning algorithm, the AI ​​model is trained according to the historical environment prediction parameters and the historical environment feedback corresponding to the historical mounting operation results to obtain the maturely trained AI model.

[0020] In one implementation, starting the mounting service according to the current environment prediction parameter includes:

[0021] The AI ​​model adjusts the mounting service startup parameters of the mounting tool according to the current environment prediction parameters;

[0022] The mounting tool starts the mounting service according to the mounting service startup parameters.

[0023] In one embodiment, the transmitting the working data to the cloud storage server through the mounting service includes:

[0024] If the data size of the working data is greater than a preset threshold, the working data is divided at the master node to obtain a plurality of files to be processed;

[0025] Each of the to-be-processed files is transmitted to the cloud storage server through the mounting service.

[0026] In one implementation, the mount service startup parameters include a parallel transmission quantity parameter, a parallel check file quantity parameter, and a transmission block size parameter.

[0027] In a second aspect, the present invention provides a data storage scheduling migration device, comprising:

[0028] A creation module is used for starting the runtime through the POD program of the master node, and the master node creates a persistent volume and a persistent volume declaration, wherein the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage mode and the read-write mode of the persistent volume;

[0029] Prediction module, used to predict the current environment prediction parameters through AI model;

[0030] A startup module, used to receive current environment prediction parameters from the AI ​​model through a mounting tool, and start the mounting service according to the current environment prediction parameters;

[0031] A transmission module, used for transmitting the working data to a cloud storage server through the mounting service;

[0032] The acquisition module is used to acquire the working data from the cloud storage server when calling the POD program from the node.

[0033] In a third aspect, the present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the data storage scheduling migration method as described in the first aspect is implemented.

[0034] In a fourth aspect, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data storage scheduling migration method as described in the first aspect.

[0035] The data storage scheduling migration method, device, computer equipment and storage medium disclosed in the present invention, when the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program in the master node, and the persistent volume declaration is used to declare the storage method and read-write method of the persistent volume; the AI ​​model predicts the current environment prediction parameters; the mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters; the working data is transmitted to the cloud storage server through the mounting service; when the slave node calls the POD program, the working data is obtained from the cloud storage server. In this way, the mounting service of the mounting tool is started by using the current environment prediction parameters generated by the AI ​​model for the cloud native Kubernetes environment, and data is transmitted on the basis of the mounting service, thereby realizing automatic optimization of data synchronization speed and stability according to the current environment prediction parameters, effectively reducing data loss and synchronization failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0037] Figure 1 A schematic diagram of a process of the data storage scheduling migration method proposed in this embodiment is shown;

[0038] Figure 2 A schematic diagram of the master node and slave nodes proposed in this embodiment is shown;

[0039] Figure 3 Another schematic diagram of the process of the data storage scheduling migration method proposed in this embodiment is shown;

[0040] Figure 4 Another schematic diagram of the process of the data storage scheduling migration method proposed in this embodiment is shown;

[0041] Figure 5 Another schematic diagram of the process of the data storage scheduling migration method proposed in this embodiment is shown;

[0042] Figure 6 A schematic diagram of the structure of the data storage scheduling migration device proposed in this embodiment is shown.

[0043] Description of the accompanying drawings:

[0044] 600 - data storage scheduling migration device; 601 - creation module; 602 - prediction module; 603 - start module; 604 - transmission module; 605 - acquisition module. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0046] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0048] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0049] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0050] Example 1

[0051] In a cloud-native Kubernetes environment, users often customize directories directly on local machines and use them as external storage for Kubernetes applications, which can easily cause errors.

[0052] In addition, when using external storage, users often choose storage directories at random without fully considering whether they are suitable for their storage location and capacity requirements. This not only reduces the efficiency of log query and problem location, but also, once a problem occurs in the program, the existing system cannot perform intelligent migration or scheduling due to the lack of intelligent visualization and effective monitoring. The system also lacks intelligent fault analysis functions, and during the data synchronization process, it is unable to make intelligent adjustments due to changes in the network environment.

[0053] Therefore, the embodiments of the present disclosure provide a data storage scheduling and migration method, which is used to intelligently enable the mounting service of the mounting tool according to the environmental prediction parameters of the current cloud-native Kubernetes environment, and use the mounting service to realize data storage and scheduling migration.

[0054] See also Figure 1 A data storage scheduling migration method includes steps S101 to S105, and each step is described in detail below.

[0055] Step S101, when the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage method and read and write method of the persistent volume.

[0056] In this embodiment, in the cloud-native Kubernetes environment, when the POD program of the master node starts running, persistent storage is required to ensure the data persistence and portability of the POD program. Therefore, the master node defines and creates persistent volume claims (PVCs) to declare storage resource characteristics, and creates persistent volumes (PVs) based on the persistent volumes to store the working data of the POD program on the master node.

[0057] See also Figure 2 , the host machine 1 of the master node includes AutoAiMaster and Kubernetes master, the host machine 2 of the slave node includes AutoAiworker and Kubernetes worker, and the Kubernetes master and Kubernetesworker each include AutoAiOperator.

[0058] When the POD program works on host 1 of the master node and needs to create a persistent volume and a persistent volume declaration, the AutoAiOperator of host 1 will receive the creation notification and send the creation notification to the AutoAiMaster service for persistent volume and persistent volume declaration.

[0059] Step S102, the AI ​​model predicts the current environment prediction parameters.

[0060] In this embodiment, in order to adapt to the network environment requirements of the current cloud-native Kubernetes environment and the storage resource requirements of working data, it is necessary to use an AI model to predict the current environment prediction parameters of the cloud-native Kubernetes environment.

[0061] Step S103, the mounting tool receives current environment prediction parameters from the AI ​​model and starts the mounting service according to the current environment prediction parameters.

[0062] In this embodiment, the mounting tool obtains the current environment prediction parameters obtained according to the current network performance data from the AI ​​model, and starts the mounting service of the mounting tool according to the current environment prediction parameters. At this time, the mounting service is more suitable for the network environment and current storage resource requirements of the current cloud native Kubernetes environment.

[0063] Step S104: transmitting the working data to the cloud storage server through the mounting service.

[0064] In this embodiment, the cloud storage server can be mounted to a local or network file system through the mount service, thereby creating a virtual storage volume or directory. After the mount is successful, the working data can be dragged, copied or moved to the mounted storage volume or directory to achieve data transfer from the local to the cloud storage server.

[0065] Step S105, when the slave node calls the POD program, the working data is obtained from the cloud storage server.

[0066] In this embodiment, when the host machine of the master node has insufficient storage space or other abnormalities, AutoAiMaster will schedule tasks to other reliable slave nodes based on the idle information of the device, create new storage space, and obtain working data from the storage server for processing to achieve data synchronization. In this way, the current environment prediction parameters output by the AI ​​model are used to schedule data migration, adapt to changes in the network environment, improve the efficiency and accuracy of task execution, and reduce human intervention and failure dynamics caused by configuration errors or information loss, thereby solving the problems of slow data synchronization and synchronization disconnection.

[0067] See also Figure 3 In a specific embodiment, step S102 includes steps S1021 to S1022, and each step is described in detail below.

[0068] Step S1021, obtaining current network performance data.

[0069] In this embodiment, the current network performance data of the cloud native Kubernetes environment is first obtained. The current network performance parameters include but are not limited to bandwidth utilization, latency, packet loss rate, number of connections and other indicators. These data can be obtained through log files through network monitoring tools, and the data can be normalized and other feature processing can be performed.

[0070] Step S1022: The AI ​​model generates the current environment prediction parameters based on the current network performance data.

[0071] In this embodiment, the AI ​​model generates current environment prediction parameters based on current network performance data. The current environment prediction parameters describe the transmission parameter performance of the mounting tool when performing mounting services under given network conditions and parameter configurations in the future.

[0072] See also Figure 4 In a specific embodiment, before step S1022, steps S401 to S402 are included, and each step is described in detail below.

[0073] Step S401, obtaining historical network performance data and its corresponding historical mounting operation results.

[0074] In this embodiment, the AI ​​model needs to be trained before prediction is made using the AI ​​model. In order for the AI ​​model to select and evaluate the importance of key features that affect file transfer efficiency, it is necessary to obtain historical network performance data and its corresponding historical mount operation results.

[0075] Step S402: Based on a decision tree and a reinforcement learning algorithm, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model.

[0076] In this embodiment, based on a decision tree and reinforcement learning algorithm, historical network performance data and its corresponding historical mounting operation results, as well as other relevant data are used as input data of the AI ​​model for training to obtain a mature AI model. The mature AI model can be used to predict the current environmental parameters and the mounting service startup parameters of the mounting tool, and then used for the startup of subsequent mounting services.

[0077] See also Figure 5 In a specific embodiment, step S402 includes steps S4021 to S4022, and each step is described in detail below.

[0078] Step S4021: Based on the decision tree, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain historical environment prediction parameters.

[0079] In this embodiment, using the knowledge extracted by the decision tree from historical network performance data and its corresponding historical mount operation results, the AI ​​model is trained to predict the environmental changes that may occur under given network performance and mount operation conditions. These parameters may include expected network performance changes, the success rate of mount operations, etc.

[0080] Step S4022: Based on the reinforcement learning algorithm, the AI ​​model is trained according to the historical environment prediction parameters and the historical environment feedback corresponding to the historical mounting operation results to obtain the maturely trained AI model.

[0081] In this embodiment, the historical environment prediction parameters are used as one of the inputs of the reinforcement learning algorithm to help the model understand the environmental states that may be encountered under given conditions. The actual results such as the completion speed and error rate of the transmission task after the actual historical execution of the mount operation are used as historical environment feedback. These feedbacks are used as "rewards" or "penalties" in reinforcement learning to guide the model to learn which parameter configurations can optimize performance under specific network conditions.

[0082] By repeatedly trying different mount operation strategies and adjusting these strategies based on historical environment prediction parameters and actual historical environment feedback, the AI ​​model eventually learns how to make the best mount operation decisions under different conditions. This process continues until the model reaches a stable state, that is, its decisions no longer change significantly, or it reaches the predetermined training standard, at which point the AI ​​model is considered to have been trained to maturity.

[0083] In a specific embodiment, step S103 includes: the AI ​​model adjusts the mount service startup parameters of the mount tool according to the current environment prediction parameters; the mount service startup parameters include a parallel transmission number parameter, a parallel check file number parameter and a transmission block size parameter; the mount tool starts the mount service according to the mount service startup parameters.

[0084] In this embodiment, after the mature AI model is trained to generate the current environment prediction parameters, the optimal mount operation strategy is made based on the reinforcement learning algorithm under the current environment prediction parameters to adjust the mount service startup parameters of the mount tool. The mount service startup parameters include the number of parallel transfers (transfers), the number of parallel check files (checkers), and the drive-chunk-size parameter.

[0085] Exemplarily, when the network bandwidth is sufficient and the system resources allow, the number of parallel transmissions and the number of parallel inspection files can be increased to improve the transmission efficiency and inspection efficiency. At the same time, the appropriate transmission block size is selected according to the average size and access mode of the file to achieve the best performance and storage utilization.

[0086] Furthermore, the mount service startup parameters of the mount tool are started according to the mount service startup parameters, so that the complex Kubernetes environment can be easily monitored and controlled through natural language instructions, which significantly reduces the complexity of operations.

[0087] It should be noted that the operating performance of the mounting tool after the mounting service startup parameters are adjusted is continuously monitored to evaluate the effectiveness of the reinforcement learning strategy; and based on the feedback results of actual operations, the decision tree and reinforcement learning parts in the AI ​​model are continuously iterated and optimized.

[0088] It should be added that in order to improve the accuracy of AI model analysis, it is necessary to build a knowledge graph that integrates the information of various component entities involved in the system. This includes associations and causal relationships between entities such as "host and Kubernetes cluster", "host and application", "application and Kubernetes cluster", "host and MinIO", "application and MinIO", "program and storage directory", "Rclone and MinIO". These relationships are constructed in the form of triples or quadruples and stored in the NebulaGraph database. Such a knowledge graph design not only facilitates AI reasoning, but also greatly improves the depth and accuracy of data analysis.

[0089] In the process of building the knowledge graph, the existing ordinary two-dimensional relational table will be converted into a richer knowledge graph format. This conversion is achieved by using ETL tools such as Graphable or dbt, which can efficiently extract data and import it into the NebulaGraph knowledge graph database, optimize the data storage structure, and facilitate subsequent applications.

[0090] The Text2Cypher technology provided by NebulaGraph allows users to convert natural language queries into Cypher graph query statements, enabling large language models (such as chatglm) to understand and generate answers based on knowledge graphs, greatly enhancing the naturalness and efficiency of user interactions.

[0091] To enhance the automation capabilities of the system, an AI orchestrator can be built that integrates an AI language model. This model can combine natural language descriptions with API call relationships to automatically build and fine-tune the scheduling and migration relationships between storage and Kubernetes images or programs.

[0092] Specifically, through the prompt word template, the big model automatically generates a Dockefile file and calls the tool to build the image; then it automatically calls Kubernetes through the API to publish the image program for testing; in addition, the big model can also be used to adjust the storage used by the POD program or notify AutoAiMaster to schedule the program to a specified Kubernetes cluster and specify a pv storage server.

[0093] After confirmation, the AI ​​agent submits these construction and scheduling tasks to the master control service (AutoAiMaster), which is responsible for executing the assignment. In addition, when the AI ​​large model performs poorly in generating solutions or scripts, we use a strategy based on PPO (Proximal Policy Optimization) to fine-tune the model. This approach optimizes the performance of the model and ensures that the system responds more accurately and efficiently.

[0094] In a specific embodiment, step S104 includes: if the data size of the working data is greater than a preset threshold, the working data is divided at the master node to obtain multiple files to be processed; and each of the files to be processed is transmitted to the cloud storage server through the mounting service.

[0095] In this embodiment, if the data size of the working data is greater than a preset threshold, the working data is divided at the master node to obtain multiple files to be processed; each file to be processed is transmitted to the cloud storage server through the mounting service to improve the efficiency and reliability of data transmission and meet the user's needs for data storage and management.

[0096] The metrics plug-in is used to collect memory, CPU, and GPU storage resource information to determine server reliability. The network reliability is checked through iperf, and related programs are restarted.

[0097] It should be noted that a log intelligent analysis module can be set up, which is dedicated to collecting highly concerned log fragments in the system, including regular and abnormal logs. These log data are combined with the constructed knowledge graph information to generate a document containing complete information. Using GraphRAG technology and AI Agent, this module can be built into a workflow that can perform real-time analysis. In this way, logs are analyzed through the framework of the knowledge graph, which naturally has global data relevance, reduces the possibility of hallucinations in large model analysis, and improves the accuracy of analysis.

[0098] In addition, the log intelligent analysis module automatically generates analysis results, allowing users to quickly locate the problem without directly viewing the log, and can call the big model through Ai Agent to get the solution. These solutions and problem analysis results are displayed through a visual interface, which is convenient for users to intuitively understand and refer to, greatly improving the efficiency and accuracy of problem handling.

[0099] In addition, an AI intelligent visualization module can be set up, which includes a lightweight front-end AI model that users can download through a browser for local use. The server uses natural language prompt templates or instruction templates to render pages, and the information submitted by the user's browser is clearly passed to the back-end service after being processed by the front-end model. Users can modify control options through manual operations, choose to use different models or model templates to meet specific needs or optimize display effects, and even let the model directly render the original information without any preprocessing.

[0100] Users can also use natural language input to guide the AI ​​language model, showing the relationships between components and their workflows. These relationships are displayed on the interface, and users can use natural language to establish relationships between components. After user confirmation, these instructions will be submitted to AutoAiMaster for execution.

[0101] Please see again Figure 2 After creating the persistent volume and persistent volume declaration, the AutoAiMaster service will store the working data of the POD program on host 1 in the persistent volume, and start the AI ​​model's prediction of the current environment parameters through the instructions of AutoAiMaster to start the mounting service of the mounting tool. The mounting tool (rclone) stores the working data in the minio cloud storage server through the mounting service; on the other hand, it will distribute the directory information required to create the persistent volume to the AutoAiworker service of host 2, so that when the POD program is rescheduled to the slave node, the working data of the minio cloud storage server can be scheduled and migrated to local use.

[0102] Among them, Lua virtual monitoring is started to track the connection status of programs and storage directories on this machine, and submit this status information to AutoAiMaster.

[0103] AutoAiMaster distributes execution tasks to AutoAiworker, and AutoAiworker sends various information to AutoAiMaster for processing.

[0104] The data storage scheduling and migration method proposed in this embodiment is that when the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage method and read-write method of the persistent volume; the AI ​​model predicts the current environment prediction parameters; the mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters; the working data is transmitted to the cloud storage server through the mounting service; when the slave node calls the POD program, the working data is obtained from the cloud storage server. In this way, the mounting service of the mounting tool is started by using the current environment prediction parameters generated by the AI ​​model for the cloud native Kubernetes environment, and data is transmitted based on the mounting service, thereby realizing automatic optimization of data synchronization speed and stability according to the current environment prediction parameters, effectively reducing data loss and synchronization failures.

[0105] Example 2

[0106] In addition, the present disclosure provides a data storage scheduling migration device 600, see Figure 6 , the device comprises:

[0107] A creation module 601 is used for, when the POD program of the master node is started and runs, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage mode and the read-write mode of the persistent volume;

[0108] Prediction module 602, used to predict current environment prediction parameters through AI model;

[0109] A startup module 603 is used to receive current environment prediction parameters from the AI ​​model through a mounting tool, and start a mounting service according to the current environment prediction parameters;

[0110] A transmission module 604, configured to transmit the working data to a cloud storage server via the mounting service;

[0111] The acquisition module 605 is used to acquire the working data from the cloud storage server when the slave node calls the POD program.

[0112] Optionally, the prediction module 602 is used to obtain current network performance data; the AI ​​model generates the current environment prediction parameters based on the current network performance data.

[0113] Optionally, the prediction module 602 is used to obtain historical network performance data and its corresponding historical mounting operation results; based on a decision tree and a reinforcement learning algorithm, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model.

[0114] Optionally, the prediction module 602 is used to train the AI ​​model based on the decision tree, according to the historical network performance data and the historical mounting operation results, to obtain historical environment prediction parameters; based on the reinforcement learning algorithm, the AI ​​model is trained according to the historical environment prediction parameters and the historical environment feedback corresponding to the historical mounting operation results to obtain the maturely trained AI model.

[0115] Optionally, the startup module 603 is used to adjust the mount service startup parameters of the mount tool according to the current environment prediction parameters through the AI ​​model; the mount service startup parameters include a parallel transmission number parameter, a parallel check file number parameter and a transmission block size parameter; the mount tool starts the mount service according to the mount service startup parameters.

[0116] Optionally, the transmission module 604 is used to divide the working data at the master node to obtain multiple files to be processed if the data size of the working data is greater than a preset threshold; and transmit each of the files to be processed to the cloud storage server through the mounting service.

[0117] The device provided in the embodiment of the present disclosure can execute the steps of the data storage scheduling migration method provided in Example 1, which will not be described again to avoid repetition.

[0118] The data storage scheduling migration device proposed in this embodiment, when the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage method and read-write method of the persistent volume; the AI ​​model predicts the current environment prediction parameters; the mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters; transmits the working data to the cloud storage server through the mounting service; when the slave node calls the POD program, the working data is obtained from the cloud storage server. In this way, the mounting service of the mounting tool is started by using the current environment prediction parameters generated by the AI ​​model for the cloud native Kubernetes environment, and data is transmitted based on the mounting service, thereby realizing automatic optimization of data synchronization speed and stability according to the current environment prediction parameters, effectively reducing data loss and synchronization failures.

[0119] Example 3

[0120] In addition, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the data storage scheduling migration method described in Embodiment 1 is implemented.

[0121] The device provided in the embodiment of the present disclosure can execute the steps of the data storage scheduling migration method provided in Example 1, which will not be described again to avoid repetition.

[0122] Example 4

[0123] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the data storage scheduling migration method described in Embodiment 1 is implemented.

[0124] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0125] The computer-readable storage medium provided in this embodiment can implement the data storage scheduling migration method provided in Example 1, and will not be described again here to avoid repetition.

[0126] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0127] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0128] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A data storage scheduling migration method, characterized in that: include: When the POD program of the master node starts running, the master node creates a persistent volume and a persistent volume declaration, the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage mode and read-write mode of the persistent volume; The AI ​​model predicts the current environmental prediction parameters; The mounting tool receives the current environment prediction parameters from the AI ​​model, and starts the mounting service according to the current environment prediction parameters; Transmitting the working data to a cloud storage server through the mounting service; When the slave node calls the POD program, the working data is obtained from the cloud storage server.

2. The data storage scheduling migration method according to claim 1, characterized in that: The AI ​​model predicts the current environment prediction parameters, including: Get current network performance data; The AI ​​model generates the current environment prediction parameters based on the current network performance data.

3. The data storage scheduling migration method according to claim 2, characterized in that: Before the AI ​​model generates the current environment prediction parameter according to the current network performance data, the method includes: Obtain historical network performance data and its corresponding historical mount operation results; Based on a decision tree and a reinforcement learning algorithm, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model.

4. The data storage scheduling migration method according to claim 3, characterized in that: The decision tree and reinforcement learning algorithm are used to train the AI ​​model according to the historical network performance data and the historical mounting operation results to obtain a maturely trained AI model, including: Based on the decision tree, the AI ​​model is trained according to the historical network performance data and the historical mounting operation results to obtain historical environment prediction parameters; Based on the reinforcement learning algorithm, the AI ​​model is trained according to the historical environment prediction parameters and the historical environment feedback corresponding to the historical mounting operation results to obtain the maturely trained AI model.

5. The data storage scheduling migration method according to claim 1, characterized in that: The starting the mounting service according to the current environment prediction parameter includes: The AI ​​model adjusts the mounting service startup parameters of the mounting tool according to the current environment prediction parameters; The mounting tool starts the mounting service according to the mounting service startup parameters.

6. The data storage scheduling migration method according to claim 5, characterized in that: The mount service startup parameters include a parallel transmission quantity parameter, a parallel check file quantity parameter and a transmission block size parameter.

7. The data storage scheduling migration method according to claim 1, characterized in that: The step of transmitting the working data to the cloud storage server through the mounting service includes: If the data size of the working data is greater than a preset threshold, the working data is divided at the master node to obtain a plurality of files to be processed; Each of the to-be-processed files is transmitted to the cloud storage server through the mounting service.

8. A data storage scheduling migration device, characterized in that: include: A creation module is used for starting the runtime through the POD program of the master node, and the master node creates a persistent volume and a persistent volume declaration, wherein the persistent volume is used to store the working data of the POD program on the master node, and the persistent volume declaration is used to declare the storage mode and the read-write mode of the persistent volume; Prediction module, used to predict the current environment prediction parameters through AI model; A startup module, used to receive current environment prediction parameters from the AI ​​model through a mounting tool, and start the mounting service according to the current environment prediction parameters; A transmission module, used for transmitting the working data to a cloud storage server through the mounting service; The acquisition module is used to acquire the working data from the cloud storage server when the POD program is called from the node.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the data storage scheduling migration method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the data storage scheduling migration method as described in any one of claims 1 to 7.

Images