Graph import module synchronization operation interface data transmission system, optimization method and device

Abstract

The application discloses a kind of figure guide import module synchronous operation interface data transmission system, optimization method and equipment, belong to interface optimization technical field, based on interface data transmission system executes corresponding following optimization method operation, including: receiving image-import request and detecting source and target storage type;If all are Ceph RBD then call RBD export relevant logic, otherwise with native logic to local file read-write to transmission file;Call RBD import and import to target Ceph cluster;Update specified metadata, clean up temporary data when abnormal, process is all carried out log record and result return, finally synchronization is completed.The application can improve the efficiency and reliability of image synchronization in a multi-backend environment, especially in a Ceph RBD-dominated environment, solving the technical problems of low performance, long time consumption, high failure rate, and inability to fully utilize the native capabilities of Ceph RBD during image synchronization.

Description

Technical Field

[0001] This invention relates to the field of Glance interfacing storage interface optimization technology, specifically to an interfacing storage interface data transmission system for synchronous operation of the image-import module of OpenStack-Glance and Ceph-RBD block storage, an optimization method for the corresponding interface data transmission system, and a computer device for optimizing cloud platform image interfaces. Background Technology

[0002] OpenStack Glance is an image service component of the open-source cloud platform OpenStack, responsible for the registration, storage, querying, and distribution of virtual machine images.

[0003] In cloud computing environments, Glance typically interfaces with various backend storage systems, including object storage (such as CephRADOS), block storage (such as Ceph RBD), file systems, and drivers provided by other third-party storage vendors. In typical deployments, especially in carrier-grade cloud platforms, Glance often adopts a multi-backend configuration model, where different cloud pools can simultaneously use multiple storage drivers (such as Ceph RBD, centralized file storage, third-party vendor drivers, etc.). Among these, Ceph RBD (RADOSBlock Device), as a high-performance distributed block storage, is widely used in image management.

[0004] In existing technologies, Glance mainly adopts a filesystem-based data transfer process when performing image-import synchronization operations: reading image data from the source backend and exporting it to a temporary directory on the local file system; writing the data from the local file system to the target backend storage; and performing additional verification and validation on the data (such as os_hash_value, checksum, etc.) after synchronization is completed.

[0005] However, while this file system relay method has good versatility (applicable to various backend types), it still has the following significant problems when interfacing with Ceph RBD:

[0006] (1) The data flow path is lengthy: multiple read and write operations are required on the local disk (reading temporary files, writing temporary files, reading temporary files again, and writing to the target storage), which increases I / O latency;

[0007] (2) Significant performance bottleneck: When synchronizing multiple backends or large-scale mirrors, the transmission time is long, especially when synchronizing between CephRBD storage backends, the local file system becomes the bottleneck.

[0008] (3) Additional verification process: Since the data is first written to disk and then transmitted, additional verification and validation are required after synchronization is completed, which further prolongs the overall time consumption;

[0009] (4) Many potential failure points: The temporary files of the file system may fail to synchronize during transmission due to insufficient space, file system errors or transmission interruption. There are few log records, and the dirty data generated when failure occurs cannot be automatically cleaned up, which increases the complexity of operation and maintenance.

[0010] Furthermore, while existing optimization solutions for Ceph RBD scenarios have proposed methods to directly use RBD client tools (such as RBD export and RBD import) to achieve cross-cluster data migration, and utilize Ceph's end-to-end data consistency and verification mechanisms by directly transmitting data at the RADOS object storage layer, thus eliminating the need for file system relay, the current official Glance implementation and the latest version of the community code repository do not yet have deep integration with the image-import interface. This can easily lead to the inability to directly call the native RBD interface in the official process of multi-backend image synchronization. In addition, it also lacks automated coordination with Glance metadata processing (os_hash_value, checksum, etc.) and mechanisms for residual data cleanup and logging in abnormal scenarios.

[0011] Therefore, this application proposes a data transmission system, optimization method, and device for a synchronous operation interface of a graph import module to solve the above-mentioned technical problems. Summary of the Invention

[0012] The main objective of this invention is to provide a data transmission system, optimization method, and device for the synchronous operation interface of a graph import module, which can improve the efficiency and reliability of image synchronization in multi-backend environments, especially in Ceph RBD-dominated environments, in order to solve the technical problems of low performance, long time consumption, high failure rate, and inability to fully utilize the native capabilities of Ceph RBD in the OpenStack Glance image synchronization process mentioned in the background art.

[0013] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0014] A graph import module synchronous operation interface data transmission system, wherein the following associated system modules are stored in the computer device:

[0015] The image source backend storage module is used to store the virtual machine image metadata that needs to be synchronized. It supports Ceph RBD, file system storage and other third-party driver storage methods. The synchronized virtual machine image metadata is sent to the image target backend storage module.

[0016] The Glance image management module is used to process image metadata, receive image-import requests initiated by users or the system, and coordinate the data transmission process.

[0017] The data transfer module is used to perform RBD data import / export and file system data transfer operations;

[0018] The exception handling and dirty data cleanup module is used to clean up local temporary files and incomplete target image objects when an interruption or exception is detected during the image metadata synchronization process, so as to avoid data residue affecting subsequent operations.

[0019] The logging and monitoring module is used to record detailed logs for key steps (export, import, verification) specified in the above modules, including operation time, dirty data processing information, and different levels of LOG records for capturing the cause of anomalies, which facilitates subsequent investigation.

[0020] In the Glance multi-backend image synchronization scenario, if the source and target are specified scenarios, the data transmission module is used to replace the original file system-based image-import data import and export mechanism with RBD export and RBDimport, and allows the use of the RBD interface during image synchronization to improve data migration efficiency.

[0021] Preferably, the data transmission module further includes:

[0022] The RBD export submodule is used to directly export the underlying data blocks of the image from the source cluster to the transit cache file when both the source and target backends are detected to be Ceph RBD.

[0023] The file system transfer submodule is used to retain the original file system transfer method in non-Ceph RBD or hybrid backend scenarios to ensure compatibility.

[0024] The RBD import submodule is used to write data blocks directly into the RBD image object corresponding to the target cluster when the target backend is transformed into Ceph RBD.

[0025] On the other hand, the present invention also discloses an optimization method for the synchronous operation interface of a graph import module, which is implemented based on any of the graph import module synchronous operation interface data transmission systems described above, and performs the following steps through a computer device:

[0026] Step S1. After receiving the image-import request, Glance parses the image metadata, detects and determines the storage type of the source backend and the target backend. If both the source and the target are Ceph RBD backends, the RBD optimization path in the following steps is entered; otherwise, the compatibility path in the original file system is entered, and the native logic is used to read and write local files to the transferred files.

[0027] Step S2. Call the native RBD export interface of RBD to stream the underlying data objects of the corresponding image in the source cluster and output them directly to the pipeline or transit buffer. Generally, the transit buffer is a configurable memory buffer or temporary file buffer.

[0028] Step S3. Call the RBD import interface to directly write the data object stream into the RBD image object of the target Ceph cluster, and use Ceph's built-in consistency and verification mechanism to complete the data persistence to disk.

[0029] Step S4. Update the specified metadata fields such as os_hash_value, checksum, locations, and stores of the image to ensure that the target backend image can be directly used by the compute nodes;

[0030] Step S5. If a transmission interruption or error is detected, perform temporary data cleanup to ensure that there are no incomplete image remnants on the local machine and the target backend.

[0031] Step S6. Record detailed information about this synchronization, including transmission time, data volume, verification results, etc., and return the execution status.

[0032] Preferably, when the image source or image target is a volume, the target storage can be selected by supporting volume type or specifying header information. Each time the client creates a virtual machine system disk, the Ceph cluster backend storage pool that is the node where the current image of Glance is located is selected. At this time, the image does not need to be synchronized and can support cloning the system disk in seconds. This is suitable for scenarios with frequent volume image conversion and reduces redundant data transfer.

[0033] Preferably, while maintaining the existing workflow, the performance and stability of cross-network transmission can be improved by adopting ultra-high-performance hardware and adjusting configurations including file system connection timeouts, so as to match the performance of the native RBD interface. This approach requires minimal changes, has low risk, and is easy to implement, but it is difficult to find hardware performance that can make the file system layer read and write operations match the RBD interface.

[0034] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0035] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0036] As can be seen from the above technical solution, the present invention provides a data transmission system, optimization method, and device for a synchronous operation interface of a graph import module. Compared with the prior art, the present invention has the following advantages:

[0037] 1. This invention, by employing the native RBD interface for direct block-level transmission, can eliminate redundant I / O operations in the file system, reduce synchronization time, and improve transmission efficiency.

[0038] 2. By setting up an end-to-end verification mechanism in Ceph RADOS, this invention can avoid additional verification steps, reduce transmission failure rate, and enhance reliability.

[0039] 3. By introducing RBD native export and import capabilities into the synchronization process, and combining exception handling and logging mechanisms, this invention can improve image synchronization efficiency, reduce redundant data transmission steps, and lower the failure rate during the synchronization process, thereby achieving efficient and reliable image migration.

[0040] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0041] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0042] Figure 1 This is a schematic diagram of the overall module framework for data transmission processing in the system of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall optimized operation process of the method of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] For details in the embodiments, please refer to Figures 1 to 2 .

[0046] like Figure 1 As shown, the graph import module synchronous operation interface data transmission system proposed in this embodiment of the invention includes the following data processing modules:

[0047] (1) Image source backend storage module: used to store virtual machine image metadata that needs to be synchronized, supporting CephRBD, file system storage and other third-party driver storage methods.

[0048] (2) Glance image management module: It is responsible for processing image metadata, receiving image-import requests initiated by users or the system, and coordinating the data transmission process.

[0049] (3) Data transmission module, specifically including:

[0050] (a) RBD export submodule: When both the source and target backends are detected to be Ceph RBD, the RBD export interface is called to directly export the underlying data blocks of the image from the source cluster to the transit cache file.

[0051] (b) File system transfer submodule: In non-Ceph RBD or hybrid backend scenarios, the original file system transfer method is retained to ensure compatibility.

[0052] (4) RBD import submodule: When the target backend is Ceph RBD, the RBD import interface is called to directly write the data block into the RBD image object corresponding to the target cluster.

[0053] In the example, after the RBD import is complete, the system can automatically update the Glance image metadata fields (such as os_hash_value and checksum), using Ceph RADOS' native end-to-end checksum mechanism to ensure image data consistency without the need for additional secondary verification.

[0054] (5) Exception handling and dirty data cleanup module: When an interruption or exception is detected during the synchronization process, it automatically cleans up local temporary files and incomplete target image objects to avoid data residue affecting subsequent operations.

[0055] (6) Log and monitoring module: used to record detailed logs at key steps (export, import, verification), including operation time, dirty data processing information and abnormal reason capture LOG different levels of records, which are convenient for subsequent investigation.

[0056] The above modules transmit instructions and interact with data through a data bus. The Glance image management module is the core coordination unit, which triggers and schedules other modules to perform corresponding operations.

[0057] In the example, the entire process of image processing and importing to the target Ceph RBD cluster, led by the Glance image management module, is executed sequentially by various functional modules according to business logic. At the same time, a dedicated bottom module provides full-process anomaly control, data governance, and operation monitoring support. The specific execution logic is as follows:

[0058] After the process starts, the Glance image management module first receives the image-related requests sent by the client, and completes the core work of metadata processing and scheduling for the requests. It also accepts image-import requests to clarify the execution logic for the subsequent processing, transmission and import of image data.

[0059] After completing metadata and scheduling, the request is transferred to the backend storage module of the image source. This module has multi-driver compatibility and can connect to Ceph RBD, local file system Filesystem and other third-party storage drivers, thereby realizing the adaptation and connection of different types of image sources.

[0060] Based on the adaptation results of the mirror source backend storage module (glance store), the data transmission module initiates the mirror data transmission operation and synchronously sends it to the mirror target backend storage module. For different source storage types, the mirror data transmission is completed by either RBD export or file system transmission.

[0061] The process involves the RBD export submodule calling `rbd export` to directly read RADOS objects. Then, the file system transfer submodule performs fallback compatibility by writing / reading local files. Finally, after the image data is transmitted, the RBD import submodule receives and processes it, calling `rbd import` to directly write it into RADOS objects. This completes the image import operation to the target Ceph RBD cluster. After the image is imported, the target backend storage module (glance store) completes the integration with Ceph RBD, the local file system (Filesystem), and other third-party storage drivers.

[0062] Throughout the entire execution of the aforementioned core business processes, the bottom-level exception handling and dirty data cleanup modules, as well as the logging and monitoring modules, run continuously and synchronously.

[0063] A. The exception handling and dirty data cleanup module can promptly identify and handle various exceptions that occur in the process, while cleaning up the dirty data generated in the process, rolling back incomplete objects, and ultimately ensuring the integrity of the mirrored data and the stable progress of the business process.

[0064] B. The logging and monitoring module records the operation logs of each module, monitors the execution status of the process, and records the operation indicators of each module (including the entire mirror synchronization process from accepting requests, exporting, and importing), providing comprehensive data support for troubleshooting and optimizing the process.

[0065] The optimized synchronization path here is further enhanced by adding detailed logging and anomaly detection points, covering transmission progress, error details, resource cleanup actions, etc., which can facilitate subsequent operation and maintenance troubleshooting and performance analysis.

[0066] Furthermore, in a further example, if the synchronization task encounters an anomaly (such as network interruption or target end write failure), the system can automatically delete the temporary data blocks generated during the export or import process to prevent storage space waste and subsequent synchronization conflicts.

[0067] In summary, this system directly transmits at the block level through the native RBD interface, eliminating redundant I / O operations in the file system, reducing synchronization time, and improving transmission efficiency; it also utilizes the Ceph RADOS end-to-end verification mechanism to avoid additional verification steps, reduce transmission failure rate, and enhance reliability.

[0068] In a specific application scenario, this application independently verified the cross-backend synchronization of a 50GB image. By comparing the image synchronization operations and effects before and after optimization, the practicality and superiority of the solution were verified. The specific verification process is as follows:

[0069] Before optimization, the image synchronization verification was performed in the OpenStack environment using the Glance image import synchronization operation. The core configuration parameters were: allow failure set to False, image import method was copy-image, and full synchronization was performed to the storage side. The synchronized image ID was 3b2e58db-0c34-4634-9f25-cc75ded68 (subsequently associated as 3b2e5db-c34-4634-9f25-cc7ed68c00), the associated storage identifier was ceph_172_25916, the image source file was zd-test-img1.raw, and the actual storage size was 53687091206 bytes. The synchronization task was created on 2025-05-27T08:49:39Z, updated on 2025-05-27T09:10:02Z, and expired on 2025-05-29T09:10:02Z. According to actual testing, the entire process of synchronizing this 50G image across backends took 20 minutes.

[0070] The optimized image synchronization verification first performs a Glance storage location deletion operation, specifying the associated path of the RBD storage as rbd: / / d2Bbd6e-f7d-4c91-ba01-42144450f3d / images / 3b2e58db-c34-4634-9f25-c7ed58c0 / sap, and then performs a cross-backend synchronization operation of the same 50G image. The image source file synchronized this time is still zd-test-img1.raw, the associated identifier is 1c78872ff74fa3b193ab89133146b6, and the actual storage size of the image is 53687091200 bytes. Viewing the information of the task corresponding to this synchronization (ID: 826ab72d-f1ef-4b99-a1fa-8d989498e9ed), its update time is 2025-05-27T09:25:53Z, the operation type is api imageimport, and the actual cross-backend synchronization of the 50G image only took 8 minutes.

[0071] After verifying the comparison before and after the above optimization, the actual effect and compatibility of the solution were fully verified. The results show that the solution can stably complete cross-backend image synchronization and automatically clean up intermediate data in case of abnormalities, ensuring image consistency and availability. For example, when integrated into the OpenStack cloud platform base system for deployment, it can effectively solve the technical problems of low efficiency and imperfect abnormality handling mechanism of existing multi-backend image synchronization. Alternatively, it is compatible with the existing multi-storage, import workflow, and RBD image management mechanisms in the OpenStack community, and has good scalability and versatility. It can be continuously promoted and applied in subsequent iterations of the OpenStack cloud platform and in real business scenarios with multiple clusters and cross-regional operations.

[0072] Furthermore, in specific usage examples, this solution also possesses a technical foundation compatible with the development direction of the Ceph community, enabling smooth integration with the native replication and verification capabilities of higher versions of Ceph.

[0073] Furthermore, in a further example, this application can also employ the following steps to verify the effect of mirror synchronization time optimization:

[0074] Step A1: Set up a dual Ceph-RBD backend storage environment including Ceph_172_75_23_10 and Ceph_172_25_9_16. Select the 50G zd-test-img1.raw image as the test object, perform cross-backend synchronization operation using the native copy-image method, and track the task status using the glance task-show command. The original logic recorded that the synchronization of the 50G image took 20 minutes.

[0075] Step A2: Optimize the core logic of file export for image synchronization. First, restore the test image to a state where it only exists in the default ceph-rbd backend. Then, perform the same copy-image cross-backend synchronization operation, view and record the task execution time. After optimization, the synchronization time of the 50G image is reduced to 8 minutes.

[0076] Step A3: Based on the optimization of the file export method, the redundant verification and hash value processing logic after the image is imported into the ceph-rbd backend is removed. The 50G zd-test-img6.raw image is selected to perform cross-backend synchronization operation, the task is tracked and the time is recorded. After optimization, the synchronization time of the 50G image is further compressed to 5 minutes.

[0077] In summary, it can be seen that by specifically optimizing the file export method and reducing redundant verification logic after importing the ceph-rbd backend, this application can achieve a step-by-step reduction in the synchronization time of large-scale images between ceph-rbd backends. The synchronization time of a 50G image across backends is gradually reduced from 20 minutes to 5 minutes, and the synchronization efficiency is greatly improved.

[0078] Furthermore, in a further example, this application may also employ the following steps to verify the integrity and scene compatibility of the mirror synchronization function:

[0079] Step B1: Configure three ceph-rbd multi-pool backend storage environments: ceph_172_75_23_10, ceph_172_25_9_16, and ceph_v_9_16. Create a 50GB zd-test-img2.raw image and ensure that it initially exists only in the default backend. Perform a full synchronization operation and verify that the image can be stably synchronized to all configured ceph-rbd multi-pool backends using the glance task-show and glance image-show commands.

[0080] Step B2: After the synchronization of multiple ceph-rbd backends is completed, delete the mirror location and corresponding data of the ceph_172_25_9_16 backend using the glance location-delete command, re-execute the cross-backend synchronization operation, and verify that the mirror can complete the recovery synchronization of the backend, thus realizing the repair of the deleted backend mirror data.

[0081] Step B3: Add a non-Ceph-RBD Cinder backend to the existing Ceph-RBD backend to build a hybrid Ceph-RBD and non-Ceph-RBD storage backend environment. Select a 50G zd-test-img6.raw image to perform a full synchronization operation to verify that the Ceph-RBD backend still maintains the optimized high-efficiency synchronization speed and that the non-Ceph-RBD backend can complete the synchronization normally with the original logic.

[0082] Step B4: Configure and bind exclusive Cinder volume types for different ceph-rbd backends, create cloud disks and deploy virtual machines based on the default ceph-rbd backend and the synchronized ceph-rbd backend images respectively, log in to the virtual machine system through the virsh console command, verify that all virtual machines can enter the file system normally, and ensure the actual business availability of the synchronized image;

[0083] Step B5: Construct a test environment with insufficient local storage space. Confirm that the available local space is 32GB and select a 50GB image as the test object. Disable the glance-api service on other nodes to ensure that the synchronization task is scheduled to the target node. Perform cross-backend synchronization operations to verify that the system can accurately capture storage space shortage anomalies and avoid file system crashes caused by anomalies. In summary, it can be seen that the image synchronization optimization solution of this application can be adapted to synchronization scenarios of ceph-rbd multi-pool backends and ceph-rbd and non-ceph-rbd mixed backends. It supports the recovery synchronization of deleted backends. The synchronized image can normally support actual business operations such as virtual machine deployment. At the same time, it has the ability to accurately capture storage space shortage anomalies, realizing the integrity of the synchronization function, scenario compatibility, and system operation security.

[0084] In addition, the test environment in the above embodiments is uniformly configured: the Linux kernel, Netfilter framework, Kubernetes cluster, OpenStack version, and Ceph version are all existing versions, the number of namespaces is set to 3 (server, router, client), and the kernel module loading command is modprobe nf_conntrack nf_conntrack_netlink.

[0085] On the other hand, such as Figure 2 As shown, this invention also discloses an optimization method for the synchronous operation interface of a graph import module, which is executed by a computer device and includes the following steps:

[0086] Step L1. Request Reception and Environment Detection: Glance receives the image-import request, detects and parses the image metadata, determines the storage type of the source backend and the target backend, and confirms whether they are both Ceph RBD.

[0087] If both the source and destination are Ceph RBD backends, enter the RBD optimization path; otherwise, enter the compatibility path via the original file system and fall back to using only the file system to write / read local temporary files.

[0088] Step L2. RBD Data Export (rbd export): Call the native RBD export interface to read the underlying object data of the corresponding image in the source RBD cluster and export it directly to the pipeline or transit buffer.

[0089] Step L3. RBD Data Import (rbd import): Call the RBD import interface to write the data object stream directly into the RBD image object of the target cluster, and use Ceph's built-in consistency and verification mechanism to complete the data persistence to disk.

[0090] Step L4. Metadata Update: Update the image's metadata fields such as locations, stores, os_hash_value, and checksum to ensure that the target backend image can be directly used by compute nodes.

[0091] Step L5. Exception Handling and Data Cleanup: If the transmission is interrupted or an exception failure error is detected, perform temporary data cleanup and clean up incomplete objects to ensure that there are no incomplete image residues on the local machine and the target backend.

[0092] Step L6. Log recording and result return: Record detailed information about this synchronization, including transmission time, data volume, verification results, etc., and return the execution status;

[0093] The synchronization was finally completed.

[0094] In summary, this method, in Glance multi-backend image synchronization scenarios, replaces the original file system-based image-import data import / export mechanism with RBD export and RBD import, and allows the use of the RBD interface during image synchronization. This enables efficient data migration, and the sparse export significantly reduces transmission overhead. Furthermore, before the synchronization process executes, it automatically detects the storage types of the source and target ends. If both are Ceph RBD backends, it enables the RBD direct connection optimization path; if a non-RBD driver exists, it falls back to the original file system transmission path, further ensuring the compatibility and stability of the method execution process.

[0095] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0096] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0097] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the graph import module synchronization operation interface optimization method in the above embodiment based on the graph import module synchronization operation interface data transmission system in the above embodiment.

[0098] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0099] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus.

[0100] Memory, used to store computer programs;

[0101] When the processor executes the program stored in the memory, it implements the above-described graph import module synchronous operation interface optimization method based on the graph import module synchronous operation interface data transmission system in the above embodiments.

[0102] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0103] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0104] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0105] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0106] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0108] Furthermore, it should be noted that the technical solutions proposed in the above embodiments are based on OpenStack Glance open-source components (e.g., version Wallaby) and Ceph RBD distributed storage (e.g., version 16.2.7). All core technologies involved are mature, open-source technologies that are publicly available, and there is no reliance on private interfaces or undisclosed protocols. Key operations in the technical solutions, such as "RBDexport / import interface calls," "metadata updates," and "abnormal data cleanup," are all implemented through standard commands of the open-source toolchain (RBD client, Glance API), specifically including directly executable commands such as `rbd export --format raw` and `rbdimport --mage-feature layering`, with clear and unambiguous operation steps. The 50G image synchronization test data provided in the embodiments (20 minutes before optimization and 8 minutes after optimization) are the average values ​​of three repeated tests in a dual Ceph RBD backend environment (3 storage pool replicas, 16GB cache size), and the data is authentic and reproducible. Details not specifically defined in this invention (such as the specific configuration parameters of the transit buffer) can be derived through conventional techniques in the art (dynamically adjusting the buffer capacity according to the image size), without causing ambiguity in the technical solution. This invention ensures that those skilled in the art can clearly understand and fully implement this technical solution by clearly defining the source / target storage type detection logic, the switching conditions for RBD and file system transmission paths, and the exception handling triggering mechanism.

[0109] Furthermore, those skilled in the art should understand that in the actual use of the embodiments of this application, there may be preset thresholds used as the basis for judging the corresponding technical solutions. These thresholds are conventional technical means commonly used in the field to implement functions such as state judgment, condition recognition, and control logic switching. The specific values, setting basis, value selection methods, determination methods, and adjustment rules of the thresholds involved in this technical solution are all conventional technical choices that can be reasonably determined by those skilled in the art based on conventional technical factors such as actual application scenarios, system working states, characteristics of the detection object, hardware performance parameters, and functional requirements, through conventional experiments, calibrations, and debugging. The specific setting and adjustment of the aforementioned thresholds will not cause this technical solution to be unimplementable as a whole, nor will it affect the realization of the core concept and the achievement of the technical effects of this technical solution.

[0110] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A synchronous operation interface data transmission system for a graph import module, characterized in that, include: The image source backend storage module is used to store the virtual machine image metadata that needs to be synchronized. The synchronized virtual machine image metadata is sent to the image target backend storage module. The Glance image management module is used to process image metadata, receive image-import requests initiated by users or the system, and coordinate the data transmission process. The data transfer module is used to perform RBD data import / export and file system data transfer operations; The exception handling and dirty data cleanup module is used to clean up local temporary files and incomplete target image objects when an interruption or exception is detected during the image metadata synchronization process, so as to avoid data residue affecting subsequent operations. In the Glance multi-backend image synchronization scenario, if the source and target are specified scenarios, the data transmission module will replace the original file system-based image-import data import and export mechanism with RBD export and RBD import, and allow the use of the RBD interface during image synchronization to improve data migration efficiency. The data transmission module is also equipped with: The RBD export submodule is used to directly export the underlying data blocks of the image from the source cluster to the transit cache file when both the source and target backends are detected to be Ceph RBD. The file system transfer submodule is used to retain the original file system transfer method in non-Ceph RBD or hybrid backend scenarios to ensure compatibility. The RBD import submodule is used to write data blocks directly into the RBD image object corresponding to the target cluster when the target backend is transformed into Ceph RBD.

2. A method for optimizing the synchronous operation interface of a graph import module, implemented based on the data transmission system for the synchronous operation interface of a graph import module as described in claim 1, characterized in that... include: Step S1. After receiving the image-import request, Glance parses the image metadata, detects and determines the storage type of the source backend and the target backend. If both the source and the target are Ceph RBD backends, it proceeds to the RBD optimization path in the following steps; otherwise, it proceeds to the compatibility path via the original file system. Step S2. Call the native RBD export interface of RBD to stream the underlying data objects of the corresponding image in the source cluster and output them directly to the pipeline or transit buffer. Step S3. Call the RBD import interface to directly write the data object stream into the RBD image object of the target Ceph cluster, and use Ceph's built-in consistency and verification mechanism to complete the data persistence to disk. Step S4. Update the specified metadata fields of the image to ensure that the target backend image can be directly used by the compute nodes; Step S5. If a transmission interruption or error is detected, perform temporary data cleanup to ensure that there are no incomplete image remnants on the local machine and the target backend. Step S6. Record the detailed information of this synchronization and return the execution status; In the method, when the image source or image target is a volume, the target storage is selected by supporting volume type or specifying header information. Each time a client creates a virtual machine system disk, the Ceph cluster backend storage pool that is the node where the current image of Glance is located is selected.

3. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the graph import module synchronization operation interface optimization method as described in claim 2.

Images