User and data life cycle server management system for laboratory collaboration environments

Abstract

The application belongs to the technical field of computer server management, and specifically discloses a user and data full life cycle server management system for a laboratory collaborative environment, which comprises a container virtualization layer, a heterogeneous transaction coordination layer and a data collaboration ecological layer, and a metadata management layer.The container virtualization layer uses Docker containers as user personal computing environment carriers and supports dynamic resource allocation through a "state saving-resource reconfiguration-environment recovery" mechanism.The heterogeneous transaction coordination layer coordinates heterogeneous systems in a Saga transaction mode and realizes cross-system transaction consistency.The data collaboration ecological layer establishes typed management of personal data volumes, shared software environment volumes and public data set volumes, supports software environment encapsulation sharing and version management, and realizes real-time updating and high-performance querying of metadata based on a hot updating architecture of Redis.The application solves the problems of inflexible resource allocation, incomplete environment isolation, permission management dilemma, and difficult data collaboration and recovery.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer server management, and particularly relates to a user and data full life cycle server management system for a laboratory collaboration environment. BACKGROUND

[0002] In the laboratory environment of colleges, research institutes and other institutions, servers are the core infrastructure for carrying out computationally intensive tasks. With the increase in the number of researchers and the diversification of projects, traditional server management modes face serious challenges.

[0003] The core defect of the direct host user management mode lies in the ambiguity of the permission boundary. The system administrator faces a fundamental dilemma in sudo permission allocation: not granting sudo permission results in users being unable to install necessary development tools and dependent libraries, severely limiting the usability of the experimental environment; granting sudo permission faces the dual risks of system stability and security. This permission management mode cannot essentially find a balance point between user autonomy and system security, and lacks effective resource isolation mechanisms, resulting in frequent resource competition and environmental pollution problems among users.

[0004] The system-level virtualization scheme based on LXC / LXD provides better isolation, but its technical architecture determines the applicability limitations in the laboratory environment. Each LXC instance requires a complete operating system environment, resulting in significant increases in storage overhead and startup time; the configuration of GPU resources is complex and lacks flexibility, making it difficult to achieve dynamic hardware resource allocation; more importantly, LXC / LXD lacks a collaboration-oriented design concept and cannot effectively support resource sharing and collaboration needs in the laboratory environment.

[0005] The problem with the Kubernetes cluster scheme is that the design concept fundamentally does not match the application scenario. The architecture design of K8s takes the high availability of multi-node clusters as the core goal, and its complex component system is too redundant in single-machine or small-scale laboratory environments. The resource scheduling of K8s is at the node granularity, which cannot achieve precise resource allocation in single-machine environments, which contradicts the need for precise resource control in laboratory environments. SUMMARY

[0006] The purpose of the present application is to solve the problems of existing resource allocation inflexibility, incomplete environment isolation, permission management dilemma, and data collaboration and recycling difficulties, and to propose a user and data full life cycle server management system for a laboratory collaboration environment.

[0007] The technical solution of the present application is: a user and data full life cycle server management system for a laboratory collaboration environment, comprising:

[0008] A container virtualization layer is used to redefine the Docker container as a user personal computing environment carrier, strictly limits user operation within the container boundary, guarantees complete root permission of the user inside the container, and realizes fine-grained management of CPU resource allocation accuracy of 0.01 core, memory allocation accuracy of 1MB, and network bandwidth control accuracy of 1Kbps.

[0009] A heterogeneous transaction coordination layer is used to coordinate the heterogeneous infrastructure system of the Docker container, the ZFS / Btrfs storage system and the Redis cache through the Saga transaction mode, and realizes transaction consistency guarantee across systems.

[0010] A data collaboration ecological layer is used to build a typed management mechanism of a personal data volume, a shared software environment volume and a public data set volume, and realizes unified management and flexible sharing of the laboratory environment.

[0011] A metadata management layer is used to realize real-time updating and high-performance querying of metadata based on the hot updating architecture of Redis.

[0012] The beneficial effects of the present application are:

[0013] 1. The present application effectively alleviates the contradiction between the requirement of users for complete system control and the requirement of the system for security isolation through the improvement of the container application mode.

[0014] 2. The present application improves the deficiency of the container technology in resource elasticity expansion, solves the limitation of the traditional container technology that cannot dynamically adjust the hardware configuration, and realizes dynamic adjustment of the hardware configuration.

[0015] 3. The present application applies the Saga transaction mode to the coordination and management of the heterogeneous infrastructure system, realizes transaction consistency guarantee across systems, guarantees the traceability of transactions, and provides a data basis for fault analysis and system optimization.

[0016] 4. The present application builds a software environment sharing mechanism at the laboratory level, improves the traditional software installation and environment configuration mode, and a complex software environment configured by a user for several hours or even several days can be completely reused by other users within a few minutes through the data volume sharing mechanism, thereby improving the overall work efficiency of the laboratory.

[0017] 5. The present application realizes timely allocation and recovery of user resources, improves the problem of resource abuse and unreasonable allocation in the laboratory environment through the integral billing system, improves the fairness and rationality of resource use, and the response time of integral updating operation is controlled within 10 milliseconds. DETAILED DESCRIPTION

[0018] Figure 1This is an overall architecture diagram of a server management system for the entire lifecycle of users and data in a laboratory collaborative environment.

[0019] Figure 2 A flowchart for container state persistence and dynamic resource allocation.

[0020] Figure 3 This is a diagram of the Saga transaction coordination architecture for heterogeneous systems.

[0021] Figure 4 This is a flowchart for intelligent compensation decision-making.

[0022] Figure 5 This is a diagram of the data volume type management architecture.

[0023] Figure 6 Diagram of permission inheritance and dynamic verification mechanism.

[0024] Figure 7 This is a diagram of the metadata hot update architecture.

[0025] Figure 8 Diagram of metadata consistency guarantee mechanism.

[0026] Figure 9 An abstract architecture diagram for the storage system.

[0027] Figure 10 This is a diagram of a multi-channel container access architecture.

[0028] Figure 11 Flowchart for shared GPU usage monitoring and dynamic billing algorithm.

[0029] Figure 12 Choose a strategy architecture diagram for the storage pool. Detailed Implementation

[0030] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the drawings are merely exemplary and are intended to illustrate the principles and spirit of the invention, and are not intended to limit the scope of the invention.

[0031] Example 1:

[0032] like Figure 1 As shown, a user and data lifecycle server management system for a collaborative laboratory environment includes:

[0033] The container virtualization layer is used to redefine Docker containers as personal computing environment carriers, enabling fine-grained management of CPU resource allocation with a precision of 0.01 cores, memory allocation with a precision of 1MB, and network bandwidth control with a precision of 1Kbps. This allows users to obtain full root privileges inside the container and strictly restricts user operations to the container boundaries.

[0034] In the embodiment, the container virtualization layer includes a state persistence module, a resource quota control module, and a health check module;

[0035] The state persistence module is configured to implement dynamic resource allocation through a three-stage mechanism of "state saving-resource reconfiguration-environment recovery", wherein the state saving time is controlled within 30 seconds, and the environment recovery time is controlled within 60 seconds.

[0036] The resource quota control module is configured to implement CPU resource adjustment by modifying the cpu.shares parameter of cgroups, implement memory adjustment by modifying the memory.limit_in_bytes parameter, and implement GPU resource adjustment by the device mapping mechanism of nvidia-docker.

[0037] The health check module is configured to verify the feasibility of the new container by checking network connectivity, SSH service status, and mount point accessibility, to ensure the seamless switching of the environment.

[0038] The heterogeneous transaction coordination layer is configured to coordinate the heterogeneous infrastructure systems of Docker containers, ZFS / Btrfs storage systems, and Redis caches through the Saga transaction mode, wherein each Saga transaction step includes an execute method and a compensate method, the execution timeout time of the execute method is set to 30 seconds, and the execution timeout time of the compensate method is set to 60 seconds.

[0039] In the embodiment, the heterogeneous transaction coordination layer includes a Saga transaction engine, an intelligent compensation module, and a transaction context manager.

[0040] The Saga transaction engine is configured to define the storage layer dataset creation, quota setting, Docker volume creation, and metadata cache update as a complete business transaction, and ensure the repeatable execution of the compensation operation through idempotency design.

[0041] The intelligent compensation module is configured to implement a context-aware intelligent decision mechanism, for storage layer failure, firstly attempt to re-execute the operation in the standby storage pool, for network timeout, distinguish operation failure and response delay, for concurrent conflict, use an exponential backoff algorithm retry mechanism with an initial delay of 100 milliseconds and a maximum delay of 5 seconds.

[0042] The transaction context manager is configured to store the intermediate state during transaction execution in JSON format, and assign a unique operation ID generated based on the Snowflake algorithm to each Saga transaction.

[0043] A data collaboration ecological layer is configured to build a typed management mechanism of a personal data volume, a shared software environment volume, and a public data set volume, the capacity quota of the personal data volume is 100 GB by default, and the capacity quota of the shared software environment volume is 50 GB by default;

[0044] In the embodiment, the data collaboration ecological layer includes a personal data volume management module, a shared environment management module, a public data management module, and a permission inheritance module.

[0045] The personal data volume management module is configured to create a private storage space with a capacity quota of 100 GB by default and extensible to 500 GB for each user, and automatically set exclusive permission of the user when the private storage space is created, and the mounting path is fixed as the / data directory.

[0046] The shared environment management module supports a hierarchical management strategy of three sharing ranges of a user level, a department level, and a global level, and has a capacity quota of 50 GB by default, supports version management and incremental updating, and is configured to implement an intelligent recommendation mechanism based on a use frequency.

[0047] The public data management module supports version management and collaborative sharing of research data, is configured to occupy an independent storage space for each version, and implements differential storage between versions through a file system snapshot mechanism.

[0048] The permission inheritance module is configured to implement real-time synchronization by using a three-layer permission verification mechanism, wherein a response time of permission checking is controlled to be within 10 milliseconds, and a permission change is synchronized through a Redis publishing and subscribing mechanism, and a synchronization delay is controlled to be within 100 milliseconds.

[0049] A metadata management layer is configured to implement real-time updating and high-performance querying of metadata based on a hot updating architecture of Redis.

[0050] In the embodiment, the metadata management layer adopts a three-level cache architecture including an L1 cache, an L2 cache, and an L3 cache, wherein the L1 cache is a local memory cache with a capacity of 1000 records and an expiration time of 5 seconds, the L2 cache is a Redis distributed cache with an unlimited capacity and an expiration time of 24 hours, and the L3 cache is a relational database, and a cache hit rate target is set to be 80% for the L1 cache and 95% for the L2 cache.

[0051] A hot updating mechanism of the metadata management layer supports dynamic modification of metadata at runtime, an updating operation is implemented through an HSET command of Redis, a response time of a single updating operation is controlled to be within 1 millisecond, and batch updating operations are supported.

[0052] The event-driven synchronization mechanism of the metadata management layer automatically triggers metadata update events when the container state changes through the Spring Boot event publishing mechanism, and the event processing delay is controlled within 50 milliseconds, thereby realizing an intelligent metadata automatic repair mechanism.

[0053] In the embodiment, the user and data full life cycle server management system for a laboratory collaboration environment further comprises:

[0054] The storage abstraction layer is used to realize unified management of different file systems such as ZFS, Btrfs and XFS; the ZFS adapter executes zfs commands through SSH connection, the command execution timeout time is set to 30 seconds, the Btrfs adapter executes btrfs subvolume operations through local commands, and the XFS adapter realizes space limitation through project quota mechanism;

[0055] The intelligent scheduling layer adopts a multi-strategy storage pool selection mechanism such as load balancing, performance optimization and nearby selection, and the load index calculation formula is: load index = (used space / total space) x 0.6 + (active I / O number / max I / O number) x 0.4, and the load index calculation period is 30 seconds.

[0056] The quota management layer is used to realize runtime quota adjustment, wherein the effective time of quota adjustment is controlled within 10 seconds, and the quota monitoring module checks the quota usage every minute, and sends an alarm notification when the usage rate exceeds 90%.

[0057] The user full life cycle management layer is used to realize automatic processing and resource recycling from user registration to resignation; after user registration, the system completes user account creation within 5 seconds, personal data volume allocation within 30 seconds, sets initial resource quota including CPU 2 cores, memory 4GB and storage 100GB, and creates a default container image based on Ubuntu 20.04 LTS for new users.

[0058] The resource recycling adopts a four-stage resource recycling mechanism, and the four-stage resource recycling mechanism is specifically: immediately after the user resigns, new resource allocation is prohibited, within 24 hours after the user resigns, all containers are stopped and data backup is created, 7 days after the user resigns, the containers and images are deleted, and 30 days after the user resigns, all data is completely cleaned up.

[0059] The user full life cycle management layer includes an intelligent credit charging module, a multi-channel access module and a resource monitoring module.

[0060] The intelligent credit charging module adopts a fixed quota mode and a differentiated charging mechanism for monthly reset for charging;

[0061] The monthly reset fixed quota mode is specifically: the default credit quota of ordinary users is 1000 points, the credit quota of graduate students is 1500 points, the credit quota of doctoral students and teachers is 2000 points, and the credit quota of project managers is 3000 points;

[0062] The differentiated charging mechanism is specifically: the exclusive graphics card mode has a basic credit consumption of 10 points / hour, the shared mode has a basic credit consumption of 6 points / hour, the no card mode has a credit consumption of 0.1 points / hour, and the CPU computing mode has a single-core credit consumption of 2 points / hour;

[0063] The multi-channel access module is configured to provide SSH agent, Web terminal and API interface multi-container access modes, and automatically generate RSA-4096-bit key pairs for each user, wherein the key rotation period is 90 days, and a single agent instance can handle 1000 concurrent connections;

[0064] The resource monitoring module is configured to collect GPU usage rate data every 5 seconds, and calculate the actual GPU usage ratio of the user by using a sliding window algorithm, wherein the window size is set to 60 seconds, and the response time of the credit update operation is controlled to be within 10 milliseconds.

[0065] In the embodiment, the user and data full life cycle server management system for a laboratory collaboration environment further comprises:

[0066] The storage pool selection strategy interface is configured to realize dynamic switching and combined use of various selection algorithms such as a specific ZFS pool selection strategy, a load balancing strategy and a performance priority strategy.

[0067] The load balancing strategy is specifically: the lightest storage pool is automatically selected by monitoring the load state of each storage pool in real time.

[0068] The performance priority strategy is specifically: according to the IOPS and bandwidth indicators of the storage pool, the SSD storage pool with an IOPS greater than 10000 is selected preferentially, the runtime switching of the strategy is supported, and the storage allocation strategy can be adjusted without restarting the service.

[0069] In this embodiment, the application improves the container application mode and involves the state persistence mechanism. Specifically, the main improvement of the user and data full life cycle server management system for laboratory collaborative environment in the container application mode is that the container is changed from the traditional "application runtime encapsulation" to the "user personal computing environment carrier". The conversion of this application concept is reflected in that the traditional container technology is application-centered, and pursues environment consistency and deployment convenience; while the system is user-centered, and pursues the balance between environment individualization and permission security. From the perspective of technical implementation, the system adopts Docker container technology at the bottom to ensure resource isolation and lightweight running; from the perspective of user experience, the system provides complete environment control ability similar to virtual machines, and users can freely install software, configure systems, and manage services. This design not only maintains the efficiency of the container technology, but also meets the needs of users for complete computing environment, effectively alleviating the contradiction between "users needing complete system control" and "systems needing security isolation" in the laboratory environment.

[0070] The user and data full life cycle server management system for laboratory collaborative environment uses Docker container as a user's personal computing environment carrier, and realizes fine-grained management with CPU resource allocation accuracy of 0.01 core, memory allocation accuracy of 1 MB, and network bandwidth control accuracy of 1 Kbps through precise resource quota control. On the basis of maintaining the original isolation characteristics of the container, users obtain complete root permission inside the container, can freely install software, configure environment, and modify system settings, while these operations are strictly limited within the container boundary, better solving the problem that user autonomy and system security are difficult to balance in traditional sudo permission management.

[0071] The user and data full life cycle server management system for laboratory collaborative environment realizes a dynamic resource allocation mechanism based on container state persistence, which is a function rarely provided in existing container management systems. As shown in Figure 2 When hardware resource adjustment is needed, the system adopts a three-stage mechanism of "state saving-resource reconfiguration-environment recovery". The state saving stage saves the file system changes of the container as a new image layer through the Docker commit command, and the saving time is controlled within 30 seconds. Considering that the actual use frequency of user dynamic resource switching in the laboratory environment is relatively low, the system sets a maximum commit layer limit (default 10 layers), and automatically merges the historical layers when the limit is reached, effectively avoiding the problem of unlimited image growth. This design maintains the lightweight concept of Docker while meeting the actual needs of user environment state saving in the laboratory environment. The system also records the runtime parameters such as environment variables, port mapping, and mounting point configuration of the container, ensuring the integrity of the environment state.

[0072] The resource reconfiguration stage creates a container configuration according to new resource requirements, CPU resource adjustment is realized by modifying the cpu.shares parameter of cgroups, memory adjustment is realized by the memory.limit_in_byte parameter, and GPU resource adjustment is realized by the device mapping mechanism of nvidia-docker. The environment recovery stage creates a new container based on the saved image and configuration parameters, and the recovery time is controlled within 60 seconds. The system verifies the availability of the new container through a health check mechanism, including network connectivity, SSH service status, mount point accessibility, etc., to ensure the seamless switching of the environment.

[0073] The dynamic resource allocation mechanism improves the deficiency of container technology in resource elasticity, realizes dynamic adjustment of hardware configuration, and provides a more flexible resource management mode for the laboratory environment.

[0074] In the embodiment, the Saga transaction mode is applied to the coordination management of the heterogeneous infrastructure system, and an intelligent compensation mechanism is designed. Specifically, the user and data full life cycle server management system for the laboratory collaboration environment has the feature of applying the Saga transaction mode to the coordination management of the heterogeneous infrastructure system in the distributed transaction processing. The traditional Saga mode is mainly used for business transaction management in the micro-service architecture, and the system extends it to the heterogeneous infrastructure system such as the Docker container, the ZFS / Btrfs storage system and the Redis cache, and realizes the transaction consistency guarantee across the system. The heterogeneous system Saga transaction coordination architecture is as shown in Figure 3 .

[0075] The Saga implementation of the user and data full life cycle server management system for the laboratory collaboration environment deeply integrates business semantics into transaction design, for example, in the data volume creation process, the user and data full life cycle server management system for the laboratory collaboration environment defines the storage layer dataset creation, quota setting, Docker volume creation and metadata cache update as a complete business transaction. Each Saga transaction step contains an execute method and a compensate method, the execution timeout time of the execute method is set to 30 seconds, and the execution timeout time of the compensate method is set to 60 seconds. The user and data full life cycle server management system for the laboratory collaboration environment ensures the repeatable execution of the compensation operation through the idempotency design, which is crucial for the reliability of the infrastructure operation.

[0076] The improvement of the user and data full life cycle server management system for the laboratory collaboration environment in the intelligent compensation mechanism is that, compared with the simple reverse operation of the traditional Saga mode, a more intelligent compensation strategy is provided. The intelligent compensation decision flow is as shown inFigure 4 As shown, the application realizes a context-aware intelligent decision mechanism, which can select the optimal compensation strategy according to the specific reason for failure and the system state. For storage layer failure, the laboratory collaborative environment-oriented user and data full life cycle server management system first attempts to re-execute the operation in the standby storage pool, and if the standby pool is not available, a complete rollback is performed; for network timeout, the laboratory collaborative environment-oriented user and data full life cycle server management system distinguishes between operation failure and response delay, and determines the actual execution result through resource state query; for concurrent conflict, the laboratory collaborative environment-oriented user and data full life cycle server management system adopts an exponential backoff algorithm retry mechanism with an initial delay of 100 milliseconds and a maximum delay of 5 seconds.

[0077] The transaction context management stores the intermediate state during transaction execution in JSON format, including created resource identifiers, configuration parameters, timestamps, and other key information. The laboratory collaborative environment-oriented user and data full life cycle server management system assigns each Saga transaction a unique operation ID generated based on the Snowflake algorithm, and records complete information such as transaction name, execution status, current step, context data, and error information. This design not only ensures the traceability of transactions, but also provides a data foundation for fault analysis and system optimization.

[0078] In this embodiment, the application constructs a data volume collaboration ecosystem based on type management and permission inheritance mechanism. Specifically, the laboratory collaborative environment-oriented user and data full life cycle server management system proposed by the application has the feature of building a complete data volume collaboration mechanism in terms of data collaboration, which improves the data sharing capability in the laboratory environment through type management and permission inheritance mechanism. Existing storage management systems mainly focus on data storage and access, and this system designs a data management architecture from the perspective of collaboration, achieving unified management and flexible sharing of software environment, data set, and personal data. The data volume type management architecture is as shown in Figure 5 .

[0079] The laboratory collaborative environment-oriented user and data full life cycle server management system adopts a separated data persistence architecture to manage user data and environment state hierarchically, achieving precise control of different types of data. User data is persisted through the data volume mechanism, mainly storing research data, project files, output results, and other user-generated content; environment state is persisted through the container image layer, mainly saving user apt install, custom software installation, environment configuration, and other operation changes. This separated design ensures that users can maintain data integrity and preserve environment configuration state in scenarios such as switching from "no card mode" to "GPU mode" resource adjustment, avoiding repeated configuration of the working environment.

[0080] The present application designs three core data volume types (personal data volume, shared software environment volume and public data set volume), each type has different life cycle management strategy and permission control mechanism. The personal data volume provides each user with a unique private storage space, with a capacity quota of 100GB by default, which can be extended to 500GB, and automatically sets user exclusive permissions when created, with a fixed mounting path of / data directory. This design ensures the persistence and cross-container access of user data, solving the fundamental problem of data persistence in container environment.

[0081] The shared software environment volume is characterized by building a laboratory-level software environment sharing mechanism, which improves the traditional software installation and environment configuration method. The system abstracts the software environment as reusable data volume resources, and users can encapsulate the configured software environment (such as conda environment, CUDA tool chain, deep learning framework stack, scientific computing library collection, etc.) into standardized environment data volume. The value of this design is that a user who spends hours or even days to configure a complex software environment can be completely reused by other users within a few minutes through the data volume sharing mechanism, improving the overall work efficiency of the laboratory.

[0082] The user and data full life cycle server management system for laboratory collaboration environment supports hierarchical management strategies with three sharing ranges of user level, department level and global level, with a capacity quota of 50GB by default, supporting version management and incremental update. The user and data full life cycle server management system for laboratory collaboration environment implements an intelligent recommendation mechanism based on usage frequency, which can actively recommend suitable shared environments according to the user's research field and historical usage records, improving collaboration efficiency.

[0083] The public data set volume supports version management and collaborative sharing of research data, with each version occupying independent storage space, and the difference between versions is realized through file system snapshot mechanism. The data set volume supports flexible setting of read-only or read-write permissions, and the creator can choose appropriate sharing strategies according to the sensitivity and collaboration needs of the data.

[0084] The permission inheritance and dynamic verification mechanism is shown in Figure 6 The permission control mechanism is characterized by realizing dynamic permission inheritance based on organizational structure and separation management of permissions inside and outside the container. By specifying the mounting host path, the system ensures that users have complete root permissions inside the container while not affecting the host system, balancing the security of permissions and user autonomy.

[0085] The user and data full life cycle server management system for laboratory collaboration environment adopts a three-layer permission verification mechanism to control the visibility and operation permissions of different users on containers, images, data volumes and other resources. The first layer is user role permission verification, ordinary users can only view and manage built-in resources and resources created by themselves; the second layer is resource access control list verification, which checks the access permissions of users on specific resources; the third layer is specific operation permission verification, which verifies whether the user has the right to perform specific operations (such as starting, stopping, deleting, etc.). The response time of permission checking is controlled within 10 milliseconds, ensuring the high performance of the system.

[0086] The permission inheritance mechanism reflects the hierarchical characteristics of laboratory management: department administrators automatically obtain the management permissions of containers, images and data volumes created by all users in the department, and can perform resource allocation and fault handling; the super administrator obtains complete control permissions of all resources, and is responsible for system-level operation and maintenance. The permission change is realized in real time through the Redis publishing and subscribing mechanism, and the synchronization delay is controlled within 100 milliseconds.

[0087] The technical implementation of data volume mounting adopts an intelligent path allocation algorithm, and the user and data full life cycle server management system for laboratory collaboration environment defines mounting path templates for each data volume type, mounts personal data volumes to / data, mounts shared software environment volumes to / opt / shared / {volume_name}, and mounts data set volumes to / datasets / {dataset_name}. The user and data full life cycle server management system for laboratory collaboration environment checks mounting point conflicts before container creation, and automatically allocates paths with numerical suffixes for data volumes mounted later if there are path conflicts. Data volume mounting is realized through the bind mount mechanism of Docker, and the execution time of mounting operation is controlled within 5 seconds. The user and data full life cycle server management system for laboratory collaboration environment supports a delayed mounting mechanism to reduce container startup time.

[0088] In the embodiment, the application adopts a hot update architecture and designs a multi-level cache mechanism in metadata management. Specifically, the user and data full life cycle server management system for laboratory collaboration environment proposed by the application has the feature of adopting a hot update metadata management architecture based on Redis, instead of the original label mechanism of Docker. Existing container management systems generally rely on the original label function of Docker for metadata management, but this approach has problems such as inability to update hot, low query performance, and limited scalability. The user and data full life cycle server management system for laboratory collaboration environment realizes real-time updating and high-performance querying of metadata by redesigning the metadata storage architecture.

[0089] As Figure 7 shown, the application adopts a three-level cache architecture to realize high-performance management of metadata, L1 cache is local memory cache, capacity 1000 records, expiration time 5 seconds, mainly used for frequently accessed metadata; L2 cache is Redis distributed cache, capacity unlimited, expiration time 24 hours, undertakes the main metadata storage and query function; L3 cache is a relational database, providing permanent storage and data consistency guarantee. The cache hit rate target is set to L1 cache 80%, L2 cache 95%, ensuring the high-performance operation of the system.

[0090] The metadata structure design adopts Hash data structure, the key name format of container metadata is "container:{container_id}", and the fields include user_id, user_name, create_time, built_in, alias and other key information. The image metadata adopts a similar structure, the key name format is "image:{image_id}", and the data volume metadata key name format is "volume:{volume_name}". This design not only provides a flexible data organization method, but also supports complex multi-dimensional query and real-time statistical analysis.

[0091] The characteristics of the hot update mechanism are to support dynamic modification of metadata at runtime, users can dynamically modify the label information during the running of the container, and the system takes effect immediately without restarting the container. The update operation is realized through the HSET command of Redis, and the response time of single update operation is controlled within 1 millisecond. The system supports batch update operation, and through the Pipeline mechanism, multiple update operations are packaged and executed, improving the update efficiency.

[0092] The application realizes an event-driven metadata synchronization mechanism. As Figure 8 shown, the system automatically triggers metadata update events when the container state changes through the event publishing mechanism of SpringBoot, ensuring real-time synchronization of Docker native state and Redis metadata. Event processing adopts asynchronous mode to avoid blocking the main business process, and the event processing delay is controlled within 50 milliseconds. The system also realizes an intelligent metadata automatic repair mechanism, which automatically discovers and repairs metadata inconsistency by scanning the actual state of the Docker daemon through a timing task.

[0093] The change notification mechanism realizes real-time notification of metadata changes through the publish / subscribe function of Redis, and the subscribers include web interface, API service, monitoring system and other components. The notification message adopts JSON format, including change type, resource ID, change field, new value and other complete information, ensuring the data consistency between the components of the system.

[0094] The application adopts a final consistency model to process data synchronization between multi-level caches, which balances the performance and consistency requirements. The system realizes real-time synchronization of metadata through an event-driven mechanism, which is an ideal synchronization state. Meanwhile, considering the possibility of accidental changes out of system control (such as direct operation through the Docker command line), the system adopts a final consistency model as a bottom safeguard mechanism. Through a timing synchronization task, the metadata in Redis is synchronized to the database every 5 minutes. During the synchronization process, data conflicts are detected, and a "Redis first" conflict resolution strategy is adopted to ensure the priority of hot update data.

[0095] The fault-tolerant mechanism includes automatic reconstruction and manual supplement recovery modes. The system performs metadata integrity checking every hour. The actual resource state is obtained by scanning the API of the Docker daemon, and the metadata in Redis is compared. When missing metadata is found, the basic metadata is automatically reconstructed. This double safeguard mechanism ensures the high reliability of metadata management, pursuing both the ideal state of real-time synchronization and the data repair needs of various abnormal situations.

[0096] In the embodiment, the application constructs an abstract storage system and designs an intelligent scheduling algorithm. Specifically, the application has the feature of establishing an abstract storage management framework in storage management, supporting multiple file systems with quota management and snapshot functions. Existing container management systems are usually bound to specific storage backends, with limited flexibility and scalability. Through the design of the storage abstraction layer, the system realizes unified management of different file systems such as ZFS, Btrfs, and XFS, providing users with more technical options.

[0097] The design of the storage backend adapter realizes the abstract and unified management of heterogeneous storage systems. The ZFS adapter executes zfs commands through SSH connection, supporting dataset creation, quota setting, snapshot creation, and other operations. The command execution timeout is set to 30 seconds. The Btrfs adapter executes btrfs subvolume operations through local commands, supporting subvolume creation, quota group setting, snapshot creation, and other functions. The XFS adapter realizes space limitation through project quota mechanism and version management function through LVM snapshot. This abstract adapter architecture design enables the system to flexibly adapt to multiple file systems with quota management and snapshot functions, providing users with more technical options. The storage system abstract architecture is shown in Figure 9 .

[0098] The storage pool selection algorithm is characterized in that a multi-strategy scheduling mechanism for a laboratory cluster environment is realized. The laboratory environment is usually composed of multiple single graphics card servers, the system supports cross-server cluster deployment, and users can flexibly select different server nodes to create containers according to the computing power demand. The load balancing strategy realizes the balanced allocation of resources by calculating the load index of each storage pool, the load index calculation formula is: load index = (used space / total space) × 0.6 + (active I / O number / maximum I / O number) × 0.4, the system selects the storage pool with the smallest load index, and the load index calculation period is 30 seconds. The performance optimization strategy sorts according to the IOPS performance index of the storage pool, and for data volumes that require high I / O performance, the SSD storage pool with IOPS greater than 10000 is preferentially selected.

[0099] The cross-node data migration mechanism realizes the file content migration between heterogeneous storage systems. For example, when data needs to be migrated from a ZFS node to an XFS node, the system first creates a corresponding storage dataset (the dataset concept of ZFS corresponds to the project quota directory of XFS) on the target node, then transmits the file content to the target node through the network, and finally updates the metadata record to complete the migration process. This file content-based migration method avoids relying on deep file system features, ensuring the reliability and compatibility of cross-heterogeneous storage system migration. The cluster proximity strategy preferentially selects the local storage pool of the server node where the user's current container is located, reduces network transmission overhead, and ensures low latency data access.

[0100] The quota management mechanism realizes accurate control and dynamic adjustment of storage resources. The system maintains a quota rule table containing fields such as data volume type, storage pool type, quota size, quota unit, and priority, and the rule matching uses priority ordering, with high-priority rules taking effect first. The system supports runtime quota adjustment, which is executed through a storage backend adapter, and the effective time of quota adjustment is controlled within 10 seconds, without affecting data access during adjustment. The quota monitoring system checks the quota usage every minute and sends an alarm notification when the usage rate exceeds 90%. Quota monitoring data is stored in a time series database, supporting historical trend analysis and capacity planning.

[0101] In the embodiment, the present application establishes a complete user full life cycle automatic management process and designs a multi-channel access mechanism in the aspect of user management. Specifically, the user full life cycle automatic management process covers the whole process of automatic processing from user employment to user resignation. Existing server management systems usually lack systematic management of user life cycle, resulting in problems such as untimely resource allocation and incomplete resource recovery. The present system improves the allocation and recovery efficiency of user resources through automatic process design.

[0102] The user registration automation process performs resource allocation in strict timing. After user registration, the system completes user account creation within 5 seconds, personal data volume allocation within 30 seconds, and sets initial resource quotas including CPU 2 cores, memory 4 GB, and storage 100 GB. The system creates a default container image based on Ubuntu 20.04 LTS for new users, configures SSH keys and network access permissions. New users are assigned the normaluser role by default, allowing them to create and manage personal containers, access public image repositories, and use read-only access to shared datasets. The system preinstalls common development tools for new users, including git, vim, python3, nodejs, and configures default shell environment and editor settings. The preconfiguration process is executed asynchronously in the background, without affecting the user's first login experience.

[0103] The resource recycling process for user departure adopts a phased processing strategy to ensure thoroughness and data security. The first phase is executed immediately after the user state changes, marking the user state as "deactivated", prohibiting new resource allocation and container creation, and forcibly terminating all active sessions. The execution time is controlled within 30 seconds. The second phase is executed within 24 hours, stopping all running containers for the user, creating a complete snapshot backup of the personal data volume, and the snapshot creation time is usually between 10 minutes and 2 hours depending on the data volume. The third phase is executed after 7 days, deleting the user's containers and images, retaining the data volume snapshot, and releasing the computing resources immediately back to the resource pool. The fourth phase is executed after 30 days, deleting the data volume snapshot and completely cleaning up all user data, and sending a final notification email to the user and administrator before cleaning up.

[0104] The multi-channel container connection mechanism provides multiple access methods and fault handling capabilities. The SSH connection uses an innovative key management mechanism to automatically generate a RSA-4096 key pair for each user. The private key is encrypted and stored in the user's browser local storage, the public key is stored in the authorized_keys file of the user's container, and the key rotation period is 90 days. The SSH proxy service implements a routing mechanism based on user identity, forwarding connection requests to the corresponding container. The proxy service supports connection multiplexing, with a single proxy instance capable of handling 1000 concurrent connections. The system monitors key indicators such as SSH connection establishment time, data transfer volume, and connection duration in real time, with a connection establishment time target of less than 3 seconds. The multi-channel container access architecture is shown in Figure 10 .

[0105] The web terminal function establishes a real-time communication channel with the container through the WebSocket protocol, supports bidirectional data transmission, the heartbeat interval of the WebSocket connection is set to 30 seconds, and the connection timeout time is set to 5 minutes. The system realizes complete VT100 terminal simulation, supports color display, cursor control, screen clearing and other terminal functions, the terminal buffer size is set to 10000 lines, and history command rollback is supported. The web terminal integrates file upload and download functions, supports drag-and-drop upload and batch download, the single file upload size limit is 100MB, and supports breakpoint resume and transmission progress display.

[0106] The fault bottom mechanism ensures that the user can access the container environment in any case. The system performs a health check on the container every 30 seconds, and the check items include the container process state, the SSH service port listening, network connectivity, and disk space usage. After three failures of the health check, an automatic repair process is triggered. The automatic repair process includes automatically restarting the sshd service when the SSH service is abnormal, reapplying network configuration when the network configuration is incorrect, cleaning up temporary files and log files when the disk space is insufficient, and controlling the execution time of the repair operation within 60 seconds. When all regular connection methods fail, the system provides an emergency access channel based on Docker exec. The emergency channel is provided through a web interface, and the user can perform basic command line operations to troubleshoot the fault.

[0107] In the embodiment, the application establishes a charging system based on an integral mechanism in resource management, improves the problem of resource abuse and unreasonable allocation in the laboratory environment, and designs a resource use monitoring mechanism. Specifically, the existing laboratory server management system generally lacks an effective resource use constraint mechanism, resulting in some users long-term occupation of a large amount of resources and other users unable to obtain necessary computing resources. The system improves the fairness and rationality of resource use through an integral charging mechanism.

[0108] The integral budget management mechanism adopts a monthly reset fixed quota budget mode. Each user automatically obtains a fixed number of integral budget quotas at the beginning of each month, and the budget quotas are differentiated according to user roles and research needs. The default integral quota of an ordinary user is 1000 points, the integral quota of a graduate student user is 1500 points, the integral quota of a doctoral student and a teacher user is 2000 points, and the integral quota of a project leader is 3000 points. The system supports temporary adjustment of the integral quota by an administrator according to special project needs, and the adjustment operation needs an approval process to ensure the fairness of resource allocation.

[0109] The differentiated billing based on resource usage patterns reflects the refined management philosophy of the system design. The dedicated graphics card mode has the highest points multiplier. When a user requests to exclusively use one or more graphics cards, the system calculates the points consumption based on the number of graphics cards and the usage time. The base points consumption for a single dedicated graphics card mode is 10 points / hour. When multiple graphics cards are exclusively used, the points consumption increases non-linearly: 25 points / hour for two graphics cards, 45 points / hour for three graphics cards, and 70 points / hour for four graphics cards. This design encourages users to use resources rationally and avoid excessive usage.

[0110] Shared GPU usage monitoring and dynamic billing algorithm process as follows Figure 11 As shown, the shared GPU usage mode is characterized by a dynamic amortization billing mechanism based on actual usage. The system deploys a resource monitoring module that collects GPU usage data every 5 seconds, including key indicators such as GPU core usage, memory usage, and power consumption. The system uses a sliding window algorithm to calculate the actual GPU usage ratio of each user, with the window size set to 60 seconds. Each user's points are deducted from the shared mode's basic points pool based on their average usage ratio within that time window. For example, when a GPU is shared by three users, the shared mode's basic points consumption is 6 points / hour. If user A uses 50% of the GPU resources, user B uses 30%, and user C uses 20%, then user A will be deducted 3 points / hour, user B 1.8 points / hour, and user C 1.2 points / hour, achieving fair billing based on actual usage ratios.

[0111] The differentiated design of the cardless mode and CPU computing mode addresses the needs of different computing scenarios. Cardless mode is primarily used for lightweight development and debugging tasks, does not consume GPU resources, and has extremely low point consumption (0.1 points / hour), mainly covering basic storage and network resource consumption. CPU computing mode is designed for scientific computing tasks requiring significant CPU resources. Point consumption is calculated based on the number of CPU cores requested by the user. The single-core CPU computing mode consumes 2 points / hour, with point consumption increasing linearly with the number of cores: 16 points / hour for 8 cores and 32 points / hour for 16 cores.

[0112] The points monitoring and early warning mechanism ensures users can rationally plan their resource usage. The system monitors the user's points balance in real time, sending an early warning notification when the balance falls below 20% of the total limit and an emergency notification when it falls below 10%. Once points are exhausted, the system automatically stops all resource-intensive containers for the user, while retaining access to lightweight containers to ensure users can save their work. The system provides a points usage history query function, allowing users to view detailed points consumption records, including usage time, resource type, and points consumed, helping users optimize their resource usage strategies.

[0113] The technical implementation of the integral system adopts a distributed computing architecture, integral calculation tasks are distributed on multiple computing nodes for execution, and the integral calculation capability of a single computing node is 1000 users / second. The system stores integral data using a Redis cluster, supports real-time updating and high-concurrency querying of integral data, and controls the response time of integral updating operations within 10 milliseconds. The accuracy of integral calculation is ensured by multiple verification mechanisms, including real-time verification, timing verification and audit verification, to ensure the fairness and accuracy of integral deduction.

[0114] In the embodiment, the application realizes a storage pool selection mechanism and a load balancing mechanism based on a policy mode in storage resource scheduling. The system constructed by the application abstracts a storage pool selection policy interface, supports dynamic switching and combined use of multiple selection algorithms, including a specific ZFS pool selection policy, a load balancing policy, a performance priority policy, etc. The storage pool selection policy architecture is as shown in Figure 12

[0115] The implementation of the policy mode enables the system to flexibly select the most suitable storage resource allocation algorithm according to different business scenarios and performance requirements. The specific pool selection policy allows administrators to specify a dedicated storage pool for a specific type of data volume, ensuring performance guarantees for critical data. The load balancing policy automatically selects the storage pool with the lightest load by monitoring the load state of each storage pool in real time. The performance priority policy prioritizes high-performance storage pools based on IOPS and bandwidth indicators. The system supports runtime switching of policies, allowing storage allocation strategies to be adjusted without restarting the service, providing good flexibility.

[0116] The application constructs a server management system that highly meets the needs of a laboratory collaboration environment through the close combination of eight core technology innovation blocks. The conceptual reconstruction of container virtualization solves the fundamental contradiction of permission management, the Saga transaction coordination of heterogeneous systems ensures the reliability of complex operations, the data volume collaboration ecosystem realizes unprecedented data sharing capabilities, the metadata hot update architecture provides high-performance dynamic management, the storage system abstraction design ensures the flexibility of technology selection, the user full life cycle management realizes complete automated operation and maintenance, the intelligent credit charging system ensures the fairness and reasonableness of resource use, and the policy-based storage scheduling mechanism realizes the intelligence and flexibility of resource allocation.

[0117] Embodiment 2:

[0118] Based on embodiment 1, the application embodiment proposes a user and data full life cycle server management method for a laboratory collaboration environment, specifically including the following steps:

[0119] S1. Container virtualization conceptual reconstruction, the implementation method is as follows:

[0120] ​S101. Redefine the Docker container as a user personal computing environment carrier, achieve fine-grained management of CPU resource allocation accuracy of 0.01 core, memory allocation accuracy of 1 MB, and network bandwidth control accuracy of 1 Kbps through precise resource quota control, and the user obtains complete root permission inside the container and can freely install software, configure the environment, and modify system settings, while these operations are strictly limited within the container boundary;

[0121] S102. Implement a dynamic resource allocation mechanism based on container state persistence, adopt a three-stage mechanism of "state saving-resource reallocation-environment recovery", the state saving stage saves the container's file system changes as a new image layer through the Docker commit command, the saving time is controlled within 30 seconds, and the runtime parameters such as the container's environment variables, port mapping, and mount point configuration are recorded;

[0122] S103. Resource reallocation stage creates container configuration according to new resource requirements, CPU resource adjustment is achieved by modifying the cpu.shares parameter of cgroups, memory adjustment is achieved by the memory.limit_in_bytes parameter, GPU resource adjustment is achieved by the device mapping mechanism of nvidia-docker, and environment recovery stage creates a new container based on the saved image and configuration parameters, and the recovery time is controlled within 60 seconds.

[0123] S2. Heterogeneous system transaction coordination, the implementation method is as follows:

[0124] S201. Design Saga transaction engine, define storage layer data set creation, quota setting, Docker volume creation, and metadata cache update as a complete business transaction, each Saga transaction step contains execute method and compensate method, the execution timeout time of execute method is set to 30 seconds, the execution timeout time of compensate method is set to 60 seconds, and the idempotency design is used to ensure the repeatable execution of compensation operation;

[0125] S202. Implement intelligent compensation mechanism, implement context-aware intelligent decision-making mechanism, for storage layer failure, the system first tries to re-execute the operation in the standby storage pool, if the standby pool is not available, perform complete rollback; for network timeout, the system distinguishes operation failure and response delay, determines the actual execution result through resource state query; for concurrent conflict, the system adopts exponential backoff algorithm retry mechanism, the initial delay is 100 milliseconds, and the maximum delay is 5 seconds;

[0126] S203. The transaction context management stores the intermediate state in the transaction execution process in JSON format, including the created resource identifier, configuration parameters, timestamp, and other key information. The system assigns a unique operation ID generated based on the Snowflake algorithm to each Saga transaction, and records the transaction name, execution status, current step, context data, error information, and other complete information.

[0127] S3. Data volume collaboration ecosystem construction, the implementation method is as follows:

[0128] S301. Create a personal data volume, provide each user with a private storage space with a capacity quota of 100GB by default and extendable to 500GB. Automatically set exclusive user permissions when creating, and the mounting path is fixed as / data directory to ensure the persistence and cross-container access of user data.

[0129] S302. Build a shared software environment volume. The system innovatively abstracts the software environment into reusable data volume resources. Users can encapsulate carefully configured software environments (such as conda environments, CUDA toolchains, deep learning framework stacks, and scientific computing library collections) into standardized environment data volumes. Support for hierarchical management strategies at user, department, and global levels with a capacity quota of 50GB by default.

[0130] S303. Manage public data set volumes. Support versioned management and collaborative sharing of research data. Each version occupies independent storage space. Differences between versions are stored through the file system snapshot mechanism. Data set volumes support flexible settings of read-only or read-write permissions.

[0131] S304. Implement the permission inheritance mechanism. The system uses a three-layer permission verification mechanism. First, check the user role permissions, then check the data volume access control list, and finally check the specific operation permissions. The response time of permission checking is controlled within 10 milliseconds. Department administrators automatically obtain management permissions for all data volumes created by users within the department. Super administrators have complete control over all data volumes.

[0132] S305. Intelligent path allocation algorithm. The system predefines mounting path templates for each data volume type. Personal data volumes are mounted to / data, shared software environment volumes are mounted to / opt / shared / {volume_name}, and data set volumes are mounted to / datasets / {dataset_name}. The system checks for mounting point conflicts before container creation. If there is a path conflict, the system automatically assigns a path with a numerical suffix to the later-mounted data volume.

[0133] S4. User lifecycle management, the implementation method is as follows:

[0134] S401. User registration automation process, after user registration, the system completes user account creation within 5 seconds, personal data volume allocation within 30 seconds, and sets initial resource quotas including CPU 2 cores, memory 4 GB, and storage 100 GB. The system creates a default container image based on Ubuntu 20.04 LTS for new users, configures SSH keys and network access permissions, and new users are assigned the normaluser role by default;

[0135] S402. Intelligent credit charging system, using a monthly reset fixed quota mode, each user automatically obtains a fixed number of credits at the beginning of each month. The default credit quota for normal users is 1000 points, graduate student users have a credit quota of 1500 points, doctoral student and teacher users have a credit quota of 2000 points, and project leaders have a credit quota of 3000 points. The base credit consumption for exclusive GPU mode is 10 points / hour, and the base credit consumption for shared mode is 6 points / hour;

[0136] S403. Phased resource recycling mechanism, after user leaves, a four-stage processing strategy is adopted: the first stage is executed immediately after the user's status changes, marking the user's status as "disabled", prohibiting new resource allocation and container creation, forcing all active sessions to be offline, and the execution time is controlled within 30 seconds; the second stage is executed within 24 hours, stopping all running containers for the user and creating a complete snapshot backup of the personal data volume; the third stage is executed after 7 days, deleting the user's containers and images while retaining the data volume snapshot; the fourth stage is executed after 30 days, deleting the data volume snapshot and completely cleaning up all user data;

[0137] S404. Multi-channel access mechanism, providing SSH agent, web terminal, API interface, and other container access methods. SSH connection uses an innovative key management mechanism to automatically generate a RSA-4096 key pair for each user. The private key is encrypted and stored in the user's browser local storage, and the public key is stored in the user's container authorized_keys file. The key rotation period is 90 days, and the SSH agent service supports connection multiplexing, with a single agent instance capable of handling 1000 concurrent connections;

[0138] S405. Resource monitoring and billing, the system deploys a resource monitoring module that collects GPU usage rate data every 5 seconds, including GPU core usage rate, video memory usage, and power consumption data. The system uses a sliding window algorithm to calculate the user's actual GPU usage ratio, with a window size of 60 seconds. Credit calculation tasks are distributed across multiple computing nodes, and the response time for credit update operations is controlled within 10 milliseconds;

[0139] S406. Storage pool intelligent selection, the system abstracts a storage pool selection strategy interface, supports dynamic switching and combined use of various selection algorithms such as specific ZFS pool selection strategies, load balancing strategies, performance priority strategies, etc., the load balancing strategy realizes balanced allocation of resources by calculating the load index of each storage pool, wherein the load index calculation formula is: load index = (used space / total space) x 0.6 + (active I / O number / max I / O number) x 0.4, and the system selects the storage pool with the smallest load index.

[0140] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of understanding the principles of the present application and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A server management system for the entire lifecycle of users and data in a laboratory collaboration environment, characterized in that, include: The container virtualization layer is used to redefine Docker containers as personal computing environment carriers, strictly restricting user operations within the container boundary and ensuring that users have full root privileges inside the container, achieving fine-grained management with CPU resource allocation precision of 0.01 cores, memory allocation precision of 1MB, and network bandwidth control precision of 1Kbps. The heterogeneous transaction coordination layer is used to coordinate heterogeneous infrastructure systems such as Docker containers, ZFS / Btrfs storage systems, and Redis cache through the Saga transaction mode, so as to achieve cross-system transaction consistency guarantees. The data collaboration ecosystem layer is used to build a typified management mechanism for personal data volumes, shared software environment volumes, and public dataset volumes, enabling unified management and flexible sharing of data in the laboratory environment. Metadata management layer, used to implement real-time updates and high-performance queries of metadata based on Redis's hot update architecture; The container virtualization layer includes a state persistence module, a resource quota control module, and a health check module. The state persistence module is used to realize dynamic resource allocation through a three-stage mechanism of "state saving - resource reconfiguration - environment recovery"; wherein the state saving time is controlled within 30 seconds and the environment recovery time is controlled within 60 seconds. The resource quota control module is used to adjust CPU resources by modifying the cpu.shares parameter of cgroups, adjust memory by the memory.limit_in_bytes parameter, and adjust GPU resources by the device mapping mechanism of nvidia-docker. The health check module is used to verify the feasibility of the new container by checking network connectivity, SSH service status and mount point accessibility, so as to ensure a seamless environment switch. The heterogeneous transaction coordination layer includes the Saga transaction engine, intelligent compensation module, and transaction context manager; The Saga transaction engine is used to define storage layer dataset creation, quota setting, Docker volume creation, and metadata cache update as complete business transactions, and ensures the repeatability of compensation operations through idempotency design. The intelligent compensation module is used to implement a context-aware intelligent decision-making mechanism. For storage layer failures, it attempts to re-execute the operation in the backup storage pool. If the backup pool is unavailable, it performs a full rollback. For network timeouts, it distinguishes between operation failures and response delays and determines the actual execution result through resource status queries. For concurrent conflicts, it adopts an exponential backoff algorithm retry mechanism with an initial delay of 100 milliseconds and a maximum delay of 5 seconds. The transaction context manager is used to store the intermediate state during the transaction execution process in JSON format and assign a unique operation ID based on the snowflake algorithm to each Saga transaction.

2. The user and data lifecycle server management system for the laboratory collaboration environment according to claim 1, characterized in that, The data collaboration ecosystem layer includes a personal data volume management module, a shared environment management module, a public data management module, and a permission inheritance module; The personal data volume management module is used to create a private storage space for each user with a default capacity quota of 100GB, expandable to 500GB, and automatically set exclusive permissions for the user during creation, with the mount path fixed to the / data directory; The shared environment management module supports hierarchical management strategies for three sharing scopes: user-level, department-level, and global-level. The default capacity quota is 50GB. It supports version management and incremental updates to implement an intelligent recommendation mechanism based on usage frequency. The public data management module supports versioned management and collaborative sharing of research data, and is used to occupy independent storage space for each version, and realizes the differential storage between versions through the file system snapshot mechanism. The permission inheritance module is used to achieve real-time synchronization using a three-layer permission verification mechanism. The response time for permission checks is controlled within 10 milliseconds, and permission changes are synchronized through the Redis publish-subscribe mechanism with a synchronization delay controlled within 100 milliseconds.

3. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 1, characterized in that, The metadata management layer adopts a three-level caching architecture, including L1 cache, L2 cache and L3 cache; among them, L1 cache is a local memory cache with a capacity of 1000 records and an expiration time of 5 seconds; L2 cache is a Redis distributed cache with unlimited capacity and an expiration time of 24 hours; L3 cache is a relational database with a cache hit rate target of 80% for L1 cache and 95% for L2 cache.

4. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 1, characterized in that, The hot update mechanism of the metadata management layer supports dynamic modification of metadata at runtime. Update operations are implemented through Redis's HSET command. The response time of a single update operation is controlled within 1 millisecond, and batch update operations are supported. The event-driven synchronization mechanism of the metadata management layer automatically triggers metadata update events when the container state changes through the Spring Boot event publishing mechanism, with event processing latency controlled within 50 milliseconds, thus realizing an intelligent automatic metadata repair mechanism.

5. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 1, characterized in that, The system also includes: The storage abstraction layer is used to achieve unified management of ZFS, Btrfs, and XFS file systems through intelligent scheduling algorithms and quota management mechanisms. Among them, the ZFS adapter executes zfs commands through SSH connection with a command execution timeout set to 30 seconds, the Btrfs adapter executes btrfs subvolume operations through local commands, and the XFS adapter implements space limits through project quota mechanisms. The intelligent scheduling algorithm specifically includes a multi-strategy storage pool selection mechanism that combines load balancing, performance optimization, and proximity selection. The quota management mechanism specifically includes: supporting runtime quota adjustments, with the effective time of quota adjustments controlled within 10 seconds, and the quota monitoring system checking quota usage every minute, sending an alarm notification when the usage rate exceeds 90%.

6. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 1, characterized in that, The system also includes: The user lifecycle management layer is used to automate the entire process from user registration to departure and to recycle resources. Resource recycling is achieved through a four-stage resource recycling mechanism, which is as follows: immediately after a user leaves, new resource allocation is prohibited; within 24 hours after a user leaves, all containers are stopped and data backups are created; 7 days after a user leaves, containers and images are deleted; and 30 days after a user leaves, all data is thoroughly cleaned up.

7. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 6, characterized in that, The user lifecycle management system includes an intelligent points-based billing module, a multi-channel access module, and a resource monitoring module. The intelligent points-based billing module uses a fixed monthly quota mode and a differentiated billing mechanism for billing. The monthly reset fixed quota mode is as follows: the default quota for ordinary users is 1,000 points, for graduate students it is 1,500 points, for doctoral students and teachers it is 2,000 points, and for project leaders it is 3,000 points. The differentiated billing mechanism is as follows: the basic credit consumption for dedicated graphics card mode is 10 credits / hour, the basic credit consumption for shared mode is 6 credits / hour, the credit consumption for cardless mode is 0.1 credits / hour, and the credit consumption for single-core CPU computing mode is 2 credits / hour. The multi-channel access module is used to provide multiple container access methods such as SSH proxy, web terminal and API interface, and automatically generate RSA-4096 bit key pairs for each user; The resource monitoring module is used to collect GPU usage data every 5 seconds and use a sliding window algorithm to calculate the user's actual GPU usage ratio.

8. The user and data lifecycle server management system for a laboratory collaboration environment according to claim 1, characterized in that, The system also includes a storage pool selection strategy interface connected to the storage abstraction layer; The storage pool selection strategy interface is used to dynamically switch and combine specific ZFS pool selection strategies, load balancing strategies, and performance priority strategies. The load balancing strategy is as follows: by monitoring the load status of each storage pool in real time, the storage pool with the lightest load is automatically selected. The performance priority strategy is as follows: based on the IOPS and bandwidth metrics of the storage pool, SSD storage pools with IOPS greater than 10,000 are selected first.

Images