Multi-user storage docking method of an AI platform and AI system

Abstract

The application relates to the technical field of data storage, and discloses a multi-user storage docking method of an AI platform and an AI system. The method comprises the following steps: monitoring data access of an AI task to generate an I / O image; combining context quantization of the AI task to generate a future access intention; when the intention meets a preset optimization condition, generating an asynchronous optimization task of a specified source physical storage system and a target physical storage system; and executing the task, performing in-transit calculation in the process of data transmission from the source system to the target system to generate derived data and write the derived data into the target system. The system comprises a monitoring unit, a prediction unit, a task generation unit and an execution unit. Through prediction and response of the future access intention, the application realizes data self-adaptive optimization and seamless docking transparent to the AI task, can significantly reduce data access delay on the premise of not interrupting business, and improves system resource utilization efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a multi-user storage docking method of AI platform and AI system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the scale and complexity of data processed by AI platforms are growing. In order to balance between cost and performance, these platforms usually adopt multiple different types and performance levels of storage systems at the same time, such as high-speed solid state disk (SSD) storage and large-capacity, low-cost hard disk (HDD) storage, forming a heterogeneous storage environment.

[0003] In the prior art, it is a significant challenge to uniformly dock these heterogeneous storage systems to the AI platform and provide efficient data access for upper AI tasks. Usually, the storage location of data is static or relies on manual data migration by administrators. When the data accessed by AI tasks is located on low-performance storage, it often causes serious I / O bottlenecks, making expensive computing resources (such as GPUs) in a waiting state for a long time, greatly reducing the efficiency of training or inference.

[0004] The data migration operation to alleviate this problem often lacks business awareness in the prior art. The migration process usually needs to pause the related AI tasks, manually modify the data access path in the application, and then restart the task, which directly leads to the interruption of business and affects the continuity of AI services. In addition, the traditional data processing process completes all preprocessing work such as data decoding and format conversion by the CPU of the main computing server, which not only increases the computing burden of the main server, but also causes the contention between computing resources and storage I / O resources, further limiting the improvement of overall performance.

[0005] Therefore, the present application proposes a multi-user storage docking method of AI platform and AI system to solve the problems of the prior art. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a multi-user storage docking method of AI platform and AI system, which solves the problem that the AI platform usually adopts an interruptive data migration or hard switching scheme when docking multiple user storage systems, resulting in poor business continuity, fragile data access path, and high data management complexity.

[0007] To solve the above technical problems, the application provides a multi-user storage docking method of an AI platform and an AI system, which introduces an innovative mechanism combining workload awareness, strategy-driven adaptive prefetching and in-transit data calculation, realizes seamless data docking and performance optimization between heterogeneous storages without interrupting AI tasks.

[0008] The first aspect of the application provides a multi-user storage docking method of an AI platform, which comprises the following steps:

[0009] S1, monitoring data access of an AI task to a logical path in a unified logical namespace, and generating an I / O image of the data access;

[0010] S2, projecting time sequence access features contained in the I / O image into a task state space defined by a task context of the AI task, to quantify a future access intention of the AI task to the logical path;

[0011] S3, in response to the future access intention satisfying a preset optimization condition, generating an asynchronous optimization task containing a data processing workflow, the asynchronous optimization task specifying a source physical storage system and a target physical storage system of data;

[0012] S4, executing the asynchronous optimization task, reading original data from the source physical storage system, and in the process of transmitting the original data to the target physical storage system, performing in-transit calculation on the original data according to the data processing workflow to generate derived data, and writing the derived data into the target physical storage system.

[0013] Preferably, in step S1, the step of generating the I / O image of the data access comprises: collecting an I / O operation sequence of the data access, and extracting time sequence access features from the I / O operation sequence, the time sequence access features including access order, data read-write granularity distribution and I / O operation frequency.

[0014] In one specific embodiment, in step S2, the step of quantifying the future access intention of the AI task to the logical path comprises: defining a task running phase in the task context as a discrete dimension of the task state space, the task state space being a multi-dimensional semantic space; taking the time sequence access features as an input vector, and projecting the input vector into a state vector in the task state space through a mapping operation, the position of the state vector representing the future access intention.

[0015] The future access intention is quantified by a prefetch intention score, and the calculation formula of the prefetch intention score is:

[0016] ;

[0017] wherein: is a prefetch intention score; is an I / O image vector composed of timing access features; is a task context; is a preset weight coefficient; is a processing function for the i-th component in the I / O image vector; is a conversion function for converting the task context into a numerical intention enhancement factor.

[0018] In one embodiment, the preset optimization condition comprises: a spatial distance between the state vector and a preset high intention region in the task state space is less than a preset distance threshold, and a performance level of the source physical storage system is lower than a preferred performance level in a data policy defined for the logical path.

[0019] Preferably, the data processing workflow is an ordered combination of one or more data processing plugins associated with the task type of the AI task; the data processing plugins include image decoding, data normalization or feature extraction plugins.

[0020] In one embodiment, in step S4, the step of performing in-transit computation on the raw data according to the data processing workflow comprises: sequentially invoking the data processing plugins according to the order of the data processing plugins in the data processing workflow to perform data preprocessing operations on the raw data flowing through the distributed storage agent deployed in the source physical storage system or the target physical storage system, by the distributed storage agent.

[0021] Preferably, the method further comprises: after the derived data is successfully written into the target physical storage system, updating the metadata of the unified logical namespace to generate a derived data view associated with a physical storage location of the derived data for the logical path. The metadata maintains a plurality of mapping relationships for the logical path pointing to the raw data view and the derived data view of the derived data.

[0022] Based on the foregoing embodiments, the method further comprises: when a new data access request is received, selecting the derived data view or the raw data view from the plurality of mapping relationships based on a task context of an AI task associated with the new data access request, and routing the new data access request to a physical storage system corresponding to the selected view.

[0023] ​Preferably, the AI platform interfaces with multiple heterogeneous physical storage systems, and deploys a distributed storage agent for each physical storage system, which is responsible for performing the in-transit computation and protocol conversion with the physical storage system.

[0024] The second aspect of the present application provides a multi-user storage interfacing AI system of an AI platform, which is applied to the method of any one of the preceding aspects, and the system comprises:

[0025] A monitoring unit for monitoring data access of an AI task to a logical path in a unified logical namespace, and generating an I / O image of the data access;

[0026] A prediction unit for projecting time-series access features contained in the I / O image into a task state space defined by a task context of the AI task, to quantify a future access intention of the AI task to the logical path;

[0027] A task generation unit for generating an asynchronous optimization task containing a data processing workflow in response to the future access intention satisfying a preset optimization condition, the asynchronous optimization task specifying a source physical storage system and a target physical storage system of data;

[0028] An execution unit for executing the asynchronous optimization task, reading original data from the source physical storage system, and performing in-transit computation on the original data according to the data processing workflow to generate derived data during transmission of the original data to the target physical storage system, and writing the derived data to the target physical storage system.

[0029] The present application provides a multi-user storage interfacing method and AI system of an AI platform. The following beneficial effects are achieved:

[0030] 1. The present application provides a unified logical namespace, and executes an asynchronous optimization task containing data migration and in-transit computation in the background, so that the data access path of an AI task remains stable when the storage backend changes; the optimization process of data layout is transparent to the upper-layer AI application, so that the interfacing and data evolution between heterogeneous storage systems are completed without interrupting the business, and the continuity of AI services is ensured.

[0031] 2. The present application projects time-series access features of an I / O image into a task state space of an AI task, so that the future data access intention can be quantitatively predicted; based on this prediction, the system can actively trigger an asynchronous optimization task to pre-fetch data from a low-performance storage system to a high-performance storage system in advance, so that the AI task can directly read data from the high-performance system when it actually needs the data, thereby reducing the data access delay and improving the effective utilization of computing resources.

[0032] 3、The in-transit computing mechanism introduced in the application enables the data preprocessing operation to be completed by the distributed storage agent during the transmission of data from the source storage to the target storage; this design unloads the data decoding, format conversion and other computing tasks from the main computing server to the storage side, separates the data preparation and core model computing loads, and reduces the CPU burden of the main server.

[0033] 4、The application realizes the automatic management of data processing and placement by binding the data processing workflow with the data strategy and combining the automatically generated derived data view; the system can automatically generate and maintain multiple copies of the same original data in different processing states according to the AI task requirements, simplifies the complexity of data version management and governance, and improves the supply flexibility of data assets for different AI application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The figure is an architecture diagram of the AI platform multi-user storage docking system of the application;

[0035] Figure 2 The figure is a functional module diagram of the AI platform multi-user storage docking AI system of the application;

[0036] Figure 3 The figure is a flowchart of an AI platform multi-user storage docking method of the application;

[0037] Figure 4 The figure is a data structure diagram of the I / O image in the application;

[0038] Figure 5 The figure is a schematic diagram of the task state space and projection operation in the application;

[0039] Figure 6 The figure is a data structure diagram of the asynchronous optimization task in the application;

[0040] Figure 7 The figure is a data flow diagram of in-transit computing in the application.

[0041] Among them, 10, monitoring unit; 20, prediction unit; 30, task generation unit; 40, execution unit; 50, metadata management unit. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the application specification. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0043] Refer to the drawingsFigure 1 , attached Figure 1 is a schematic diagram of an architecture of an AI platform multi-user storage interfacing system according to an embodiment of the present application; the system architecture includes an AI platform, one or more AI tasks running on the AI platform; the AI platform provides a unified logical namespace for the AI tasks, and interfaces a physical storage resource pool through the unified logical namespace.

[0044] An AI task accesses data through a logical path defined in the unified logical namespace; the physical storage resource pool is composed of a plurality of heterogeneous physical storage systems, attached Figure 1 , for example, a network file system, a distributed object storage, and a Hadoop distributed file system, but not limited thereto;

[0045] The AI platform deploys a distributed storage agent for each managed physical storage system; the distributed storage agent is deployed between the AI platform and the physical storage system, and is responsible for receiving data operation instructions from the AI platform and interacting with the corresponding physical storage system.

[0046] Referring to attached Figure 2 , attached Figure 2 is a functional module diagram of an AI platform multi-user storage interfacing AI system according to an embodiment of the present application; the AI system can be a standalone server, or a collection of one or more software modules deployed in the AI platform.

[0047] The monitoring unit 10 is configured to monitor data access of an AI task to a logical path in the unified logical namespace, and generate an I / O image of the data access; the monitoring unit 10 sends the generated I / O image to the prediction unit 20;

[0048] The prediction unit 20 is configured to receive the I / O image and a task context of the AI task obtained from a policy engine, project time-series access features contained in the I / O image into a task state space defined by the task context of the AI task, to quantify a future access intention of the AI task to the logical path; the prediction unit 20 sends the generated future access intention to the task generation unit 30;

[0049] The task generation unit 30 is configured to receive the future access intention, and determine whether the future access intention satisfies a preset optimization condition; when the optimization condition is satisfied, the task generation unit 30 obtains a corresponding data processing workflow from the policy engine, and generates an asynchronous optimization task; the asynchronous optimization task is sent to the execution unit 40;

[0050] An execution unit 40 is configured to execute the asynchronous optimization task; the execution unit 40 schedules the distributed storage agent located on the source physical storage system and the target physical storage system, controls the distributed storage agent to read the original data from the source physical storage system, performs in-transit computation during the data transmission process, and writes the generated derived data to the target physical storage system; after the task is completed, the execution unit 40 notifies the metadata management unit 50;

[0051] The metadata management unit 50 is configured to update the metadata of the unified logical namespace after receiving the task completion notification, and establish a new data view associated with the derived data for the logical path.

[0052] A policy engine is configured to store and manage the data policy associated with the logical path, the data processing workflow, and the task context information of the AI task; the policy engine provides the required information to the prediction unit 20 and the task generation unit 30.

[0053] In a specific embodiment, the monitoring unit 10, the prediction unit 20, the task generation unit 30, the execution unit 40, the metadata management unit 50, and the policy engine communicate with each other through a predefined application program interface (API) or an internal message queue to ensure reliable transmission and synchronization of data and instructions; for example, the monitoring unit 10 can submit the generated I / O image data to the prediction unit 20 by calling a POST / api / v1 / io_profile interface.

[0054] This part will describe the steps of the multi-user storage docking method of the AI platform provided by the present application in detail with reference to the flowchart shown in Figure 3 The flowchart of the multi-user storage docking method of the AI platform provided by an embodiment of the present application; the method is executed in the system and device shown in Figure 3 and the flowchart of the multi-user storage docking method of the AI platform provided by an embodiment of the present application; the method is executed in the system and device shown in Figure 1 and the flowchart of the multi-user storage docking method of the AI platform provided by an embodiment of the present application; the method is executed in the system and device shown in Figure 2

[0055] With reference to the system and device shown in Figure 3 , the flowchart of the multi-user storage docking method of the AI platform provided by an embodiment of the present application; the method is executed in the system and device shown in Figure 3

[0056] When the AI task initiates data access to a logical path through the unified logical namespace, the data access request is routed to the distributed storage agent corresponding to the target physical storage system; the distributed storage agent is responsible for monitoring and recording all I / O operations for the logical path within a preset time window to form an I / O operation sequence; based on the I / O operation sequence, the distributed storage agent extracts and calculates a set of time-series access features, and finally generates a structured I / O image.

[0057] With reference to the​​Figure 4 , attached Figure 4 is a data structure diagram of I / O image in embodiments of the present application; I / O image is a vectorized data structure, which includes multiple feature fields for describing data access patterns; in a specific embodiment, I / O image includes access sequentiality score, data read-write granularity distribution and I / O operation frequency.

[0058] Access sequentiality score is a quantitative value for representing the continuity of I / O operations; distributed storage agent first extracts the file offset of each read or write operation from the I / O operation sequence to form an offset sequence; based on the offset sequence, the access sequentiality score is calculated using the following formula:

[0059] ;

[0060] In the formula: is the access sequentiality score; is the total number of operations in the I / O operation sequence; is the file offset of the i-th I / O operation; is the data block size of the i-th I / O operation; is a decision function, when its argument is equal to 0, the function value is 1, indicating that the i-th operation occurs immediately after the i-1-th operation; when the argument is not equal to 0, the function value is 0. Data read-write granularity distribution is used to describe the statistical characteristics of AI task read-write data block size; distributed storage agent counts the data block size of all operations in the I / O operation sequence and calculates its mean, variance or generates a histogram to represent whether the data access granularity tends to large file read-write or small file read-write. For example, the histogram can be divided into multiple preset intervals, such as "<4KB", "4KB-64KB", "64KB-1MB" and ">1MB", and the percentage of I / O operation number falling into each interval is counted to form a vector that can represent the granularity characteristics.

[0061]

[0062] I / O operation frequency is used to describe the intensity of I / O request; distributed storage agent calculates the total number of I / O operations completed per unit time (IOPS) and the total amount of data transmitted per unit time (throughput) to represent the strength of current data access load.

[0063]

[0064] ​​​​Finally, the distributed storage agent combines the calculated feature values such as access orderliness score, data read-write granularity distribution, and I / O operation frequency into an I / O image, and sends it to the prediction unit 20 for subsequent future access intention prediction.

[0065] After the prediction unit 20 receives the I / O image generated by the distributed storage agent, it performs step S2; the core of this step is to combine the I / O access pattern observed from the physical layer with the task context of the AI task obtained from the application layer, so as to make a prediction on the subsequent data access behavior of the AI task and quantify it as a future access intention.

[0066] Referring to the accompanying drawings Figure 5 , the accompanying drawings Figure 5 are schematic diagrams of task state space and projection operation in embodiments of the present application; the prediction unit 20 first constructs a task state space according to the task context of the AI task obtained from the strategy engine; the task context includes the type of the AI task (such as model training, model inference) and the current running phase (such as data loading phase, model calculation phase, checkpoint saving phase); the task state space is a multi-dimensional semantic space, and different regions or dimensions of the task state space directly correspond to specific running phases in the task context, such as data loading region, model calculation region.

[0067] Then, the prediction unit 20 takes the timing access features (such as high orderliness, large granularity read-write) in the I / O image as an input vector and performs a projection operation; the projection operation is a mapping process that maps the input vector to the task state space; the result of the mapping is a state vector; the final position of the state vector in the task state space represents the system's judgment of the real intention behind the current I / O behavior; for example, when a high-order, large-granularity read I / O image is projected into the task state space, if its state vector falls into the data loading region, the system determines that its future access intention is to start large-scale model training data reading.

[0068] For subsequent automated processing, the future access intention is further quantified into a specific numerical value, i.e., a prefetch intention score; the prediction unit 20 calculates the prefetch intention score based on the I / O image vector and the task context using the following formula:

[0069] ;

[0070] Wherein: is the prefetch intention score, a scalar value; is the I / O image vector composed of timing access features; is the task context; These are preset weighting coefficients used to adjust the contribution of different temporal access features to the final score; For the first I / O profile vector The processing function for each component is used to normalize or transform the eigenvalues; A transformation function for converting task context into a numerical intent enhancement factor, used to map discrete, non-numerical task contexts (e.g., data loading phase) to a numerical intent enhancement factor.

[0071] In one specific embodiment, the processing function It could be a linear normalization function that maps different time-series access feature values ​​to an interval for weighted summation; a transformation function. It can be a lookup function that converts the task context into specific values ​​based on a predefined mapping table; for example, the mapping table can be defined as: {'Data loading stage': 1.5, 'Model calculation stage': 0.5, 'Checkpoint saving stage': 1.2}, where a higher value indicates that the stage has a stronger prefetching tendency.

[0072] After the calculation is completed, the prediction unit 20 will obtain the prefetching intention score. The task is sent to the task generation unit 30 so that it can determine whether an optimization task needs to be triggered.

[0073] Task generation unit 30 receives the prefetching intent score calculated by prediction unit 20. Then, step S3 is executed; the purpose of this step is to determine whether the current data access mode meets a preset condition that is worthwhile for storage optimization, and if the condition is met, to construct a specific and executable asynchronous optimization task.

[0074] The task generation unit 30 first queries the strategy engine for the data strategy associated with the current logical path; this data strategy defines the minimum and preferred performance levels allowed for this logical path; simultaneously, the system learns the performance level of the source physical storage system where the original data resides; based on the prefetching intent score, the performance level of the source physical storage system, and the preferred performance level defined in the data strategy, the task generation unit 30 makes a judgment according to the following asynchronous optimization task triggering conditions:

[0075] ;

[0076] In the formula: It is a Boolean value; when its value is true, it indicates that the triggering condition is met. Score for pre-fetching intention; A preset intent score threshold is used to define the strength of the access intent; a performance level value of a source physical storage system where the current data is located; a value of a preferred performance level defined in the data policy for the logical path; the higher the performance level value, the stronger the performance of the storage system.

[0077] The performance level value is pre-configured by the system administrator according to the hardware medium (such as NVMe-SSD, SATA-SSD, HDD) of the physical storage system, network bandwidth, and IOPS performance indicators; for example, a storage system based on NVMe-SSD can be defined as performance level 3, a storage system based on SATA-SSD can be defined as performance level 2, and a storage system based on HDD can be defined as performance level 1.

[0078] is true, that is, the pre-fetch intention score exceeds the threshold value, and the performance level of the current data storage is lower than the preferred performance level required by the policy. is true, that is, the pre-fetch intention score exceeds the threshold value, and the performance level of the current data storage is lower than the preferred performance level required by the policy.

[0079] Referring to FIG. 4, FIG. 4 is a flowchart of the task generation process of the task generation unit 30. Figure 6 , FIG. 5 is a schematic diagram of a data structure of an asynchronous optimization task in the embodiment of the present application; the task generation unit 30 generates a structured asynchronous optimization task, which contains all the information required for execution; the asynchronous optimization task includes: task ID, source information, target information, and data processing workflow. Figure 6 The task ID is a unique identifier of the task; the source information specifies the specific location of the original data, including the identifier of the source physical storage system and the physical path of the data therein; the target information specifies the target physical storage system to which the derived data should be written, which is usually a storage system that meets the preferred performance level in the data policy.

[0080] The data processing workflow is an ordered list of data processing plugins obtained from the policy engine according to the AI task type; the data processing plugin list specifies the data preprocessing operations and their order that need to be performed on the original data during data transmission.

[0081] Finally, the task generation unit 30 sends the constructed asynchronous optimization task to the execution unit 40 for scheduling and execution.

[0082] The execution unit 40 starts step S4 after receiving the asynchronous optimization task issued by the task generation unit 30; step S4 is responsible for the actual reading, processing, and writing operations of the data, and the core is to integrate the data preprocessing process with the data transmission process.

[0083] Referring to FIG. 6, FIG. 6 is a flowchart of the execution process of the execution unit 40.

[0084] Figure 7 , FIG. 7 is a schematic diagram of a data structure of an asynchronous optimization task in the embodiment of the present application; the task generation unit 30 generates a structured asynchronous optimization task, which contains all the information required for execution; the asynchronous optimization task includes: task ID, source information, target information, and data processing workflow. Figure 7 ​This is a schematic diagram of the data flow for computation in transit in an embodiment of the present invention; the execution unit 40 parses the source information and target information in the asynchronous optimization task, and issues instructions to the distributed storage agent A located in the source physical storage system and the target physical storage system and the distributed storage agent B on the target side.

[0085] According to the instructions, the source-side distributed storage agent A reads the raw data from the source physical storage system. The raw data is not sent directly to the target physical storage system, but enters an in-transit computing pipeline in the form of a data stream. The in-transit computing pipeline is composed of one or more data processing plug-ins connected in sequence according to the data processing workflow defined in the asynchronous optimization task.

[0086] The execution process of the computation workflow in transit can be formally expressed by the following formula:

[0087] ;

[0088] In the formula: This is the final derived data for the pipeline output during the calculation process; The raw data read from the source physical storage system; The first in the data processing workflow The data transformation functions executed by a data processing plugin, for example... It can be an image decoding function. It can be a data normalization function; The total number of data processing plugins included in the data processing workflow; This is a function compounding operator, indicating that the output of the previous function is the input of the next function, meaning that the data stream is processed sequentially through each plugin in the workflow.

[0089] like Figure 7 As shown, the raw data first flows through the first data processing plugin (implementing the function). This generates intermediate data, which then flows through a second data processing plugin (implementing the function). This process continues until the last plugin in the workflow is completed; this process can be completed by the source-side distributed storage agent A, the target-side distributed storage agent B, or both collaboratively.

[0090] The specific execution location of the computation pipeline in transit can be selected based on the load balancing strategy. For example, if the computing resources of the node where the source physical storage system is located are idle, most of the computation can be executed on the distributed storage agent A, which can reduce the amount of data transmitted in the network. Conversely, if the load on the source end is high, more computation tasks can be allocated to the distributed storage agent B on the target side.

[0091] Finally, the derived data generated after the whole in-transit computing pipeline is processed is received by the distributed storage proxy B on the target side and written to the target physical storage system; after the task is completed, the execution unit 40 sends a task completion notification to the metadata management unit 50.

[0092] After receiving the task completion notification from the execution unit 40, the metadata management unit 50 starts the subsequent processing flow; the purpose of this flow is to record the results of this data optimization at the metadata level of the unified logical namespace, and to ensure that the subsequent data access of the AI task can be correctly routed to the optimal data copy.

[0093] Before the asynchronous optimization task is executed, only one original data view pointing to the original data in the source physical storage system exists in the metadata of the logical path; after the derived data is successfully written to the target physical storage system, the metadata management unit 50 adds a derived data view to the metadata for the logical path; the derived data view includes the physical location information pointing to the newly generated derived data in the target physical storage system, and the identification of the data processing workflow used to generate the derived data.

[0094] The updated metadata maintains multiple mapping relationships for the logical path; one logical path can correspond to one original data view and zero or more derived data views; this multi-data view structure can be formally described by the following formula:

[0095] ;

[0096] In the formula: represents the metadata mapping set of the logical path ; is the original data view pointing to the original data, and its structure is (physical storage system ID, physical path); is the th derived data view, and its structure is (physical storage system ID, physical path, data processing workflow ID).

[0097] By maintaining such a multi-data view, the system shields the complexity of the underlying physical storage for the upper-layer AI task; when a new data access request arrives at the AI platform, the system needs to decide which physical data copy to route the request to; for this purpose, the system uses a transparent access routing function to make the selection.

[0098] The transparent access routing function selects a most matching view from the metadata mapping set based on the task context of the AI task associated with the new data access request and the data processing requirements of the task; the routing process can be represented by the following function:

[0099] ;

[0100] wherein: is the final selected data view; is the transparent access routing function; is the task context of the AI task associated with the new data access request; is the set of metadata mappings of the current logical path;

[0101] The routing function decides as follows: check if the required data processing workflow matches a data processing workflow ID in some derived data view ; if there is a matching derived data view, select that derived data view; if there is no matching derived data view, or the new AI task does not require any pre-processing, select the original data view ;

[0102] Finally, the new data access request will be routed to the physical storage system pointed by for execution.

[0103] In a more specific embodiment, if there is no derived data view that exactly matches the required processing, the routing function may also calculate the additional computational cost required to transform from each of the existing views (including the original view ) to the target data state, and select the view with the lowest cost as the starting point for routing and processing; this maximizes the reuse of existing derived data and further reduces the overall overhead of data processing.

[0104] While embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, combinations, and variations can be made by those skilled in the art without departing from the principles and spirits of the present application, the scope of which is defined by the appended claims and their equivalents.​​

Claims

1. A multi-user storage docking method of an AI platform, characterized in that, The method comprises the following steps: S1, monitoring data access of an AI task to a logical path in a unified logical namespace, and generating an I / O image of the data access; S2, projecting time access features contained in the I / O image into a task state space defined by a task context of the AI task, to quantify a future access intention of the AI task to the logical path; Wherein, the step of quantifying the future access intention of the AI task to the logical path comprises: defining a task running phase in the task context as a discrete dimension of the task state space, and the task state space is a multi-dimensional semantic space; projecting the time access features as an input vector into a state vector in the task state space through a mapping operation, a position of the state vector represents the future access intention, and the future access intention is quantified as a prefetch intention score, which is calculated based on the I / O image vector and the task context; S3, in response to the future access intention satisfying a preset optimization condition, generating an asynchronous optimization task containing a data processing workflow, the asynchronous optimization task specifying a source physical storage system and a target physical storage system of data; S4, executing the asynchronous optimization task, reading original data from the source physical storage system, and performing in-transit calculation on the original data according to the data processing workflow to generate derived data during transmission of the original data to the target physical storage system, and writing the derived data to the target physical storage system. 2.The multi-user storage docking method of an AI platform according to claim 1, wherein, In step S1, the step of generating the I / O image of the data access comprises: collecting an I / O operation sequence of the data access, and extracting time access features from the I / O operation sequence, the time access features including access sequence, data read-write granularity distribution, and I / O operation frequency. 3.The multi-user storage docking method of an AI platform of claim 1, wherein, In step S3, the step of responding to the future access intention satisfying the preset optimization condition comprises: a spatial distance between the state vector and a preset high intention region in the task state space is less than a preset distance threshold, and a performance level of the source physical storage system is lower than a preferred performance level in a data policy defined for the logical path.

4. The multi-user storage docking method of an AI platform according to claim 1, wherein, The data processing workflow is an ordered combination of one or more data processing plugins associated with the task type of the AI task; the data processing plugins include image decoding, data normalization, or feature extraction plugins.

5. The multi-user storage docking method of an AI platform according to claim 4, wherein, In step S4, the step of performing in-transit calculation on the original data according to the data processing workflow to generate derived data comprises: a distributed storage agent deployed in the source physical storage system or the target physical storage system sequentially calls the data processing plugins according to the order of the data processing plugins in the data processing workflow to perform data preprocessing operations on the original data flowing through the distributed storage agent.

6. The multi-user storage docking method of an AI platform according to claim 1, wherein, The method further comprises: After the derived data is successfully written into the target physical storage system, metadata of the uniform logical namespace is updated to generate a derived data view for the logical path, the derived data view being associated with a physical storage location of the derived data; The metadata maintains a plurality of mapping relationships for the logical path, the mapping relationships pointing to an original data view of the original data and a derived data view of the derived data.

7. The multi-user storage docking method of an AI platform according to claim 6, wherein, The method further includes: When a new data access request is received, the derived data view or the original data view is selected from the plurality of mapping relationships based on a task context of an AI task associated with the new data access request, and the new data access request is routed to a physical storage system corresponding to the selected view.

8. The multi-user storage docking method of an AI platform according to claim 1, wherein, The AI platform interfaces with a plurality of heterogeneous physical storage systems and deploys a distributed storage agent for each physical storage system, the distributed storage agent being responsible for performing in-transit computation and protocol conversion with the physical storage system.

9. A multi-user storage docking AI system of an AI platform, applied to the method of any one of claims 1-8. The system includes: A monitoring unit configured to monitor data access of an AI task to a logical path in a uniform logical namespace and generate an I / O image of the data access; A prediction unit configured to project time-series access features contained in the I / O image into a task state space defined by a task context of the AI task to quantify a future access intention of the AI task to the logical path; A task generation unit configured to generate an asynchronous optimization task including a data processing workflow in response to the future access intention satisfying a preset optimization condition, the asynchronous optimization task specifying a source physical storage system and a target physical storage system of data; An execution unit configured to execute the asynchronous optimization task, read original data from the source physical storage system, and perform in-transit computation on the original data according to the data processing workflow to generate derived data during transmission of the original data to the target physical storage system, and write the derived data into the target physical storage system.

Images