MDSplus Scientific Experiment Data Download and Normalization Delivery Method

By performing permission and availability checks before downloading experimental data in MDSplus, assessing data volume and time, and generating a processing plan, the problem of not being able to confirm delivery conditions before data delivery is solved. This enables direct use of data files and consistent processing, improving the predictability and efficiency of data delivery.

CN123115234APending Publication Date: 2026-11-17HEFEI INSTITUTE OF PHYSICAL SCIENCE CHINESE ACADEMY OF SCIENCES
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611601391.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Before downloading experimental data from MDSplus, it is difficult to confirm the delivery conditions of the target data, obtain the scale and time expectation of the data download task, and the target data cannot be formed into a directly usable data file according to the data processing requirements specified by the user.

Method used

By receiving natural language data delivery requests, parsing experimental data access elements and data delivery elements, retrieving MDSplus metadata sources, determining download permissions and data availability, assessing data volume and download time, generating a download and processing plan, reading and processing data according to the plan, and forming downloadable files that conform to the target format.

Benefits of technology

Before the actual download, identify tasks that are not authorized or cannot be completed, reduce invalid data access, improve the predictability of data delivery tasks, ensure that the output data corresponds to the user's requirements, and reduce redundant data processing and format conversion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115234A_ABST
    Figure CN123115234A_ABST
Patent Text Reader

Abstract

This invention discloses a method for downloading and normalizing MDSplus scientific experimental data, belonging to the field of scientific experimental database data access and processing technology. The method parses natural language data delivery requests to determine the target data object in MDSplus; before actual downloading, it performs download permission and data availability checks; after successful checks, it assesses the amount of data to be downloaded and the download time, and generates a download and processing plan based on data delivery elements; it reads the target data according to the plan, resamples and normalizes it according to the target sampling scale, and exports a downloadable file according to the target download format. This invention can reduce invalid downloads in cases of unauthorized access or unavailable data, improve the predictability of data download tasks, and enable processed experimental data to directly form data files that meet user delivery requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of scientific experiment database data access and processing technology, specifically to the MDSplus scientific experiment data download and normalized delivery method. Background Technology

[0002] MDSplus is a data management system for organizing and accessing scientific experimental data. In nuclear fusion experimental devices such as tokamaks, experimental data can be organized hierarchically according to experimental trees, experimental launch numbers, and signal nodes. Researchers typically connect to the MDSplus data server through a client program, read experimental signals according to the target experimental tree, experimental launch number, and node path, and use the obtained data for plotting, statistical analysis, model training, or multi-launch data comparison.

[0003] Chinese patent document CN122226524A (publication date: June 16, 2026) discloses an MDSplus authentication and fine-grained access control method. Addressing the identity authentication and access control issues in the native MDSplus access mechanism, it sets up a proxy gateway between the client and the MDSplus data server. The proxy gateway can parse client requests, perform identity authentication, and implement fine-grained access verification based on access elements such as user, experimental tree, gun number, and signal. After successful access verification, the request is forwarded to the MDSplus data server, and the returned data is transmitted to the client. This type of technology can implement secure control over data access behavior, but it primarily addresses whether a user can access the target data and whether data requests can be securely forwarded.

[0004] Chinese patent document CN122614890A (publication date August 21, 2026) discloses an MDSplus access code generation method based on a large language model. Addressing the tediousness of manually determining experimental trees, shot numbers, and node paths, as well as manually writing access codes, this method utilizes a large language model to generate MDSplus data access codes. This method can parse natural language data access requests, extracting access elements such as experimental shot numbers, experimental tree names, signal names, node paths, physical variables, or time windows, and retrieving candidate access objects from the MDSplus metadata source. After filtering candidate objects using information such as shot number availability and signal alias mapping relationships, the final access object can be determined, and MDSplus access codes can be generated under the corresponding metadata constraints. Some solutions can also perform trial runs on the generated code to identify problems such as server connection failures, non-existent experimental trees, non-existent shot numbers, non-existent node paths, or empty data.

[0005] The above solutions improve the security of MDSplus data access and the access code generation process, respectively. However, researchers' actual needs are not always limited to obtaining access codes or raw node data. For example, when processing high-frequency experimental waveforms, users may further specify the data time window, final sampling size, normalization method, and target file format. In this case, it is still necessary to consider whether the target data can meet the delivery conditions, the data size required for complete download, and the estimated processing time, and further complete time window truncation, resampling, normalization, and file format conversion.

[0006] For experimental signals with large datasets, if the corresponding task scale information is not available before complete data acquisition, users will find it difficult to predict file size, memory usage, and task duration. For tasks where the target node lacks data, the time range does not match, or the current data length does not meet delivery requirements, it may only be discovered during the actual data acquisition process that the expected delivery cannot be completed. After data acquisition, if separate programs for data preprocessing, resampling, normalization, and format conversion are required, it will be difficult to maintain consistency in the processing flow and standards between different data tasks.

[0007] Therefore, how to confirm the delivery conditions of the target data and obtain the scale and time expectation of the data download task before the MDSplus experimental data is officially downloaded, while ensuring that the target data is formed into a directly usable data file according to the data processing requirements specified by the user, has become an urgent technical problem to be solved. Summary of the Invention

[0008] The main objective of this invention is to provide a method for downloading and normalizing MDSplus scientific experimental data. This method aims to confirm the delivery conditions of the target data and obtain the scale and time expectation of the data download task before the formal download of MDSplus experimental data, while ensuring that the target data is formed into a directly usable data file according to the data processing requirements specified by the user.

[0009] To achieve the above objectives, this invention proposes an MDSplus scientific experiment data download and normalization delivery method, comprising: receiving a natural language data delivery request; parsing to obtain experimental data access elements and data delivery elements, wherein the data delivery elements include a target download format, a target sampling scale, and a normalization method; retrieving the MDSplus metadata source based on the experimental data access elements to determine a target data object including a target experimental tree, experimental gun number, signal node, and node path; before complete download, determining download permissions based on the current user identity and the target data object to determine read or download permissions, and determining data availability based on the metadata source and the results of a trial read to determine whether the target data object exists, whether the node has data, and whether the requested time range within the user-specified time window is covered by data, wherein the trial read... A complete download is not performed. After all the aforementioned judgments pass, the data volume is evaluated based on the target data's metadata, data dimensions, time window, sampling frequency, and target download format. The download time is evaluated based on historical reading speed, server response time, and network status. The download time evaluation includes the task time for reading, processing, and file writing. Based on the judgment results, evaluation results, and data delivery elements, a download and processing plan is generated, including data reading, preprocessing, resampling, normalization, and format export. The MDSplus server is connected according to the plan. The target signal and time dimension are read, data is extracted according to the time window, and resampling or sampling is performed according to the target sampling scale. Normalized data is generated from the preprocessed data according to the normalization method. The normalized data and the original data to be retained by the user are written into a downloadable file according to the target download format, and the file and execution results are returned.

[0010] Preferably, the experimental data access elements include at least one of the following: experimental device name, experimental gun number, experimental tree name, signal name, node path, physical variable, time window, and time dimension requirements; the data delivery elements also include at least one of the following: whether to retain the original data, whether to retain the time column, file naming rules, and download path.

[0011] Preferably, when the experimental tree name or node path is not explicitly given in the natural language data delivery request, candidate experimental trees, signal nodes and node paths are retrieved based on the signal name, physical variables and experimental gun number information already in the request, using the MDSplus metadata source and signal alias mapping relationship. The final target data object is then selected based on the matching of the candidate object with the parsed access elements, the data availability status under the specified gun number and the access conditions explicitly stated in the user request.

[0012] Preferably, the download permission determination is also based on the data level of the target data; the permission information used for the download permission determination comes from at least one of the following: permission database, access control table, unified identity authentication system, experimental data platform interface, configuration file, MDSplus server return status, and trial access results.

[0013] Preferably, the data availability determination further includes determining at least one of the following: whether the time dimension of the target data is readable, whether the target data is empty, whether the user-specified time window is within the actual time range of the target data, and whether the length of the target data meets the target sampling scale; when it is determined that the target data does not meet the conditions for download and delivery, the download task is stopped and the corresponding unavailability reason is returned.

[0014] Preferably, the data volume assessment includes determining at least one of the following: the number of original data points, the number of data points after time window truncation, the number of data points after resampling, the number of variables, the expected memory usage, the expected file size, and the compressed file size.

[0015] Preferably, the download time assessment is also based on at least one of the following: the number of segmented reads, the data preprocessing time, and the file writing time.

[0016] Preferably, the download and processing plan includes at least one of the following: direct reading, reading by time window, segmented reading, batch reading, caching intermediate results, resampling, normalization, format conversion, compression and packaging, and processing strategies that terminate the task and return the failure stage and reason when downloading or processing cannot continue.

[0017] Preferably, after reading the target data, at least one of the following preprocessing methods is performed on the target data: time window truncation, missing value handling, outlier handling, time axis alignment, unit retention, and data type conversion; when the natural language data delivery request specifies a target sampling size, resampling or sampling is performed according to the time axis of the target data and the length of the original data so that the processed data meets the target sampling size.

[0018] Preferably, the normalization method includes at least one of min-max normalization, z-score normalization, normalization by physical quantity reference range, normalization by single-shot data, normalization by multi-shot overall data, and normalization by signal separately; during the normalization process, the normalization method, the range of data involved in the calculation, and at least one of the rules for handling maximum, minimum, mean, standard deviation, and outlier values ​​are recorded.

[0019] Preferably, the target download format includes at least one of CSV, TXT, JSON, HDF5, MAT, NPY, NPZ, Parquet, and compressed file formats; the downloadable file contains normalized data and saves at least one of the following according to the data delivery request: time array, raw data, experimental gun number, experimental tree name, node path, signal unit, normalization parameter, and processing record; the downloadable file is returned to the user, along with at least one of the following: file size, actual download time, actual processing time, data summary, and processing log; when downloading or processing fails, the failure stage and reason for failure are returned.

[0020] Compared with the prior art, the present invention has the following beneficial effects: This application performs permission checks based on the current user's identity and the target data object before actual downloading, and further checks the data availability status of the target experimental tree, experimental gun number, node path, and target node. This allows for the identification of unauthorized tasks and tasks that cannot complete data delivery before full data reading, reducing invalid data access. After successful checks, the application assesses the amount of data to be downloaded and the download time, and integrates the assessment results, target download format, target sampling scale, and normalization method into the download and processing plan generation. This allows the resource assessment results to further influence the actual data reading and processing process, improving the predictability of data delivery tasks. After completing the target data reading, resampling, and normalization according to the download and processing plan, a downloadable file is directly generated according to the target format. This reduces the need for users to perform repetitive data processing and format conversion after obtaining the original MDSplus data, and ensures that the output data corresponds to the user's data delivery requirements. Attached Figure Description

[0021] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1 The overall flowchart of the MDSplus scientific experiment data download and normalized delivery method provided in the embodiments of this application is shown. Detailed Implementation

[0022] The technical solution of this application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are used to illustrate the technical solution of this application and the cooperation relationship between the various technical steps, and are not intended to limit the protection scope of this application. For conventional communication, data storage, program calling and file writing operations that can be directly understood by those skilled in the art, conventional implementation methods in the art can be used.

[0023] Example 1 This embodiment provides a method for downloading and normalizing MDSplus scientific experiment data. This method can be executed by a data access agent. The data access agent can run on a data service platform or computer device capable of accessing the MDSplus scientific experiment database, and interacts with the MDSplus metadata source, permission information source, MDSplus data server, and file storage location through a conventional communication interface. The data access agent invokes a pre-configured program to perform natural language request parsing, target data object determination, pre-download judgment, resource assessment, task planning, data reading, data processing, and file export. In this application, the data access agent refers to the execution entity running on a data service platform or computer device that invokes a pre-configured program to perform the aforementioned data access and delivery steps; experimental data access elements refer to request fields used to retrieve and locate experimental data objects; data delivery elements refer to request fields used to specify delivery conditions such as file format, sampling scale, and normalization processing; the target data object refers to the experimental data to be accessed, determined by the experimental tree, experimental gun number, signal node, and node path.

[0024] like Figure 1 As shown, the overall process of this embodiment includes steps S1 to S5 in sequence: Step S1 parses the natural language data delivery request and determines the target data object; Step S2 performs download permission judgment and data availability judgment; Step S3 evaluates the data volume and download time and generates a download and processing plan; Step S4 reads the signal data and time dimension according to the plan, and performs time window interception, sampling scale processing and normalization; Step S5 generates a downloadable file according to the target download format and returns the delivery result. The above steps S1 to S5 correspond to the following steps in sequence: Figure 1 The five process frames are arranged from top to bottom.

[0025] This embodiment uses MDSplus scientific experimental data from the Experimental Advanced Superconducting Tokamak (EAST) nuclear fusion device as the processing object for illustration. The user-submitted natural language data delivery request is: download the data of the BII1 signal from shot 147001 between 2s and 4s, normalize it, save it as a CSV file, and retain 10,000 sampling points. This request simultaneously contains information about the target experimental data and the data delivery requirements; therefore, the data access agent first converts the natural language data delivery request into a structured data download task.

[0026] Step S1: Parse the natural language data delivery request and determine the target data object.

[0027] The data access agent first performs semantic parsing on the natural language data delivery request, extracting experimental data access elements from the request. In this embodiment, the parsed experimental data access elements include at least the experimental gun number 147001, the signal name BII1, and the time window of 2 to 4 seconds.

[0028] Experimental data access elements can be further included, based on the actual information contained in the request, such as the experimental apparatus name, experimental tree name, node path, physical variables, and time dimension requirements. For experimental data access elements explicitly given in the natural language request, the data access agent directly generates the corresponding structured fields. For experimental tree names or node paths not explicitly given in the request, the corresponding content is treated as access objects that need to be determined through subsequent metadata matching.

[0029] The data access agent simultaneously parses data delivery elements. In this embodiment, the target download format is CSV, the target sampling size is 10,000 sampling points, and the normalization method is min-max normalization. Based on the user's delivery requirements, data delivery elements such as whether to retain the original data, whether to retain the time column, file naming rules, and download path can also be generated. The target sampling size refers to the number of sampling points that the delivered file should contain, and is not equivalent to the total number of sampling points of the original data in the target signal node.

[0030] After completing the request parsing, the data access agent queries the MDSplus metadata source based on information such as the experimental gun number, signal name, physical variables, and time window. The MDSplus metadata source can provide at least the experimental tree name, gun number range, signal name, signal alias, node path, data dimension, and time dimension, which are related to the target data location. The MDSplus metadata source is the metadata source that provides the aforementioned target object location and data dimension information.

[0031] When a natural language request does not directly specify the experimental tree name or node path, the data access agent matches candidate target data objects based on signal names, physical variables, and experimental gun numbers. During the matching process, signal alias mapping relationships, historical access records, or node information provided by the user can also be considered. The data access agent determines the final target data object based on the degree of matching between the candidate object and the user request, the availability of the candidate object under the corresponding gun number, and the constraints in the user request. Here, the candidate target data object refers to the combination of candidate experimental trees, experimental gun numbers, signal nodes, and node paths formed through the metadata retrieval; the degree of matching refers to the consistency between the candidate object and the access elements such as experimental gun numbers, signal names, and physical variables already present in the request; availability refers to whether the candidate signal node has readable data under the corresponding experimental gun number; and user constraints refer to the time window and data delivery conditions specified in the request.

[0032] In this embodiment, after MDSplus metadata source matching, the target experimental tree is determined to be E1T5, the target experimental gun number is 147001, the node path corresponding to the target signal node is \BII1, and the time dimension reading method corresponding to this node is determined. Thus, the BII1 signal request expressed in the user's natural language is converted into a target data object that can be actually accessed by the MDSplus data server.

[0033] Step S2 involves performing permission checks and data availability checks before downloading.

[0034] Before performing a complete data download, the data access agent first determines whether the current user has permission to read or download the target data. In this embodiment, permission is determined based on the current user's identity, the target experimental tree E1T5, the experimental gun number 147001, the node path \BII1, and the corresponding data level.

[0035] The information used for permission determination can come from a permission database, access control list, unified identity authentication system, experimental data platform interface, configuration files, MDSplus server return status, or tentative access results. In actual implementation, one or more of these permission information sources can be selected based on the existing permission management methods of the experimental data platform, without requiring changes to the organizational structure of the MDSplus data itself.

[0036] When the permission assessment indicates that the current user does not have permission to read or download the target data, the data access agent stops the subsequent download task and returns the reason for insufficient permissions. If the range of data for which permissions can be requested can be determined from the permission information, the corresponding data range or information requiring manual confirmation can also be returned.

[0037] After the permission check is passed, the data access agent continues to perform a data availability check. The permission check determines whether the current user can access the target data, while the data availability check further confirms whether the target data itself can meet the current data delivery task. These two checks address different task conditions.

[0038] The data access agent determines whether the target experimental tree E1T5 exists, whether the experimental gun number 147001 exists, whether the node path \BII1 exists, and whether the node has readable data under the 147001 gun, based on the MDSplus metadata source and the results of lightweight exploratory reading.

[0039] Lightweight exploratory reads are used to confirm the readable status of the target data without performing the full data download required for this task. The data access agent also determines whether the time dimension of the target node is readable and obtains the actual time range corresponding to the target data. In this application, a lightweight exploratory read refers to a readable status check performed before formally downloading the complete target signal data, used to confirm whether the target node has readable data and the required time dimension, and is not intended to complete the file delivery of the target data.

[0040] Since the user request in this embodiment is limited to a time window of 2 to 4 seconds, the data access agent further determines whether this time window falls within the actual time range of the target data. Only when the time range requested by the user can be covered by the target data will the corresponding data delivery task continue to be executed.

[0041] The data access agent also checks whether the target data is empty and determines whether it meets the processing conditions for forming the target sampling scale based on the target data length. This embodiment requires that 10,000 sampling points be retained in the end. Therefore, confirming the data length before the complete download can eliminate situations that cannot meet the delivery requirements, such as empty target data, insufficient data length, or invalid time range.

[0042] If the experimental tree, gun number, node path, target signal, or time dimension cannot be read, or the user-specified time window exceeds the actual data range, the data access agent stops the current download task and returns the corresponding unavailability reason. After both permission and data availability checks pass, the target data enters the resource evaluation phase.

[0043] Step S3: Assess the data volume and download time, and generate a download and processing plan.

[0044] The data access agent first assesses the amount of data for the current data delivery task. The assessment is based on information including the target data object's metadata, data dimensions, user-specified time window, sampling frequency, number of signals, and target download format.

[0045] Data volume assessment can include at least one of the following: the number of original data points, the number of data points after time window truncation, the number of data points after resampling, the number of variables, the estimated memory usage, the estimated file size, and the compressed file size. For the number of data points and file size, estimations can be made based on the data dimensions, data types, and corresponding data ranges represented by the MDSplus metadata, using conventional data volume statistics methods.

[0046] In this embodiment, after evaluation based on the data length and time dimension of the target node, it is determined that the original data within the time range of 2s to 4s contains approximately 22,000 sampling points. Since the user requires a final delivery of 10,000 sampling points, the data access agent simultaneously uses these 10,000 sampling points as the target data size for subsequent resampling.

[0047] The data access agent further evaluates the download time required to complete the current task. This download time evaluation can be based on a combination of historical read speeds of similar nodes, MDSplus server response time, current network status, number of segmented reads, data preprocessing time, and file write time. For target nodes with historical task records, the corresponding historical read records can be used as the basis for evaluating the time of the current task.

[0048] In this embodiment, the download time assessment is not merely an estimate of the data transmission process within the network, but rather a comprehensive expectation of the time required for the current data delivery task, from data reading to processing and file writing. Therefore, when reading large amounts of data or processing multiple signals, task size information can be obtained before the complete data reading is complete.

[0049] After completing permission checks, data availability checks, data volume assessments, and download time assessments, the data access agent generates a download and processing plan based on the above results, the target download format, the target sampling size, and the normalization method.

[0050] The download and processing plan is used to determine subsequent data reading and processing procedures. The plan may include at least one of the following: direct reading, reading by time window, segmented reading, batch reading, caching intermediate results, resampling, normalization, format conversion, compression and packaging, and failure rollback strategies. The specific plan content adopted is determined based on the resource assessment results of the current target data object and the user's data delivery requirements. The download and processing plan, determined based on the current judgment results, resource assessment results, and delivery requirements, arranges the subsequent reading and processing steps to guide the reading, preprocessing, and final file generation of the target signal data.

[0051] In this embodiment, the generated download and processing plan includes reading the target node \BII1 and its corresponding time dimension, extracting 2s to 4s of data from the reading results, processing the extracted data into 10,000 sampling points, performing min-max normalization on the processed data, and exporting the processing results into a CSV file.

[0052] Therefore, the results of permission judgment and data availability judgment determine whether the task enters the formal download stage. The data volume and download time assessment results participate in the formation of the subsequent download processing plan. The download and processing plan is further used to guide the actual data reading, sampling processing, normalization and file output, forming a continuous data processing relationship.

[0053] Step S4: Read the target data according to the download and processing plan, and perform preprocessing and normalization.

[0054] The data access agent establishes a data access connection with the MDSplus data server according to the download and processing plan, opens the E1T5 experimental tree, and reads the signal data corresponding to node path \BII1 based on experimental gun number 147001. If the current task requires processing within a time range, it also reads the time dimension corresponding to that node.

[0055] After obtaining the raw signal values ​​and time array, the data access agent extracts the data corresponding to the specified time range from the raw data within a user-defined time window of 2 to 4 seconds. The time window truncation uses the time dimension of the node data as the basis for determining the data location, ensuring that the truncated signal data corresponds to the corresponding time data.

[0056] When real-world data contains missing or outlier values, the data access agent can perform missing and outlier handling according to pre-configured data processing rules. For data that simultaneously reads multiple related signals or requires a unified time base, time axis alignment can also be performed. Signal units in the read results can be retained with the processing results. Furthermore, conventional data type conversions can be performed based on subsequent calculations or file format requirements.

[0057] This embodiment requires retaining 10,000 sampling points. The data access agent performs resampling or sampling based on the timeline of the truncated data and the length of the original data, processing the original data within the range of 2 to 4 seconds into 10,000 sampling points. The aforementioned approximately 22,000 original sampling points are estimated values ​​formed based on the metadata of this embodiment. During the resampling process, the correspondence between the processed signal values ​​and the corresponding time positions is maintained. The specific resampling method can adopt conventional processing methods for time series data in this field, and this embodiment does not impose additional limitations on specific interpolation or sampling algorithms.

[0058] After completing the sampling scale processing, the data access agent processes the signal data according to the user-specified min-max normalization method. Let the first agent participate in the normalization... The data value is The minimum value within the current normalized data range is The maximum value is ;in, This indicates the index of the sampling point in the normalized data. Indicates the first The signal data value of each sampling point and These represent the minimum and maximum values ​​within the range of data involved in the calculation, respectively. At that time, the normalized first The data value is We obtain the following formula:

[0059] Through the above processing, signal values ​​within a specified data range are converted into a unified normalized representation. Min-max normalization is a data processing method that is directly understood in the art; the calculation relationship here is used to illustrate the specific execution process of the user-specified normalization requirements in this embodiment.

[0060] During the normalization process, the data access agent records the current normalization method, the range of data involved in the normalization, and the corresponding maximum and minimum values. Depending on the normalization method and data processing needs, the mean, standard deviation, and outlier handling rules can also be recorded. These processing records are then correlated with the processed experimental data so that the output file reflects the processing conditions used in this data processing.

[0061] The data access agent can simultaneously retain both the original data after time window truncation and sampling scale processing, and the corresponding normalized data. This embodiment retains both the original and normalized data while maintaining their correspondence with the time array, thus facilitating the selection of either the original or normalized signal values ​​in subsequent analysis.

[0062] Step S5: Export the data according to the target download format and return the download result.

[0063] After data preprocessing and normalization, the data access agent generates a data file according to the target download format in the data delivery elements. In this embodiment, the target download format is CSV, so the processed time array, raw signal data, and normalized signal data are written into the CSV file.

[0064] The CSV file generated in this embodiment includes at least the fields time, raw_value, and normalized_value. The time field is used to store the time data corresponding to each sampling point after resampling, the raw_value field is used to store the original signal value after resampling, and the normalized_value field is used to store the corresponding normalized signal value.

[0065] The data file can also store information related to this data delivery, such as the experimental shot number, experimental tree name, node path, signal unit, normalized parameters, and processing records. For information that is not suitable for direct writing to the main data column, a corresponding relationship can be established with the data file for storage, allowing users to identify the experimental data object corresponding to the file and its processing conditions.

[0066] In this embodiment, the generated file name can be east_147001_E1T5_BII1_2s_4s_minmax_10k.csv. This name reflects the experimental shot number, experimental tree, signal name, time range, normalization method, and target sampling size. The file naming method can also be executed according to the file naming rules specified in the user's request.

[0067] Once the file is generated, the data access agent returns the downloadable data file to the user, along with one or more of the following information: file size, actual download time, actual processing time, data summary, and processing log. By correlating the actual information after task execution with the evaluation results before downloading, the agent can provide the user with the execution status and results of the current data delivery task.

[0068] When a failure occurs during data reading, preprocessing, normalization, or file writing, the data access agent records the processing stage of the failure and returns the corresponding failure reason and processing information that can be determined based on the current execution status, thereby avoiding returning a uniform failure result to the user that cannot distinguish the location of the failure.

[0069] Through the above process, the user's natural language data delivery request is first converted into a clear MDSplus target data object and data delivery requirements. Before downloading the complete data, the data access agent performs permission checks and data availability checks to exclude tasks with unauthorized permissions, tasks where the target data does not exist, is empty, or cannot meet the specified time range.

[0070] Once the target data is ready for delivery, the data volume and download time are assessed, and a download and processing plan is developed based on the assessment results and the user's data delivery requirements. The actual reading process follows this plan, performing time window truncation, sampling scale processing, and normalization, ultimately directly generating a data file that meets the target format. Therefore, users do not need to write separate programs for data truncation, resampling, normalization, and format conversion after obtaining the raw MDSplus data; they can obtain the processing results corresponding to their data delivery requests.

[0071] Example 2 This embodiment, based on Embodiment 1, further explains the parsing of natural language data delivery requests, the completion of target data objects, and the task formation methods under different data delivery requirements.

[0072] User-submitted data delivery requests can include the complete experimental tree name, experimental gun number, and node path, or they can describe the required data using only the experimental device name, physical variable name, or signal name. When the user request does not explicitly provide the experimental tree name or node path, the data access agent first extracts the identifiable experimental data access elements from the request and treats the missing experimental tree name or node path as content to be determined.

[0073] The data access agent queries the MDSplus metadata source based on the experimental gun number, signal name, and physical variables, and combines this with signal alias mapping relationships to form candidate target data objects. Candidate target data objects can include candidate experimental trees, candidate signal nodes, and candidate node paths. When multiple candidate nodes correspond to the same natural language signal name, the data access agent filters them based on constraints such as the matching of candidate objects with the parsed access elements in the user request, the data availability status of candidate nodes under the specified experimental gun number, and the time window provided by the user, thus determining the final target data object.

[0074] Using the above method, even when users do not directly know the names and node paths of the internal experimental trees in MDSplus, their natural language descriptions can be mapped to actual data objects existing in MDSplus. After the target data object is determined, download permission checks, data availability checks, and subsequent data delivery processes are then executed.

[0075] In addition to the target download format, target sampling size, and normalization method, a data delivery request may also include at least one of the following: whether to retain the original data, whether to retain the time column, file naming rules, and download path. The data access agent stores the above data delivery elements together with the experimental data access elements in the current data download task, so that subsequent data reading, preprocessing, file generation, and result return can be executed according to the constraints of the same task.

[0076] When a user requests to retain both raw and normalized data, the data access agent maintains the correspondence between the raw and normalized data after normalization and exports them together according to the target file format. When a user only requests normalized data, the normalized data can be used as the primary delivery data. When a user requests to retain the time column, the processed time array is mapped one-to-one with the data from each sampling point and written to the target file.

[0077] When the user provides a file naming rule, the generated file will have a name that follows that rule. When the user provides a download path, the generated data file can be saved to the corresponding path or a file storage location corresponding to that path. File writing and path access can be performed using conventional methods that achieve the same functionality in computer data file processing.

[0078] Example 3 This embodiment further explains the different implementation methods of pre-download permission judgment and data availability judgment.

[0079] Before performing a complete data download, the data access agent obtains permission information based on the current user's identity, the target experimental tree, the target experimental cannon number, the node path, and the data level of the target data, and uses the permission information to determine whether the current user has the permission to read or download the target data.

[0080] Permission information can be provided by a permission database. In this case, the data access agent queries the relevant permission records based on the current user's identity and the target data object, and determines the range of data the current user can access based on the query results.

[0081] Permission information can also come from access control lists or unified identity authentication systems. For environments where an experimental data platform has already been deployed, the data access permissions for the current user can also be obtained through the experimental data platform interface.

[0082] In other implementations, permission information can be recorded in a configuration file or determined based on the status returned by the MDSplus server or the results of probing access. Different sources of permission information can be used individually or in combination according to the configuration of the existing experimental data management system.

[0083] If the permission check indicates that the current user does not have read or download permissions for the target data, the current complete data download will be stopped. The reason for insufficient permissions can then be returned. Based on the currently available permission information, the system can also return the data range for which the user can request permissions or information requiring further confirmation.

[0084] After the permission check is passed, the data access agent continues to determine data availability. This data availability determination does not solely rely on the user having access permissions; rather, it further confirms whether the target data itself can satisfy the current data delivery request.

[0085] First, the existence of the target experimental tree, target experimental gun number, and target node path is determined based on the MDSplus metadata source. If the metadata determination cannot fully reflect the actual data status of the node, a lightweight exploratory read can be performed to determine whether the target node has data under the current gun number and whether the data can be read.

[0086] When a user requests to read data based on a time range, it is also determined whether the time dimension of the target node is readable. After obtaining the time dimension, the user-specified time window is compared with the actual time range of the target data. If the user-specified delivery time window exceeds the actual time range of the target data, the target data object is determined to not meet the current data delivery conditions.

[0087] The data access agent can also determine whether the target data is empty. When the user specifies the target sampling size, it further determines whether the target data length meets the corresponding data processing requirements.

[0088] The different availability conditions mentioned above can be selected for judgment based on the current data delivery task. By completing these judgments before the complete data is read, user permission conditions and the deliverability conditions of the target data itself can be processed separately, thereby identifying states such as insufficient permissions, non-existent data objects, no data on the target node, missing time dimension, and time range not meeting requirements before the formal download.

[0089] Example 4 This embodiment further explains the process of data volume assessment and download time assessment.

[0090] After the target data passes the permission and data availability checks, the data access agent generates resource assessment information based on the target data object and the current data delivery requirements.

[0091] Information used for data volume assessment can include the target data's metadata, data dimensions, time window, sampling frequency, number of signals, and target file format. This information can be used to determine or estimate the original number of data points, the number of data points after the time window is truncated, and the number of data points after resampling.

[0092] For data delivery tasks involving multiple data variables, the number of variables can also be determined. By combining the data dimensions, the number of data points, and the data type, the expected memory usage during data processing can be estimated, following conventional computer data storage methods.

[0093] The data access agent can also estimate the expected file size based on the content of the data to be output and the target file format. When subsequent tasks include compression, it can also generate an estimate of the compressed file size. Not all of the above data volume metrics need to be used simultaneously; appropriate metrics can be selected based on the download and processing content of the current task.

[0094] Download time can be evaluated based on historical read speeds of similar nodes, MDSplus server response time, and current network conditions. If the current download processing plan involves segmented reads, the number of segmented reads can also be used as a time evaluation factor.

[0095] Since the data delivery result of this application is not only the raw data returned by the MDSplus server, the download time assessment can also take into account the data preprocessing time and file writing time. The resulting time assessment result can correspond to the complete task process from reading the target data to generating the target file.

[0096] For target nodes with historical data delivery records, the reading speed and processing time of similar data can be obtained from the historical records and used as the basis for evaluating the current task. For tasks without corresponding historical records, task time estimates can be made based on currently available metadata, server response status, and network status. This application does not limit the specific time prediction formula.

[0097] Both the data volume assessment results and the download time assessment results are provided to the download and processing plan generation process. In this way, the resource assessment is not merely a task prompt displayed to the user, but serves as input information for subsequent selection of data reading and processing methods.

[0098] Example 5 This embodiment further illustrates the formation and execution of the download and processing plan.

[0099] After the permission and data availability checks are passed, the data access agent uses the check results, data volume assessment results, download time assessment results, and the user-specified target download format, normalization method, and target sampling size as inputs to the download and processing plan.

[0100] The download and processing plan may include at least one of the following: direct reading, reading by time window, segmented reading, batch reading, caching intermediate results, resampling, normalization, format conversion, compression and packaging, and failure rollback strategy.

[0101] If the data required for the current task can be obtained through a single reading process, and the data delivery request does not have a specified timeframe, a data processing plan that includes direct reading can be created.

[0102] When users explicitly specify a time window, content processed according to the time window can be added to the plan. The data access agent determines the data corresponding to the requested time range based on the time dimension of the target node and performs subsequent processing on the corresponding data.

[0103] For tasks with a large amount of data to be processed, as indicated by the data volume assessment, segmented reading can be included in the download and processing plan. During segmented reading, the complete data reading process is divided into multiple reading processes, and subsequent data merging or corresponding processing is performed on the results of each part of the reading.

[0104] For tasks involving multiple data objects that can be processed together, the download and processing plan can also include batch reading. The data obtained from batch reading retains its target experimental tree, experimental gun number, signal node, or node path information, so that different data objects can be distinguished during subsequent processing and file delivery.

[0105] Intermediate results can also be cached during processing. Cached content can be data that has been read but not yet fully processed, or intermediate data that has undergone time window truncation, resampling, or other preprocessing. Caching can be implemented using conventional data storage methods in this field.

[0106] When the download and processing plan includes resampling, the data access agent processes the read data according to the target sampling size. When the plan includes normalization, normalized data is generated according to the normalization method specified in the data delivery request. When the plan includes format conversion, the processing results are converted to the target download format.

[0107] For tasks requiring compressed delivery results, the download and processing plan may include compressed packaging. If, during target data reading or data processing, a state arises that prevents the current plan from being completed, the current processing can be terminated according to a pre-defined failure rollback strategy, while retaining the identifiable failure stage and cause. The failure rollback strategy described in this application refers to a processing method that terminates the corresponding task and retains the identifiable failure stage and cause when data reading or data processing cannot continue according to the current plan.

[0108] The different plans mentioned above are not isolated from each other. The data access agent can combine multiple processing methods into the same download and processing plan based on the permission status, data availability status, data size, expected processing time, and target delivery requirements of the same task, and execute them in the data flow sequence specified in the plan.

[0109] Example 6 This embodiment further explains the preprocessing of MDSplus data after reading and the control of the target sampling scale.

[0110] The data access agent connects to the MDSplus data server according to the download and processing plan, opens the target experiment tree, and reads the target node data based on the target experiment gun number. When the task requires time information, it simultaneously reads the time dimension corresponding to the target data.

[0111] The read results can include the original signal value, time array, signal unit, node path, experimental gun number, experimental tree name, read time, and access status. The target data object identifier and information required for data processing within this information can be passed to subsequent data processing steps as the current task continues.

[0112] When the user specifies a time window, the data range corresponding to the time window is determined based on the time array, and the original signal data is truncated within the time window. The truncated signal data and the corresponding time data maintain the same data correspondence order.

[0113] For missing values ​​in the results, missing value handling can be performed. For cases with outlier data, outlier handling can be performed according to the current data processing rules. Missing value handling and outlier handling can adopt conventional data processing methods suitable for the data types in the current experiment.

[0114] Timeline alignment can be performed when the task contains data that requires a unified time base. After data is read, the original signal units can be retained, and data type conversion can be performed as needed for subsequent data calculations or file output.

[0115] When the user specifies a target sampling size, the data access agent performs resampling or sampling based on the timeline and the length of the original data. After resampling or sampling is completed, the number of sampled points in the exported data corresponds to the target sampling size specified by the user.

[0116] For example, if a user requests to export 10,000 sampling points, after completing the time window truncation, the data can be processed into 10,000 sampling points based on the time axis and data length of the truncated data. The correspondence between the time data and the signal data is maintained during the processing.

[0117] By incorporating time window truncation and target sampling size control into the same data delivery process, the final exported data range and size can directly correspond to the data delivery conditions submitted by the user, without requiring the user to perform the same data truncation and sampling processing again after obtaining the file.

[0118] Example 7 This embodiment further illustrates different normalization methods and normalization processing records.

[0119] After completing the necessary preprocessing, the data access agent normalizes the target data according to the normalization method specified in the natural language data delivery request. When the user does not further specify other processing conditions, the corresponding normalization can be performed according to the data range already determined in the data delivery task.

[0120] The normalization method can be min-max normalization. The implementation process can follow the method described in Example 1.

[0121] Normalization can also be achieved using z-score standardization. Let the z-score be the z-score that participates in the standardization process. Each signal data value is The mean of the data used in this calculation is The standard deviation is ;in, This represents the index of the sampling point in the data used for calculation. Indicates the first The signal value at each sampling point This represents the arithmetic mean of the data used in the calculation. This represents the standard deviation of the data set. At that time, the first The standardized result of each data value is We obtain the following formula:

[0122] When using z-score standardization, the data access agent records the corresponding mean and standard deviation.

[0123] Normalization can also be implemented according to the reference range of the physical quantity. In this case, the target data is normalized relative to the corresponding reference range based on the data processing requirements of the current physical quantity.

[0124] For single-shot experimental data, normalization can be performed according to the data range corresponding to the current experimental shot number. For processing tasks involving multi-shot experimental data, the data from all multiple shots can be used as the data range for the corresponding normalization process, and normalization can be performed on the overall data of all multiple shots.

[0125] For data processing tasks containing different signals, the data corresponding to each signal can be used as the normalization calculation range to perform normalization by signal.

[0126] The data access agent records the normalization method and the range of data involved in the calculation during the normalization process. Depending on the actual normalization method used, it can also record the corresponding parameters among the maximum, minimum, mean, and standard deviation. If the preceding processing includes outlier handling, the corresponding outlier handling rules can also be recorded as part of the current data processing record.

[0127] When a user requests that the original data be retained, normalization does not replace the existing processed data; instead, it generates normalized data that corresponds to the original data. If the user only requires the normalized result, the normalized data can be the primary data delivery content.

[0128] By saving the corresponding normalization parameters and the range of data involved in the calculation, the normalization results in the data file can be kept in correspondence with the processing conditions that led to those results.

[0129] Example 8 This embodiment further illustrates different target file formats and data delivery results.

[0130] After data processing is complete, the data access agent generates a downloadable file based on the target download format specified in the data delivery request. The target download format can be at least one of the following: CSV, TXT, JSON, HDF5, MAT, NPY, NPZ, Parquet, or a compressed file format.

[0131] When the target download format is CSV or TXT, the time array and signal data can be saved in the usual organization manner for corresponding text data files. When the target download format is JSON, the experimental data and its corresponding information can be saved in a structured data format.

[0132] When the target download format is HDF5 or MAT, the time array, raw data, normalized data, and target data object information can be saved according to the conventional data organization method of the corresponding file format. When the target download format is NPY or NPZ, the processed data can be saved according to the corresponding array file format.

[0133] When the target download format is Parquet, the processed structured experimental data can be written to a data file in the corresponding format. If delivery in a compressed package is required, one or more generated data files and their processing records can be compressed and packaged together.

[0134] The downloadable file may include at least one of the following: time array, raw data, normalized data, experimental shot number, experimental tree name, node path, signal unit, normalization parameter, and processing record. The specific content saved is determined by the data delivery request and the processing results of the current task.

[0135] After the file is generated, the data access agent can return the file itself and corresponding delivery information to the user. The delivery information may include at least one of the following: file size, actual download time, actual processing time, data summary, and processing log.

[0136] The data summary, formed based on the currently generated data and task information, is used to explain the basic status of the data delivery result to the user. The processing log records the data access and processing stages that the current task has executed. The data summary is an overview of the delivery status based on the generated data and task information, while the processing log records the data access and processing stages that the task has executed; both correspond to the target data object and the current delivery task.

[0137] When downloading or processing fails, the data access agent returns the failure stage and reason. The failure stage can correspond to the actual stage occurring during target data determination, permission judgment, data availability judgment, data reading, data preprocessing, normalization processing, or file writing. The returned failure reason is based on the execution status available for the current step, without requiring further data processing after a failure.

[0138] Through the above file generation and result return process, after the experimental data in MDSplus completes the target object determination, pre-download judgment, task planning and data processing, it can directly form data delivery results in the file format specified by the user, and maintain the correspondence between experimental data, processed data and corresponding processing information.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solution of this application, and are not intended to limit the scope of protection of this application. Based on an understanding of the technical concept of this application, those skilled in the art can make corresponding adjustments to the request parsing method, source of permission information, data availability judgment method, resource assessment content, download and processing plan, and data output format according to the actual MDSplus experimental data organization method, data access environment, and data delivery requirements.

[0140] Without departing from the technical concept of this application, adjustments can be made to the execution order of the above technical steps without affecting the technical logic, conventional data processing methods with the same function can be replaced with equivalent ones, or compatible technical features in the above embodiments can be combined to form corresponding implementation methods. As long as such adjustments, replacements, or combinations are still based on the technical relationships of target data object determination, pre-download permission judgment and data availability judgment, data volume and download time assessment, download and processing plan generation, data reading and processing according to the plan, and the formation of target format data delivery results, they all fall within the implementation forms that can be covered by the technical concept of this application.

Claims

1. A method for downloading and normalizing MDSplus scientific experimental data, characterized in that, include: Receive a natural language data delivery request, parse it to obtain experimental data access elements and data delivery elements, the data delivery elements including target download format, target sampling size and normalization method; Based on the experimental data access elements, the MDSplus metadata source is retrieved to determine the target data objects, including the target experimental tree, experimental gun number, signal node, and node path. Before a full download, download permissions are determined based on the current user identity and the target data object to determine read or download permissions. Data availability is determined based on the metadata source and the results of the trial read to determine whether the target data object exists, whether the node has data, and whether the requested time range within the user-specified time window is covered by data. Trial reads do not perform a full download. After all the aforementioned judgments are passed, the data volume is evaluated based on the target data's metadata, data dimensions, time window, sampling frequency, and target download format, and the download time is evaluated based on historical reading speed, server response time, and network status. The download time evaluation includes the task time for reading, processing, and writing files. Based on the judgment results, evaluation results, and data delivery elements, a download and processing plan is generated that includes data reading, preprocessing, resampling, normalization, and format export. Connect to the MDSplus server according to the plan, read the target signal and time dimension, extract data according to the time window, resample or sample according to the target sampling scale, and generate normalized data from the preprocessed data according to the normalization method. The system writes the normalized data and the original data that the user requests to retain into a downloadable file according to the target download format, and returns the file and the execution result.

2. The MDSplus scientific experiment data download and normalization delivery method according to claim 1, characterized in that, The experimental data access elements include at least one of the following: experimental device name, experimental gun number, experimental tree name, signal name, node path, physical variable, time window, and time dimension requirements; the data delivery elements also include at least one of the following: whether to retain the original data, whether to retain the time column, file naming rules, and download path.

3. The MDSplus scientific experiment data download and normalization delivery method according to claim 2, characterized in that, When the experimental tree name or node path is not explicitly given in the natural language data delivery request, candidate experimental trees, signal nodes and node paths are retrieved based on the signal name, physical variables and experimental gun number information already in the request, using the MDSplus metadata source and signal alias mapping relationship. The final target data object is then selected based on the matching of the candidate object with the parsed access elements, the data availability status under the specified gun number and the access conditions explicitly stated in the user request.

4. The MDSplus scientific experiment data download and normalization delivery method according to claim 1, characterized in that, The download permission determination is also based on the data level of the target data; the permission information used for the download permission determination comes from at least one of the following: permission database, access control table, unified identity authentication system, experimental data platform interface, configuration file, MDSplus server return status, and trial access results.

5. The MDSplus scientific experiment data download and normalization delivery method according to claim 4, characterized in that, The data availability determination also includes determining whether the time dimension of the target data is readable, whether the target data is empty, whether the user-specified time window is within the actual time range of the target data, and whether the length of the target data meets the target sampling scale at least one of the following: when it is determined that the target data does not meet the conditions for download and delivery, the download task is stopped and the corresponding unavailability reason is returned.

6. The MDSplus scientific experiment data download and normalization delivery method according to claim 1, characterized in that, The data volume assessment includes determining at least one of the following: the number of original data points, the number of data points after time window truncation, the number of data points after resampling, the number of variables, the expected memory usage, the expected file size, and the compressed file size.

7. The MDSplus scientific experiment data download and normalization delivery method according to claim 6, characterized in that, The download time assessment is also based on at least one of the following: the number of segmented reads, the data preprocessing time, and the file writing time.

8. The MDSplus scientific experiment data download and normalization delivery method according to claim 7, characterized in that, The download and processing plan includes at least one of the following: direct reading, reading by time window, segmented reading, batch reading, caching intermediate results, resampling, normalization, format conversion, compression and packaging, and processing strategies that terminate the task and return the failure stage and reason when downloading or processing cannot continue.

9. The MDSplus scientific experiment data download and normalization delivery method according to claim 1, characterized in that, After reading the target data, at least one of the following preprocessing methods is performed on the target data: time window truncation, missing value handling, outlier handling, time axis alignment, unit preservation, and data type conversion. When the natural language data delivery request specifies a target sampling size, resampling or sampling is performed according to the time axis of the target data and the length of the original data to make the processed data meet the target sampling size.

10. The MDSplus scientific experiment data download and normalization delivery method according to claim 9, characterized in that, The normalization method includes at least one of min-max normalization, z-score normalization, normalization by physical quantity reference range, normalization by single-shot data, normalization by multi-shot overall data, and normalization by signal separately; during the normalization process, the normalization method, the range of data involved in the calculation, and at least one of the rules for handling maximum, minimum, mean, standard deviation, and outlier values ​​are recorded.

11. The MDSplus scientific experiment data download and normalization delivery method according to claim 10, characterized in that, The target download format includes at least one of CSV, TXT, JSON, HDF5, MAT, NPY, NPZ, Parquet, and compressed file formats; the downloadable file contains normalized data and saves at least one of the following according to the data delivery request: time array, raw data, experimental gun number, experimental tree name, node path, signal unit, normalization parameter, and processing record; the downloadable file is returned to the user, along with at least one of the following: file size, actual download time, actual processing time, data summary, and processing log; when downloading or processing fails, the failure stage and reason for failure are returned.

Citation Information

Patent Citations

  • MDSplus authentication and fine-grained access control method, equipment and medium

    CN122226524A

  • MDSplus access code generation method based on large language model

    CN122614890A