A data migration method, system, device and medium for oiling laboratory historical data

Abstract

This invention discloses a method, system, equipment, and medium for migrating historical data from an oil and chemical laboratory, belonging to the field of data migration technology. The method includes: classifying historical data from the oil and chemical laboratory into unstructured and structured data using a classification model; performing corresponding standardization processing steps on the unstructured and structured data respectively to obtain standardized data; labeling the standardized data with experimental categories, experimental equipment, and timestamps; associating the unstructured and structured data according to the correspondence of historical data from the oil and chemical laboratory to generate quadruples; assigning weights to the experimental categories, experimental equipment, and data access frequency included in the quadruples to prioritize them; and migrating the quadruples according to the priority ranking. This invention ensures the quality of the migrated data and guarantees the data correlation after standardization, as well as the efficiency of the data migration.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, specifically to a method, system, equipment, and medium for migrating historical data from an oil and chemical laboratory. Background Technology

[0002] The petrochemical laboratory is a core facility for conducting oil product analysis, equipment performance testing, and process optimization research in the petrochemical field. During experiments, various types of data are generated, including reaction temperature, pressure sensor readings, equipment vibration spectra, spectral images, experimental videos, and text reports. Furthermore, experiments in the petrochemical laboratory are characterized by their long processing times and the generation of massive amounts of data. Without proper data migration, this data can become chaotic, complex, and difficult to trace and manage.

[0003] In existing technologies, data migration is often involved in the finance, manufacturing, and medical industries. However, due to the multimodal and large-scale nature of data in oil and chemical laboratories, conventional data migration methods cannot meet the requirements of migration efficiency and post-migration data quality in oil and chemical laboratories. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is: how to address the issue that existing methods cannot meet the requirements of migration efficiency and post-migration data quality in oil and chemical laboratories due to the multimodal and large-scale nature of the data.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a data migration method for historical data from an oil and chemical laboratory, comprising: classifying historical data from the oil and chemical laboratory into two categories—unstructured data and structured data—using a classification model; performing corresponding standardization processing steps on the unstructured data and structured data respectively to obtain standardized data; labeling the standardized data with experimental categories, experimental equipment, and timestamps, and associating the unstructured data with the structured data according to the correspondence of historical data from the oil and chemical laboratory, generating a quadruple in the format of unstructured data-structured data-experiment category-experimental equipment; assigning weights to the experimental categories, experimental equipment, and data access frequency contained in the quadruple to prioritize the quadruple; and migrating the quadruple according to the priority ranking.

[0007] As a preferred embodiment of the data migration method for historical data from an oil and chemical laboratory according to the present invention, the historical data includes oil and chemical experiment images, oil and chemical experiment report texts, and time-frequency data, which are multimodal data; the classification model is a modal recognition model trained based on historical data; the classification model is a residual network 50-bidirectional long short-term memory network hybrid neural network model; the residual network 50 is used to extract the spectral features of the oil and chemical experiment images, and the petrochemical equipment model code is embedded as a location code to generate a spectral feature vector; the oil and chemical experiment report text is modeled using a bidirectional long short-term memory network, and a semantic vector is constructed by embedding an oil and chemical domain dictionary; the energy spectrum of the time-frequency data is extracted through wavelet packet transform to obtain a time-frequency feature vector; the contribution weights of the spectral feature vector, semantic vector, and time-frequency feature vector are balanced through a self-attention mechanism to output the classification probability of structured and unstructured data.

[0008] This preferred approach utilizes ResNet50 to extract image spectral features, BiLSTM to model text semantics, and wavelet packet transform to extract time-frequency features. Then, it fuses multimodal features through a self-attention mechanism. This not only improves the accuracy of classifying unstructured and structured data but also ensures a balance of contributions between different modalities. As a result, it effectively solves the problem of complex historical laboratory data with large modal differences, and significantly improves the accuracy of data classification and recognition.

[0009] As a preferred embodiment of the data migration method for historical data of an oil and chemical laboratory described in this invention, the standardization process includes: selecting a corresponding standard dimension mapping table according to the experimental category and experimental equipment to unify the dimensions and fields in the structured data; extracting key features from the unstructured data; and performing corresponding standardization processing according to the format category of the key features, which includes images, charts, text, and videos.

[0010] As a preferred embodiment of the data migration method for historical data from an oil and chemical laboratory according to the present invention, the step of associating unstructured data with structured data includes converting both the unstructured and structured data into feature vectors of the same dimension, represented as follows:

[0011] ,

[0012] ,

[0013] in, For feature vectors of unstructured data, For unstructured data encoders, For the first Unstructured data, For the feature vectors of structured data, For structured data encoders, For the first Structured data, For indexing unstructured data, This serves as an index for structured data; cosine similarity is used to measure the correlation between the two, expressed as:

[0014] ,

[0015] in, The correlation between feature vectors of unstructured data and feature vectors of structured data. For vectors The model, For vectors The model; for each unstructured data The most relevant structured data is represented as follows:

[0016] ,

[0017] in, For unstructured data Match the most relevant structured data. For structured data collections, The value corresponding to the maximum correlation between the feature vectors of unstructured data and the feature vectors of structured data. .

[0018] This preferred solution encodes unstructured and structured data into feature vectors respectively, and uses cosine similarity to measure the correlation between them. Then, it automatically matches the most relevant structured data for each unstructured data. This not only realizes the automated association of cross-modal data, but also avoids the inefficiency and error-proneness of manual comparison and matching, thereby improving the automation level of data fusion and the accuracy of matching.

[0019] As a preferred embodiment of the data migration method for historical data of an oil and chemical laboratory according to the present invention, the generation format is a quadruple of unstructured data-structured data-experiment category-experiment equipment, including determining the corresponding experiment category based on the associated data pairs, and the feature prototype of each experiment category is represented as follows:

[0020] ,

[0021] in, As a characteristic prototype of the experimental category, For experimental category The number of data pairs included. Let the data pair be the experiment category; calculate the similarity between the fusion feature of the data pair and the feature prototype of the experiment category, and take the highest similarity as the category to which it belongs, as shown below:

[0022] ,

[0023] ,

[0024] in, For the fusion characteristics of data pairs, The total number of experiment categories. According to the category, The corresponding value when the correlation between the fused features of the data pair and the feature prototypes of the experimental category reaches its maximum value. Based on the source and feature attributes of the structured data, corresponding experimental equipment is matched. Each experimental equipment corresponds to a distribution range of specific structured features, represented by a set:

[0025] ,

[0026] in, The set of distribution ranges corresponding to specific structured features for each experimental device. For the feature range of the device in the first dimension, For the feature range of the device in the second dimension, For the equipment in the Dimensional feature range For dimension; if structured data eigenvectors Falling into the equipment Within the feature range, the matched experimental equipment is represented as:

[0027] ,

[0028] in, For structured data Matching experimental equipment, For all experimental equipment; the generated quadruple representation in the format of unstructured data-structured data-experiment category-experimental equipment is as follows:

[0029] ,

[0030] in, It is a quadruple. It is a tuple.

[0031] This preferred solution identifies experiment categories based on the similarity between data pairs and the prototype features of the experiment category. It also generates a quadruple of unstructured data-structured data-experiment category-experimental equipment by matching structured feature vectors within the range of equipment feature distribution. This enables automated labeling and attribution of experiment categories and experimental equipment, significantly improving the integrity and accuracy of experimental data management and laying the foundation for data organization for subsequent migration and utilization.

[0032] As a preferred embodiment of the data migration method for historical data in an oil and chemical laboratory according to the present invention, the following steps are taken: Weights are assigned to the experimental categories, experimental equipment, and data access frequency of the four-tuple to prioritize the four-tuples. This includes: determining the weight of the experimental categories included in the four-tuple based on the frequency of occurrence of the experimental categories and the number of associated equipment; the experimental categories include oil product experiments, oil and gas experiments, and equipment testing; determining the weight of the experimental equipment included in the four-tuple based on the attenuation coefficient and usage frequency of the experimental equipment; determining the weight of the data access frequency based on the historical access volume of the data included in the four-tuple; and obtaining the priority parameters of the four-tuple based on the weights of the experimental categories, the experimental equipment, and the access frequency, expressed as follows:

[0033] ,

[0034] in, For the priority parameter of the quadruple, The business coefficients for the experimental categories included in the quadruple. The weights of the experiment categories included in the quadruple. The business coefficients of the experimental equipment included in the quaternion. The weights of the experimental equipment included in the quadruple. The business coefficient representing the frequency of data access. Weights are assigned based on the frequency of data access.

[0035] This preferred solution assigns weights to the experiment category, experiment equipment, and data access frequency in the quadruple and calculates priority parameters. This allows the migration process to prioritize the critical data that is most important to the experimental business and accessed most frequently, thereby improving the utilization efficiency of storage and migration resources, ensuring the priority of high-value data during migration, and meeting the actual needs of laboratory data management.

[0036] As a preferred embodiment of the data migration method for historical data of an oil and chemical laboratory described in this invention, the migration of quadruples according to priority sorting includes: first migrating high-priority quadruples to high-performance storage nodes, then migrating low-priority quadruples to low-performance storage nodes, and simultaneously slicing quadruples with large memory usage by timestamps before migration.

[0037] This invention provides a data migration system for historical data from an oil and chemical laboratory.

[0038] To address the aforementioned technical problems, this invention provides the following technical solution: a data migration system for historical data from an oil and chemical laboratory, comprising: a classification module, a standardization processing module, a quadruple generation module, a sorting module, and a migration module; the classification module is used to classify historical data from the oil and chemical laboratory into two categories: unstructured data and structured data, using a classification model; the standardization processing module is used to perform corresponding standardization processing steps on the unstructured data and structured data respectively to obtain standardized data; the quadruple generation module is used to label the standardized data with experimental categories, experimental equipment, and timestamps, and, based on the correspondence of historical data from the oil and chemical laboratory, associate the unstructured data with the structured data to generate quadruples in the format of unstructured data-structured data-experiment category-experimental equipment; the sorting module is used to assign weights to the experimental categories, experimental equipment, and data access frequency contained in the quadruples to prioritize the quadruples; the migration module is used to migrate the quadruples according to the priority ranking.

[0039] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of a data migration method for historical data of an oil chemical laboratory.

[0040] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of a data migration method for historical data of an oil chemical laboratory.

[0041] The beneficial effects of this invention are as follows: By classifying data into unstructured data and structured data and standardizing the data, this invention ensures the quality of the data after migration, and also guarantees the data correlation and the efficiency of data migration after standardization. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 The present invention provides an overall flowchart of a data migration method for historical data from an oil and chemical laboratory, as an embodiment of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0045] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a data migration method for historical data from an oil and chemical laboratory, comprising:

[0046] S1. The historical data of the oil and chemical laboratory is divided into two categories: unstructured data and structured data using a classification model.

[0047] S2. Perform the corresponding standardization processing steps on unstructured data and structured data respectively to obtain standardized data.

[0048] S3. Label the standardized data with experimental categories, experimental equipment, and timestamps. Based on the correspondence of historical data from the oil and chemical laboratory, associate the unstructured data with the structured data to generate a quadruple in the format of unstructured data-structured data-experiment category-experimental equipment.

[0049] S4. Assign weights to the experimental categories, experimental equipment, and data access frequency included in the quadruplets to prioritize the quadruplets.

[0050] S5. Transfer the quadruplets according to their priority.

[0051] It should be noted that the oil and chemical laboratory has accumulated a large amount of historical experimental data during its long-term operation. This data includes both unstructured data such as experimental images and experimental report texts, and structured data such as experimental parameter tables. Due to differences in experimental equipment models, experimental conditions, and data formats, the direct use of historical data presents problems such as inconsistent formats, difficulties in efficient management and retrieval. Furthermore, the access frequency and business value of some experimental data vary significantly. Migrating all data using the same method would not only consume migration and storage resources but also reduce the utilization efficiency of high-value data.

[0052] Example 2, an embodiment of the present invention, provides a data migration method for historical data from an oil and chemical laboratory based on the previous embodiment, comprising:

[0053] Furthermore, in step S1, the historical data of the oil and chemical laboratory is divided into two categories, unstructured data and structured data, using a classification model, including the following steps:

[0054] Common experiments in oil and chemical laboratories include oil viscosity testing, lubricating oil spectral analysis, distillation column efficiency analysis, and catalyst activity evaluation. These experiments involve equipment such as high-temperature and high-pressure reactors, spectrometers, vibration sensor arrays, and gas chromatographs, ultimately producing data such as reaction temperature time-series data, spectral absorption peak images, equipment vibration spectra, experimental video recordings, and experimental report texts. Taking the lubricating oil oxidation stability experiment as an example, completing a full accelerated oxidation experiment requires approximately 72 hours of continuous operation, utilizing a high-pressure reactor equipped with a platinum catalytic mesh, an online infrared spectral monitoring unit, and a temperature-pressure sensor array sampling every 10 seconds. The final output includes 2000 sets of spectral data, 100,000 temperature and pressure records, 15GB of high-definition reaction process video, and a 40-page PDF experimental report. Therefore, a long-running oil and chemical laboratory produces a massive amount of multimodal data, and these data are strongly correlated. During data migration, it is necessary to standardize this data first to facilitate unified management and migration.

[0055] Historical data is multimodal data, and the classification model is a modality recognition model trained on historical data.

[0056] The classification model is a hybrid neural network model of residual network 50-bidirectional long short-term memory network.

[0057] The spectral features of the oil-chemical experiment images were extracted using a Residual Network 50 (ResNet-50) and the petrochemical equipment model code was embedded as a location code to generate a spectral feature vector. Specifically, when extracting the spectral features of the oil-chemical experiment images using the ResNet-50 network, the oil-chemical experiment images were input into the ResNet-50 network to extract spectral absorption peak features. According to the equipment model code table, the spectral calibration parameters corresponding to the reactor model were converted into a 128-dimensional vector and then fused with the spectral feature vector channel by channel to obtain the spectral feature vector.

[0058] The text of the oil and chemical experiment report is modeled using a bidirectional long short-term memory (BiLSTM) network, and a semantic vector is constructed by embedding an oil and chemical domain dictionary. Specifically, when modeling the oil and chemical experiment report text using BiLSTM sequence modeling, an oil and chemical domain dictionary is constructed based on the knowledge graph of the oil and chemical laboratory. For non-standard terms appearing in the experiment report, they are mapped to standard terms in the oil and chemical domain dictionary through dictionary matching to obtain semantic vectors.

[0059] The energy spectrum of time-frequency data is extracted using wavelet packet transform (WPT) to obtain the time-frequency feature vector. Specifically, when extracting the energy spectrum of time-frequency data using wavelet packet transform, the decomposition level of the wavelet basis function is adjusted according to the aging coefficient of the experimental equipment. The energy spectrum is then high-pass filtered and high-frequency vibration feature components are retained to obtain the time-frequency feature vector.

[0060] By balancing the contribution weights of spectral feature vectors, semantic vectors, and time-frequency feature vectors through a self-attention mechanism, the classification probabilities of structured and unstructured data are output.

[0061] Since the original dimensions of spectral, time-frequency, and semantic features may differ, they need to be mapped to a unified dimension through a linear layer. Dimension, construct the self-attention input feature matrix .in, The feature matrix is ​​used as the input for self-attention. For the set of real numbers, for A matrix of order real numbers, Using query-key similarity as the dimension, self-attention calculates the correlation between features, then normalizes the results using a normalized exponential function (Soft Maximum, Softmax) to obtain the weights. The diagonal elements of the weight matrix are taken as the final contribution weights of each feature, resulting in spectral weights. Time-frequency weighting semantic weight ,satisfy Diagonal elements Indicates the first The weight of each feature in its fusion process.

[0062] Based on self-attention weights, the aligned feature vectors are weighted and summed to obtain the fused global features, represented as:

[0063] ,

[0064] in, For the fused global features, For spectral weights, For time-frequency weighting, For semantic weights, For spectral eigenvectors, For time-frequency feature vectors, It is a semantic vector.

[0065] The fused features are input into a linear classifier, and the probability distribution of each category is output through Softmax:

[0066] ,

[0067] ,

[0068] in, This represents the probability distribution for each category of the output. For the sample to belong to the first The probability of a class satisfies , The output count is the total number of categories, and e is the index of the output category. The linear layer weight matrix of the classifier. , for A matrix of order real numbers; For classifier bias terms, , for A vector of the set of real numbers of dimension 1.

[0069] In the specific implementation process, it is necessary to first construct a ResNet-50 hybrid neural network model for oil and chemical laboratories. After acquiring spectral image data of the inner wall of the reactor, an improved ResNet-50 network architecture is used for spectral feature extraction. An equipment model encoding embedding layer is inserted after the third residual block of the network. The reactor equipment nameplate information, such as material model H316L and volume parameter 2000L, is converted into a 128-dimensional feature vector through a hash algorithm and then concatenated with the spectral feature map. To solve the spectral shift problem caused by high temperature environment, a temperature compensation mechanism is introduced into the feature fusion layer. When the real-time monitored temperature exceeds 180℃, the spectral correction module based on multinomial regression is activated to dynamically adjust the feature channel weights. For the text processing of experimental reports, a BiLSTM was used to load a knowledge graph of refining processes, constructing a domain dictionary containing 3278 petrochemical terms. At the word embedding layer, non-standard expressions such as distillation tower pressure fluctuations were mapped to standard terms from the Atmospheric Distillation Test Method for Petroleum Products (ASTM D86). Dependency parsing was introduced to constrain entity relationships, ensuring that catalyst deactivation must be associated with a reaction temperature >230℃. In the time-frequency feature extraction stage, wavelet packet transform was performed on the raw signals collected by the vibration sensors. When the equipment's cumulative operating time exceeded 5000 hours, it automatically switched to the Daubechies wavelet basis for 8-level decomposition, extracting energy spectrum features in the 32-256Hz high-frequency band.

[0070] Furthermore, in step S2, corresponding standardization processing steps are performed on the unstructured data and the structured data respectively to obtain standardized data, including the following steps:

[0071] Extract key features from unstructured data and perform corresponding standardization processing based on the format categories of the key features. The format categories include images, charts, text, and videos. The standardization processing includes image noise reduction, keyframe extraction, semantic recognition based on an oil-based dictionary, entity extraction, semantic annotation, and mapping non-standard terms to one or more standard fields.

[0072] When performing the corresponding standardization processing steps on unstructured and structured data respectively, the structured data is cleaned for format, noise is removed, and missing values ​​are filled. The corresponding standard dimension mapping table is selected according to the experimental category and experimental equipment to unify the dimensions and fields in the structured data.

[0073] Specifically, methods for unifying the dimensions and fields in structured data by selecting the corresponding standard dimension mapping table based on the experimental category and equipment include:

[0074] Select the standard dimensions of the formulas under ASTM, DIN, or GB standards according to the experimental category and experimental equipment. Unify the dimensions of the structured data to the standard dimensions according to the preset standard dimension mapping table. Perform similarity matching on the fields of the structured data according to the constructed standard field dictionary, and unify the fields of the structured data to the standard fields.

[0075] For example, an oil chemical laboratory conducts oxidation stability tests on lubricating oils using a German-made high-temperature, high-pressure reactor and a domestically produced gas chromatograph. The structured data generated includes reaction pressure, temperature curves, and chromatographic peak parameters. The pressure and temperature curves need to conform to the dimensional mapping table under DIN 51425, converting bars to MPa. The chromatographic peak parameters need to conform to the dimensional mapping table under GB / T 28784. Simultaneously, based on the standard field dictionary, the pressure parameter *Druck_peak* recorded by the German reactor is mapped to the standard field *MaxPressure_DIN*, and the Chinese field *pressure peak* is mapped to *MaxPressure_GB*. This processing allows experimental data from equipment conforming to ASTM, DIN, or GB standards to be converted into a unified standardized format, significantly improving data standardization.

[0076] Furthermore, in step S3, the standardized data is labeled with experimental category, experimental equipment, and timestamp. Based on the correspondence of historical data from the oil and chemical laboratory, the unstructured data is associated with the structured data to generate a quadruple in the format of unstructured data-structured data-experiment category-experimental equipment. This includes the following steps:

[0077] Unstructured data is associated with structured data to generate a four-tuple of unstructured data-structured data-experiment category-experiment equipment, achieved through three steps: feature association, category mapping, and equipment matching. Based on the correspondence of historical data from the oil and chemical laboratory, images, videos, reports, and experimental data for the same experiment are mapped to corresponding experimental data.

[0078] By calculating feature similarity, a mapping relationship between unstructured and structured data is established. First, both unstructured and structured data are converted into feature vectors of the same dimension, represented as follows:

[0079] ,

[0080] ,

[0081] in, The feature vector of unstructured data; For unstructured data encoders, text is represented by Bidirectional Encoder Representations from Transformers (BERT), and images are represented by Convolutional Neural Networks (CNN). For the first Unstructured data; The feature vector of structured data; For structured data encoders, such as normalization and dimensionality reduction using Principal Component Analysis (PCA); For the first Structured data; Indexes for unstructured data; For indexing structured data.

[0082] Cosine similarity is used to measure the correlation between two things, and it is expressed as:

[0083] ,

[0084] in, The correlation between feature vectors of unstructured data and feature vectors of structured data. For vectors The model, For vectors The model.

[0085] For each unstructured data The most relevant structured data is represented as follows:

[0086] ,

[0087] in, For unstructured data Match the most relevant structured data. For structured data collections, The value corresponding to the maximum correlation between the feature vectors of unstructured data and the feature vectors of structured data. .

[0088] Based on the associated (unstructured-structured) data pairs, the corresponding experiment categories are determined, and the feature prototype of each experiment category is represented as follows:

[0089] ,

[0090] in, As a characteristic prototype of the experimental category, For experimental category The number of data pairs included. This is an experimental category.

[0091] Calculate the similarity between the fusion features of the data pairs and the feature prototypes of the experimental categories, and take the highest similarity as the category to which the data pair belongs, as follows:

[0092] ,

[0093] ,

[0094] in, For the fusion characteristics of data pairs, The total number of experiment categories. According to the category, The corresponding value when the correlation between the fused features of the data pair and the feature prototypes of the experimental category reaches its maximum value. .

[0095] Based on the source and feature attributes of the structured data, corresponding experimental equipment is matched. Each experimental equipment corresponds to a distribution range of specific structured features, represented by a set:

[0096] ,

[0097] in, The set of distribution ranges corresponding to specific structured features for each experimental device. For the feature range of the device in the first dimension, For the feature range of the device in the second dimension, For the equipment in the The feature range of the dimension.

[0098] If structured data eigenvectors Falling into the equipment Within the feature range, the matched experimental equipment is represented as:

[0099] ,

[0100] in, For structured data Matching experimental equipment, For all experimental equipment.

[0101] Integrating the above results, the generated quadruple representation in the format of unstructured data-structured data-experiment category-experimental equipment is as follows:

[0102] ,

[0103] in, It is a quadruple. It is a tuple.

[0104] Furthermore, in step S4, weights are assigned to the experimental categories, experimental devices, and data access frequencies included in the quadruplets to prioritize them, including the following steps:

[0105] Methods for assigning weights to experiment categories, experimental equipment, and the frequency of access to recorded data include:

[0106] The weights of the experimental categories included in the quadruple are determined based on the frequency of occurrence of the experimental categories and the number of associated devices. The experimental categories include oil product experiments, oil and gas experiments, and equipment tests. The weights of the experimental devices included in the quadruple are determined based on the attenuation coefficient and usage frequency of the experimental devices. The weights of the data access frequency are determined based on the historical access volume of the data included in the quadruple.

[0107] Based on the weights of the experiment category, the experiment equipment, and the access frequency, the priority parameters of the quadruple are obtained, expressed as follows:

[0108] ,

[0109] in, For the priority parameter of the quadruple, The business coefficients for the experimental categories included in the quadruple. The weights of the experiment categories included in the quadruple. The business coefficients of the experimental equipment included in the quaternion. The weights of the experimental equipment included in the quadruple. The business coefficient representing the frequency of data access. This is a weighting factor for data access frequency. In this invention, an experimental category coefficient is set. Test equipment coefficient Access frequency coefficient Priority will be given to ensuring the execution of core experimental categories.

[0110] Specifically, weights are assigned to experiment categories, experimental equipment, and the frequency of access to recorded data, and a priority calculation model based on fuzzy comprehensive evaluation is developed. When calculating the weights for experiment categories, an equipment relevance factor is introduced. This results in a weighted bonus for oil explosion tests involving multiple devices working together, expressed as:

[0111] ,

[0112] in, For the number of associated devices, This is the equipment correlation factor.

[0113] The equipment weight calculation uses a hyperbolic decay function:

[0114] ,

[0115] in, It is a hyperbolic decay function. The cumulative operating hours of the equipment are used to ensure that experimental data from older equipment receives higher priority. Data access frequency weighting uses a sliding window statistical method, calculating the access popularity index over a 72-hour period. The final priority parameter P = 0.5 × experimental category weight + 0.3 × experimental equipment weight + 0.2 × access frequency weight. When the P value exceeds the threshold of 0.8, a real-time migration command is triggered.

[0116] Furthermore, in step S5, the quadruplets are migrated according to priority order, including the following steps:

[0117] First, migrate high-priority quadruples to high-performance storage nodes, then migrate low-priority quadruples to low-performance storage nodes. At the same time, quadruples with large memory usage are sliced ​​according to timestamps before migration.

[0118] The high-performance storage nodes are all-flash array clusters deployed in the core server room of the oil and chemical laboratory, ensuring a migration bandwidth of 40Gbps. When migrating high-priority quadruplets to the high-performance storage nodes, Secure Sockets Layer / Transport Layer Security (SSL / TLS) is used for encrypted transmission, and ShangMi 4 (SM4) algorithm is used to encrypt sensitive experimental data. During encryption, the initialization vector is derived from the device-encoded hash value, and the key management module is integrated into the laboratory's Hardware Security Module (HSM).

[0119] The low-performance storage nodes are object storage systems integrated with the Glacier Cold Storage Strategy (Glacier), deployed in an off-site backup center. The Glacier cold storage strategy uses the Zstandard compression algorithm for high compression ratio processing. As can be seen from the monitoring interface, after tiered storage optimization, 31TB of high-performance storage and 79TB of cold storage are used for 125TB of historical data, saving data storage costs.

[0120] Methods for slicing quaternions with large memory usage by timestamps include slicing video data by keyframes. For example, a 2-hour catalytic cracking reaction video can be divided into 348 segments.

[0121] Wavelet transform was used to slice the spectral image. For giant quadruples exceeding 10GB, an adaptive compression algorithm based on principal component analysis was used to retain the features of the first 20 principal components. Combined with wavelet time-frequency slicing technology, the amount of data in a single migration was compressed to 12% of the original size.

[0122] The text data is sliced ​​according to the keywords of the text segments.

[0123] In addition, the data migration method also includes generating hash values ​​based on the structured data, timestamps, slice pointers and device codes in the quadruple; and writing the hash values ​​of the quadruple data into Hyperledger Fabric chaincode.

[0124] Example 3 is an embodiment of the present invention. This embodiment provides a data migration system for historical data of an oil and chemical laboratory, including a classification module, a standardization processing module, a quadruple generation module, a sorting module, and a migration module.

[0125] The classification module is used to divide the historical data of the oil and chemical laboratory into two categories: unstructured data and structured data, using a classification model.

[0126] The standardization processing module is used to perform corresponding standardization processing steps on unstructured data and structured data respectively to obtain standardized data.

[0127] The quadruple generation module is used to label standardized data with experimental categories, experimental equipment, and timestamps. Based on the correspondence of historical data from the oil and chemical laboratory, it associates unstructured data with structured data to generate quadruples in the format of unstructured data-structured data-experiment category-experimental equipment.

[0128] The sorting module is used to assign weights to the experimental categories, experimental devices, and data access frequencies contained in the quadruplets in order to prioritize the quadruplets.

[0129] The migration module is used to migrate quadruples according to priority.

[0130] This embodiment also provides an electronic device applicable to a data migration method for historical data of an oil and chemical laboratory, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the data migration method for historical data of an oil and chemical laboratory as proposed in the above embodiment.

[0131] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements a data migration method for historical data of an oil chemical laboratory as proposed in the above embodiment.

[0132] The storage medium proposed in this embodiment and the data migration method for historical data of an oil chemical laboratory proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0133] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for data migration of historical data from an oil and chemical laboratory, characterized in that: include, The historical data of the oil and chemical laboratory was divided into two categories, unstructured data and structured data, using a classification model. Perform corresponding standardization processing steps on unstructured data and structured data respectively to obtain standardized data; Standardized data is labeled with experimental categories, equipment, and timestamps. Based on the historical data correspondence of the oil and chemical laboratory, unstructured and structured data are linked to generate a quadruple in the format of unstructured data-structured data-experiment category-experiment equipment, including... Convert unstructured and structured data into feature vectors of the same dimension, as follows: in, For feature vectors of unstructured data, For unstructured data encoders, For the first Unstructured data, For the feature vectors of structured data, For structured data encoders, For the first Structured data, For indexing unstructured data, For indexing structured data; Cosine similarity is used to measure the correlation between two things, and it is expressed as: in, The correlation between feature vectors of unstructured data and feature vectors of structured data. For vectors The model, For vectors The model; For each unstructured data Matching the most relevant structured data, represented as follows: in, For unstructured data Match the most relevant structured data. For structured data collections, The value corresponding to the maximum correlation between the feature vectors of unstructured data and the feature vectors of structured data. ; Based on the correlated data pairs, the corresponding experiment categories are determined, and the feature prototype of each experiment category is represented as follows: in, As a characteristic prototype of the experimental category, For experimental category The number of data pairs included. For experimental categories; Calculate the similarity between the fusion features of the data pairs and the feature prototypes of the experimental categories, and take the highest similarity as the category to which the data pair belongs, denoted as . in, For the fusion characteristics of data pairs, The total number of experiment categories. According to the category, The corresponding value when the correlation between the fused features of the data pair and the feature prototypes of the experimental category reaches its maximum value. ; Based on the source and feature attributes of the structured data, corresponding experimental equipment is matched. Each experimental equipment corresponds to a distribution range of specific structured features, represented by a set. in, The set of distribution ranges corresponding to specific structured features for each experimental device. For the feature range of the device in the first dimension, For the feature range of the device in the second dimension, For the equipment in the Dimensional feature range For dimensions; If structured data eigenvectors Falling into the equipment Within the feature range, the matched experimental equipment is represented as follows: in, For structured data Matching experimental equipment, For all experimental equipment; The generated quadruple format is unstructured data-structured data-experiment category-experiment equipment. in, It is a quadruple. It is a tuple; Weights are assigned to the experiment categories, experimental equipment, and data access frequency of the quadruples to prioritize them. The quadruplets are migrated based on priority order.

2. The data migration method for historical data of an oil and chemical laboratory as described in claim 1, characterized in that: The historical data includes oiling experiment images, oiling experiment report text, and time and frequency data, which are multimodal data. The classification model is a modality recognition model trained based on historical data; The classification model is a hybrid neural network model of residual network 50 and bidirectional long short-term memory network; The residual network 50 is used to extract the spectral features of the oil chemical experiment images and embed the petrochemical equipment model code as the location code to generate the spectral feature vector. The text of the oil and chemical experiment report is modeled using a bidirectional long short-term memory network, and a semantic vector is constructed by embedding an oil and chemical domain dictionary. The energy spectrum of the time-frequency data is extracted by wavelet packet transform to obtain the time-frequency feature vector; By balancing the contribution weights of spectral feature vectors, semantic vectors, and time-frequency feature vectors through a self-attention mechanism, the classification probabilities of structured and unstructured data are output.

3. The data migration method for historical data of an oil and chemical laboratory as described in claim 2, characterized in that: The standardization process includes selecting the corresponding standard dimension mapping table according to the experiment category and experimental equipment to unify the dimensions and fields in the structured data; Extract key features from unstructured data and perform standardization processing according to the format category of the key features, including images, charts, text, and videos.

4. The data migration method for historical data of an oil and chemical laboratory as described in claim 3, characterized in that: The process involves assigning weights to the experimental categories, experimental equipment, and data access frequency included in the quadruplets to prioritize them. The weights of the experimental categories included in the quadruple are determined based on the frequency of occurrence of the experimental categories and the number of associated devices. The experimental categories include oil product experiments, oil and gas experiments, and equipment tests. The weights of the experimental equipment included in the quadruple are determined based on the attenuation coefficient and usage frequency of the experimental equipment. The weight of data access frequency is determined based on the historical access volume of the data contained in the quadruple; Based on the weights of the experiment category, the experiment equipment, and the access frequency, the priority parameters of the quadruple are obtained, denoted as follows: in, For the priority parameter of the quadruple, The business coefficients for the experimental categories included in the quadruple. The weights of the experiment categories included in the quadruple. The business coefficients of the experimental equipment included in the quaternion. The weights of the experimental equipment included in the quadruple. The business coefficient representing the frequency of data access. Weights are assigned based on the frequency of data access.

5. The data migration method for historical data of an oil and chemical laboratory as described in claim 4, characterized in that: The process of migrating quadruplets based on priority sorting is described. include, First, migrate high-priority quadruples to high-performance storage nodes, then migrate low-priority quadruples to low-performance storage nodes. At the same time, quadruples with large memory usage are sliced ​​according to timestamps before migration.

6. A data migration system for historical data of an oil and chemical laboratory, employing a data migration method for historical data of an oil and chemical laboratory as described in any one of claims 1 to 5, characterized in that, include: The module includes a classification module, a standardization module, a quadruple generation module, a sorting module, and a transfer module. The classification module is used to classify historical data from the oil and chemical laboratory into two categories: unstructured data and structured data, using a classification model. The standardization processing module is used to perform corresponding standardization processing steps on unstructured data and structured data respectively to obtain standardized data; The quadruple generation module is used to label the standardized data with experimental categories, experimental equipment and timestamps, and associate unstructured data with structured data according to the correspondence of historical data of the oil and chemical laboratory, generating quadruples in the format of unstructured data-structured data-experiment category-experimental equipment. The sorting module is used to assign weights to the experimental categories, experimental equipment, and data access frequency contained in the quadruplets in order to prioritize the quadruplets. The migration module is used to migrate quadruples according to priority order.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of a data migration method for historical data of an oil chemical laboratory according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a data migration method for historical data of an oil chemical laboratory according to any one of claims 1 to 5.

Images