Archive data storage management method and system based on cloud computing

Abstract

The application relates to the technical field of file data storage management, in particular to a file data storage management method and system based on cloud computing, which comprises the following steps: acquiring file data sets generated in different implementation stages of a project, generating a stage sequence identifier, a data type identifier and a source node identifier for each file data record, and obtaining a file identifier data set; sorting the stage sequence identifiers to construct a stage continuous sequence; mapping the file identifier data set and the stage continuous sequence to obtain a stage-type distribution matrix; calculating the file data structure completeness according to the stage-type distribution matrix; judging the structure completeness to obtain a project structure state identifier; and selecting different storage strategies according to different project structure state identifiers. The application realizes storage strategy switching driven by file data structure completeness, and improves the orderliness and reliability of file data storage management.

Description

Technical Field

[0001] This invention relates to the field of archival data storage and management technology, specifically to an archival data storage and management method and system based on cloud computing. Background Technology

[0002] When archival data is centrally stored and managed through a cloud computing platform, it is typically based on a distributed storage architecture. This involves uploading, classifying, managing in layers, and scheduling copies of the data as files or data blocks to achieve capacity expansion and optimized access. However, some archival data exhibits characteristics of continuous generation from multiple sources, structural dependencies, and gradual formation of complete sets. Its overall structure is not determined in the initial stage but is gradually established as data is continuously supplemented. For example, research project archives are continuously formed during the project initiation, implementation, and acceptance stages, and the entity relationships between data dynamically change as the process evolves.

[0003] Existing technologies lack a dynamic identification mechanism for the formation process of archival structures, making it difficult to determine when archives reach a structurally complete state. Furthermore, they still implement storage and management strategies based on independent data objects, failing to differentiate storage strategies according to the state of structural evolution. This makes it easy to perform decentralized storage or scheduling before the structure is formed, making it difficult to guarantee the structural closure of archives and the traceability of relationships between data objects. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud-based method and system for archival data storage and management, in order to solve the problem that existing technologies cannot implement differentiated control of storage strategies based on the archival structure and formation status.

[0005] To achieve the above objectives, in one aspect, the present invention provides a cloud computing-based archival data storage and management method, the method comprising:

[0006] Step S1: Obtain the set of archival data generated at different implementation stages of the project, and generate a stage sequence identifier, data type identifier, and source node identifier for each archival data record to obtain the archival identifier data set.

[0007] Step S2: Sort the stage sequence identifiers according to the project implementation process to construct a continuous stage sequence; map the data type identifiers in the archive identifier data set to the corresponding positions in the continuous stage sequence to obtain the stage-type distribution matrix.

[0008] Step S3: Determine whether each stage in the continuous stage sequence has a data type identifier based on the stage-type distribution matrix, and calculate the ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous stage sequence to obtain the stage coverage; extract the actual data type set corresponding to each stage based on the stage-type distribution matrix; obtain the expected data type set corresponding to each stage based on the preset project process constraints; calculate the type coverage based on the matching degree between the actual data type set and the expected data type set; extract the data type sets corresponding to any two stages in the continuous stage sequence, and count the number of reference relationships between any two stages based on the data reference relationship identifier; calculate the inter-stage connection consistency value based on the matching degree between the number of reference relationships and the preset number of reference relationships; calculate the deviation of stage coverage, type coverage, and inter-stage connection consistency value based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints to obtain the structural constraint deviation value; combine the stage coverage, type coverage, inter-stage connection consistency value, and structural constraint deviation value to obtain the archive data structure integrity; determine the structure integrity to obtain the project structure status identifier.

[0009] Step S4: When the project structure status is marked as unfixed, write the archive identifier data set into the updatable storage area; when the project structure status is marked as fixed, generate a structure mapping index based on the stage continuous sequence and the stage-type distribution matrix, and write the archive identifier data set into the immutable storage area according to the structure mapping index.

[0010] Preferably, the method for mapping data type identifiers in the archive identifier data set to corresponding positions in the stage continuous sequence to obtain the stage-type distribution matrix includes:

[0011] Based on the type-stage constraint relationship between the data type identifier and the stage sequence identifier, determine the candidate stage set for each archival data record; select the stage position that satisfies the node-stage correspondence from the candidate stage set to obtain the first stage position set; select the stage position whose generated timestamp falls within the corresponding stage time interval from the first stage position set to obtain the second stage position set.

[0012] When the second stage location set contains multiple stage locations, select the stage locations that satisfy the stage dependency relationship from the second stage location set to obtain a candidate stage subset; for the archive data records with multiple stage locations in the candidate stage subset, perform priority comparison processing according to the stage sequence order to obtain a unique stage location identifier; write the data type identifier into the corresponding position of the continuous stage sequence according to the unique stage location identifier to construct a stage-type distribution matrix.

[0013] Preferably, the method for performing priority comparison processing on archive data records with multiple stage positions in the candidate stage subset to obtain a unique stage position identifier includes:

[0014] Obtain the data reference relationship identifier in the archival data record; determine the target data record referenced by the archival data record based on the data reference relationship identifier, and obtain the stage position corresponding to the target data record.

[0015] The stage position that matches the stage position of the target data record in the candidate stage subset is selected as the unique stage position identifier; when there is no stage position that matches the stage position of the target data record in the candidate stage subset, the stage position that is later in the sequence is selected as the unique stage position identifier.

[0016] Preferably, the method for calculating the structural integrity of archival data by combining stage coverage, type coverage, inter-stage consistency value, and structural constraint deviation value includes:

[0017] The stage coverage, type coverage, and inter-stage consistency values ​​are normalized to obtain normalized stage coverage, normalized type coverage, and normalized inter-stage consistency values; the deviation suppression coefficient is calculated based on the structural constraint deviation values. ;in, This represents the structural constraint deviation value. This is the preset deviation suppression parameter.

[0018] The basic completeness is calculated based on the coverage of the normalization stage, the coverage of the normalization type, and the consistency value of the connection between normalization stages. ;in, Based on completeness, For the coverage of the normalization phase, To normalize type coverage, To ensure consistency between normalization stages, , and These are the weighting coefficients, and The structural integrity of the archival data is calculated based on the deviation suppression coefficient and the basic integrity. .

[0019] Preferably, the weighting coefficient , and The methods for determining this include:

[0020] Obtain the project type identifier of the current project; query the corresponding baseline weight coefficient combination from the preset weight configuration library based on the project type identifier; count the number of archive data records corresponding to each stage based on the stage-type distribution matrix, and calculate the time distribution dispersion based on the distribution of the number of archive data records corresponding to each stage in the stage continuous sequence.

[0021] Based on the correspondence between the time distribution dispersion and the preset dispersion threshold range, the combination of weight adjustment coefficients is determined from the preset weight adjustment mapping table; the combination of baseline weight coefficients and the combination of weight adjustment coefficients are multiplied item by item, and the product results are normalized to obtain the dynamically adjusted weight coefficients. , and .

[0022] Preferably, the method for determining the structural integrity and obtaining the project structural status identifier includes:

[0023] Within a preset judgment period, obtain the current archive data structure integrity and obtain the project structure status identifier generated in the previous judgment period.

[0024] When the project structure status identifier generated in the previous judgment period is in an unfixed state and the current archive data structure integrity is not less than the first integrity threshold, the project structure status identifier will be determined to be in a fixed state.

[0025] When the project structure status identifier generated in the previous judgment period is in a solidified state and the current archive data structure integrity is not greater than the second integrity threshold, the project structure status identifier will be determined to be in an unsolidified state.

[0026] When the current archive data structure integrity is between the first integrity threshold and the second integrity threshold, the project structure status identifier is maintained as the project structure status identifier generated in the previous judgment period; wherein, the first integrity threshold is greater than the second integrity threshold.

[0027] Based on the same inventive concept, the present invention also provides a cloud-based archival data storage and management system, the system comprising:

[0028] The document identification generation module is used to obtain the document data set generated at different implementation stages of the project, and generate a stage sequence identifier, data type identifier, and source node identifier for each document data record, thus obtaining a document identification data set.

[0029] The phase mapping modeling module is used to sort the phase sequence identifiers according to the project implementation process and construct a continuous phase sequence; it maps the data type identifiers in the archive identifier data set to the corresponding positions in the continuous phase sequence to obtain the phase-type distribution matrix.

[0030] The structural integrity calculation module is used to determine whether each stage in a continuous sequence of stages has a data type identifier based on the stage-type distribution matrix, calculate the ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous sequence, and obtain the stage coverage. It then extracts the actual data type set corresponding to each stage based on the stage-type distribution matrix; obtains the expected data type set corresponding to each stage based on preset project process constraints; calculates the type coverage based on the matching degree between the actual data type set and the expected data type set; extracts the data type sets corresponding to any two stages in the continuous sequence of stages, and counts the number of reference relationships between any two stages based on data reference relationship identifiers; calculates the inter-stage connection consistency value based on the matching degree between the number of reference relationships and the preset number of reference relationships; calculates the deviation of stage coverage, type coverage, and inter-stage connection consistency value based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints, and obtains the structural constraint deviation value; combines stage coverage, type coverage, inter-stage connection consistency value, and structural constraint deviation value to obtain the archive data structure integrity; and finally, it determines the structural integrity to obtain the project structure status identifier.

[0031] The hierarchical storage control module is used to write the archive identification data set into the updatable storage area and record the version sequence number when the project structure status is identified as unfixed; when the project structure status is identified as fixed, it generates a structure mapping index based on the stage continuous sequence and stage-type distribution matrix, and writes the archive identification data set into the immutable storage area according to the structure mapping index, while generating multi-node replica data.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] 1. By constructing a continuous sequence of stages and a stage-type distribution matrix, and combining it with preset project process constraints, the system calculates the stage coverage, data type coverage, and data connection relationships between stages to obtain the structural integrity of the archival data. Based on the structural integrity, the system determines the project structure status, thereby achieving structured modeling and dynamic status identification of the entire process of archival data from generation to archiving. This enables the archival data storage method to adaptively switch according to the degree of structural integrity, improving the orderliness and reliability of archival data storage and management.

[0034] 2. By introducing structural constraint deviation values ​​and deviation suppression coefficients to construct a nonlinear integrity calculation model, and combining it with a weight dynamic adjustment mechanism based on time distribution dispersion, the structural integrity can comprehensively reflect the stage distribution balance and the degree of structural constraint satisfaction, enhance the sensitivity of structural integrity to abnormal distribution and key deficiencies, and thus improve the accuracy and stability of project structural status identification. Attached Figure Description

[0035] Figure 1 This is a flowchart of the cloud computing-based archive data storage and management method of the present invention;

[0036] Figure 2 This is a block diagram of the cloud-based archive data storage management system of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1: As Figure 1 As shown, this embodiment provides a cloud computing-based archival data storage and management method, the method including:

[0039] Obtain the set of archival data generated at different implementation stages of the project, and generate a stage sequence identifier, data type identifier, and source node identifier for each archival data record to obtain an archival identifier data set; sort the stage sequence identifiers according to the project implementation process to construct a continuous stage sequence.

[0040] Taking a research project as an example, the project implementation process follows the sequence of "project approval and demonstration → scheme design → key technology breakthroughs → prototype manufacturing → system integration → commissioning and operation → performance evaluation → project acceptance." The stage sequence identifiers use fixed-length numerical codes: P01 corresponds to project approval and demonstration, P02 to scheme design, P03 to key technology breakthroughs, P04 to prototype manufacturing, P05 to system integration, P06 to commissioning and operation, P07 to performance evaluation, and P08 to project acceptance. Data type identifiers use pre-defined enumerated value strings, including "task book," "technical report," "design drawings," "experimental data," "meeting minutes," "test report," "funding documents," "acceptance materials," "process records," and "software code." Source node identifiers use equipment codes or user ID strings. Archive data generated by experimental equipment uses the equipment's unique ID as the source node identifier, while archive data submitted by personnel uses the submitter's research group number combined with their employee ID as the source node identifier.

[0041] The archival data set is accessed through the data interface of the scientific research project management system. Each archival data record carries three basic attribute fields upon generation: the original string of the stage name recorded by the business system, the file type extension or the manually selected type label, and the upload device number. The generation process of the stage sequence identifier is as follows: the original string of the stage name is matched with the standard names of each stage in the preset stage mapping table using a string matching method based on keyword inclusion relationships. If a match is successful, the corresponding stage sequence identifier is retrieved; if a match fails, a confirmation prompt is sent to the archivist, and the mapping relationship is recorded for subsequent reuse after confirmation. The generation process of the data type identifier is as follows: firstly, the file extension is queried from the preset type mapping table (generated from historical archived data statistics). In the mapping table, ".doc" and ".docx" correspond to "technical report", ".dwg" and ".step" correspond to "design drawing", ".xls", ".csv" and ".dat" correspond to "experimental data", ".mp4" and ".wav" correspond to "process record", and ".c", ".py" and ".f90" correspond to "software code". If the extension cannot be determined, the type label selected manually on the submission interface is read. The process of generating the source node identifier is as follows: extract the device number field from the HTTP header of the upload request. If the device number field is empty, use the submitter's login ID and concatenate it with the research group number prefix.

[0042] In a data archiving event, six archive data records were received simultaneously. The first record, originally named "Preliminary Design Review," matched the mapping table to obtain P02, had a file extension of .docx, and mapped to "Technical Report." The uploading device ID was LAB-PC-07, and the source node identifier record was LAB-PC-07. The second record, originally named "Structural Testing," matched the mapping table to obtain P03, had a file extension of .dat, and mapped to "Experimental Data." The uploading device ID was RF-AMP-03, and the source node identifier record was RF-AMP-03. The third record, originally named "Power Supply Testing," matched the mapping table to obtain P05, had a file extension of .csv, and mapped to "Experimental Data." The uploading device ID was PS-CTRL-12, and the source node identifier record was PS-CTRL-12. Article 4: The original stage name is "Budget Adjustment Application". No corresponding standard stage name is found in the mapping table, triggering a confirmation process. After manual confirmation by the archivist, it is associated with P01. The file extension is .pdf. The default mapping for .pdf files is not included in the preset mapping table. The manual type label is "Funding Document". The uploader's research group is ACC, employee number 1024, and the source node identifier record is ACC1024. Article 5: The original stage name is "Acceptance Expert Group Opinions". Matching the mapping table yields P08. The file extension is .pdf. The manual type label is "Acceptance Materials". The uploader's research group is ACC, employee number 1001, and the source node identifier record is ACC1001. Article 6: The original stage name is "Control System Source Code". Matching the mapping table yields P06. The file extension is .c. Mapping yields "Software Code". The uploaded device number is CTRL-DEV-05, and the source node identifier record is CTRL-DEV-05. After each record generates a stage sequence identifier, data type identifier, and source node identifier, it is stored together with the original archive data content into the archive identifier data set. Subsequent archived data is then appended to the archive identifier data set in chronological order.

[0043] When constructing a continuous sequence of phases, a preset project implementation process sequence definition table is read. The phase sequence identifiers in the table are arranged in order of process as (P01, P02, P03, P04, P05, P06, P07, P08), and the set of phase sequence identifiers is sorted accordingly. If there are parallel phases in the project, the parallel phases are assigned the same sequential position in the continuous sequence of phases.

[0044] Based on the type-stage constraint relationship between the data type identifier and the stage sequence identifier, determine the candidate stage set for each archival data record; select the stage position that satisfies the node-stage correspondence from the candidate stage set to obtain the first stage position set; select the stage position whose generated timestamp falls within the corresponding stage time interval from the first stage position set to obtain the second stage position set.

[0045] When the second stage location set contains multiple stage locations, stage locations that satisfy the stage dependency relationship are selected from the second stage location set to obtain a candidate stage subset; for the archive data records in the candidate stage subset that have multiple stage locations, priority comparison processing is performed according to the stage sequence order to obtain a unique stage location identifier; the method for performing priority comparison processing according to the stage sequence order to obtain a unique stage location identifier for archive data records in the candidate stage subset that have multiple stage locations includes:

[0046] Obtain the data reference relationship identifier in the archival data record; determine the target data record referenced by the archival data record based on the data reference relationship identifier, and obtain the stage position corresponding to the target data record.

[0047] From the candidate stage subset, select the stage position that matches the stage position of the target data record as the unique stage position identifier; if there is no stage position in the candidate stage subset that matches the stage position of the target data record, select the stage position that is later in the continuous stage sequence as the unique stage position identifier. Write the data type identifier into the corresponding position in the continuous stage sequence based on the unique stage position identifier to construct a stage-type distribution matrix.

[0048] For example, the type-stage constraint relationship is stored as a two-dimensional lookup table, where the row index is the data type identifier and the column elements are the set of allowed stage sequence identifiers. Technical reports correspond to the set {P01, P02, P03, P04, P05, P06, P07, P08}, experimental data corresponds to the set {P03, P04, P05, P06}, test reports correspond to the set {P04, P06, P07}, funding documents correspond to the set {P01, P08}, and software code corresponds to the set {P03, P05, P06}. The data type identifier of the second archival data record is experimental data, corresponding to the candidate stage set {P03, P04, P05, P06}. The data type identifier of the supplementary archival data record is test report, corresponding to the candidate stage set {P04, P06, P07}.

[0049] The node-stage correspondence is stored as a set of mapping rules. The device number prefix LAB corresponds to the stage sequence identifier set {P01, P02, P03, P04}, RF corresponds to {P03, P04, P05, P06}, PS corresponds to {P04, P05, P06, P07}, CTRL corresponds to {P05, P06, P07, P08}, and the research group number ACC corresponds to {P01, P02, P03, P04, P05, P06, P07, P08}. The source node identifier of the second archive data record is RF-AMP-03, the prefix RF corresponds to the set {P03, P04, P05, P06}, and the first stage location set is {P03, P04, P05, P06}. The source node identifier of the supplementary archive data record is CTRL-DEV-05, and the prefix CTRL corresponds to the set {P05,P06,P07,P08}. The first stage position set {P06,P07} is selected from the type candidate stage set {P04,P06,P07}.

[0050] The time intervals for each stage are generated from the start and end dates in the project task book. The time interval for P03 is from May 1st to August 31st, and the time interval for P06 is from April 1st to June 30th. The generation timestamps are taken from the creation time field in the file system metadata. The second archive data record's generation timestamp is July 20th, falling within the P03 time interval. The second stage location set {P03} is selected from the first stage location set {P03, P04, P05, P06}. The supplementary archive data record's generation timestamp is May 25th, falling within the P06 time interval. The second stage location set {P06, P07} is selected from the first stage location set {P06, P07}.

[0051] Stage dependencies are stored as a directed relation set, where P06 depends on P05, and P07 depends on P05. The archive identifier data set already contains a third and a sixth archive data record. The stage sequence identifier for the third archive data record is P05, and the stage sequence identifier for the sixth archive data record is P06. P05 already satisfies the dependency condition. In the second stage position set {P06, P07}, both P06 and P07 satisfy the dependency relationship, making {P06, P07} the candidate stage subset.

[0052] The supplementary archival data record stores a data reference relationship identifier in its file header metadata field. This identifier is the file hash value of the target archival data record. The data reference relationship identifier is generated by the submitter selecting a reference target from the list of archived data records when the archival data record is submitted, or automatically extracted and associated by the system through parsing the file identifier referenced in the document content and then written into the file header metadata field. The file hash value matches the third archival data record in the archival identifier data set, and the stage sequence identifier of the third archival data record is P05. The candidate stage subset {P06, P07} does not contain the stage sequence identifier P05, and the candidate stage subset remains unchanged. The continuous stage sequence is (P01, P02, P03, P04, P05, P06, P07, P08), with P07 following P06, and the unique stage position identifier is determined to be P07. The second stage position set of the second archival data record is {P03}, and the unique stage position identifier is P03. The unique stage position identifier of the supplementary archival data record is P07.

[0053] A stage-type distribution matrix is ​​constructed using the continuous sequence of stages as the row index set and the data type identifier as the column index set. The element values ​​in the stage-type distribution matrix represent the number of archival data records corresponding to the combination of stage sequence identifier and data type identifier. The second archival data record makes the element in the stage-type distribution matrix with stage sequence identifier P03 and data type identifier "experimental data" valued at 1. Supplementing the archival data record makes the element in the stage-type distribution matrix with stage sequence identifier P07 and data type identifier "test report" valued at 1. Elements not written to the matrix have a value of 0. The element update rule for the stage-type distribution matrix is: when a new archival data record is written with the same combination of stage sequence identifier and data type identifier, the corresponding element value is incremented by 1.

[0054] Based on the stage-type distribution matrix, determine whether each stage in the continuous sequence of stages has a data type identifier. Calculate the ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous sequence to obtain the stage coverage. Extract the actual data type set corresponding to each stage based on the stage-type distribution matrix. Obtain the expected data type set corresponding to each stage based on the preset project process constraints. Calculate the type coverage based on the degree of matching between the actual data type set and the expected data type set.

[0055] Extract the data type sets corresponding to any two stages from the continuous sequence of stages, and count the number of reference relationships between any two stages based on the data reference relationship identifier; calculate the inter-stage connection consistency value based on the degree of matching between the number of reference relationships and the preset number of reference relationships.

[0056] Based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints, deviations are calculated for stage coverage, type coverage, and inter-stage connection consistency values ​​to obtain structural constraint deviation values. The stage coverage, type coverage, and inter-stage connection consistency values ​​are then normalized to obtain normalized stage coverage, normalized type coverage, and normalized inter-stage connection consistency values. Finally, a deviation suppression coefficient is calculated based on the structural constraint deviation values. ;in, This represents the structural constraint deviation value. This is the preset deviation suppression parameter.

[0057] The basic completeness is calculated based on the coverage of the normalization stage, the coverage of the normalization type, and the consistency value of the connection between normalization stages. ;in, Based on completeness, For the coverage of the normalization phase, To normalize type coverage, To ensure consistency between normalization stages, , and These are the weighting coefficients, and The structural integrity of the archival data is calculated based on the deviation suppression coefficient and the basic integrity. The weighting coefficients , and The methods for determining this include:

[0058] Obtain the project type identifier of the current project; query the corresponding baseline weight coefficient combination from the preset weight configuration library based on the project type identifier; count the number of archive data records corresponding to each stage based on the stage-type distribution matrix, and calculate the time distribution dispersion based on the distribution of the number of archive data records corresponding to each stage in the stage continuous sequence.

[0059] Based on the correspondence between the time distribution dispersion and the preset dispersion threshold range, the combination of weight adjustment coefficients is determined from the preset weight adjustment mapping table; the combination of baseline weight coefficients and the combination of weight adjustment coefficients are multiplied item by item, and the product results are normalized to obtain the dynamically adjusted weight coefficients. , and The structural integrity of archival data is calculated based on the deviation suppression coefficient and the basic integrity. .

[0060] The continuous sequence of stages contains eight stage sequence identifiers, P01 to P08. The non-zero elements in the stage-type distribution matrix are as follows: P01 corresponds to funding documents with a value of 1, P02 corresponds to technical reports with a value of 1, P03 corresponds to experimental data with a value of 1, P05 corresponds to experimental data with a value of 1, P06 corresponds to software code with a value of 1, P07 corresponds to test reports with a value of 1, P08 corresponds to acceptance materials with a value of 1, and all other elements have a value of 0.

[0061] In the continuous sequence of stages (P01, P02, P03, P04, P05, P06, P07, P08), stages P01, P02, P03, P05, P06, P07, and P08 have non-zero elements in the stage-type distribution matrix, while stage P04 does not have any non-zero elements. The number of stages with existing data type identifiers is 7, and the total number of stages in the continuous sequence of stages is 8. The stage coverage is 7 / 8 = 0.875.

[0062] In the preset project process constraints, the expected data type set corresponding to P01 is {task book, funding documents}, the expected data type set corresponding to P02 is {technical report, design drawings}, the expected data type set corresponding to P03 is {experimental data, design drawings, technical report, software code}, the expected data type set corresponding to P04 is {design drawings, experimental data, test report}, the expected data type set corresponding to P05 is {experimental data, software code, design drawings}, the expected data type set corresponding to P06 is {experimental data, test report, software code}, the expected data type set corresponding to P07 is {test report}, and the expected data type set corresponding to P08 is {acceptance materials, technical report}.

[0063] The actual data type sets corresponding to each stage in the stage-type distribution matrix are as follows: P01 corresponds to {funding documents}, P02 corresponds to {technical reports}, P03 corresponds to {experimental data}, P04 corresponds to the empty set, P05 corresponds to {experimental data}, P06 corresponds to {software code}, P07 corresponds to {test reports}, and P08 corresponds to {acceptance materials}. The matching degree of each stage is as follows: P01 = 1 / 2 = 0.5, P02 = 1 / 2 = 0.5, P03 = 1 / 4 = 0.25, P04 = 0, P05 = 1 / 3 ≈ 0.333, P06 = 1 / 3 ≈ 0.333, P07 = 1 / 1 = 1.0, P08 = 1 / 2 = 0.5, and the type coverage is approximately (0.5 + 0.5 + 0.25 + 0 + 0.333 + 0.333 + 1.0 + 0.5) / 8 ≈ 0.427.

[0064] Data reference relationships are stored in the archival data record metadata as a reference hash list. The third archival data record references the second archival data record, the sixth archival data record references the third archival data record, and supplementary archival data records reference the third archival data record. In the preset number of reference relationships, P03 references P02 at least once, P06 references P05 at least once, P07 references P05 at least once, and P08 references both P06 and P07 at least once each. The preset number of reference relationships for other combinations is 0. Statistical analysis shows that P05 references P03 once, P06 references P05 once, P07 references P05 once, and the number of reference relationships for other stage combinations is 0.

[0065] The matching degree of each preset reference relationship is as follows: P03 references P02 with a value of 0, P06 references P05 with a value of 1, P07 references P05 with a value of 1, P08 references P06 with a value of 0, and P08 references P07 with a value of 0. The consistency value between stages is (0+1+1+0+0) / 5=0.4. The expected coverage range in the preset project process constraints is 8. The stage coverage deviation is |0.875-1|×(8 / 8)=0.125, the type coverage deviation is |0.427-1|=0.573, and the connection deviation is |0.4-1|=0.6. The weights of the three items are all 1 / 3. The structural constraint deviation value D=(0.125+0.573+0.6) / 3≈0.433.

[0066] The normalization stage coverage A = 0.875, the normalization type coverage B = 0.427, and the consistency value between normalization stages C = 0.4. Preset deviation suppression parameters were calibrated based on the statistical distribution of structural constraint deviation values ​​from historically archived projects. The bias suppression coefficient is 2.0, and the deviation suppression coefficient P = exp(-2.0 × 0.433) ≈ 0.421. The project type is identified as "Major Instrument R&D", and the corresponding benchmark weight coefficient combination in the preset weight configuration library is (0.25, 0.35, 0.40). The stage-type distribution matrix is ​​summed row by row to obtain the sequence of the number of archive data records corresponding to each stage [1, 1, 1, 0, 1, 1, 1, 1], with a mean of 0.875, a standard deviation of 0.3307, a coefficient of variation of 0.378, and a time distribution dispersion of 1 - 0.378 = 0.622.

[0067] In the preset dispersion threshold range, [0, 0.4) corresponds to the low dispersion range, [0.4, 0.7) corresponds to the medium dispersion range, and [0.7, 1.0] corresponds to the high dispersion range. The preset weight adjustment mapping table shows the weight adjustment coefficient combination for the low dispersion range as (1.2, 0.9, 0.9), for the medium dispersion range as (1.0, 1.0, 1.0), and for the high dispersion range as (0.9, 1.1, 1.0). The product of each term is (0.25, 0.35, 0.40), and the normalized weight coefficients are... , , Basic completeness Q = 0.25 × 0.875 + 0.35 × 0.427 + 0.40 × 0.4 ≈ 0.528. Archival data structure completeness S = 0.421 × 0.528 ≈ 0.222.

[0068] Within a preset judgment period, obtain the current archive data structure integrity and obtain the project structure status identifier generated in the previous judgment period.

[0069] When the project structure status identifier generated in the previous judgment period is in an unfixed state and the current archive data structure integrity is not less than the first integrity threshold, the project structure status identifier will be determined to be in a fixed state.

[0070] When the project structure status identifier generated in the previous judgment period is in a solidified state and the current archive data structure integrity is not greater than the second integrity threshold, the project structure status identifier will be determined to be in an unsolidified state.

[0071] When the current archive data structure integrity is between the first integrity threshold and the second integrity threshold, the project structure status identifier is maintained as the project structure status identifier generated in the previous judgment period; wherein, the first integrity threshold is greater than the second integrity threshold.

[0072] When the project structure status is marked as unfixed, the archive identification data set is written to the updatable storage area; when the project structure status is marked as fixed, a structure mapping index is generated based on the phase continuous sequence and the phase-type distribution matrix, and the archive identification data set is written to the immutable storage area according to the structure mapping index.

[0073] The preset judgment cycle is triggered after each archive data archiving event is completed. A maximum silent interval of 24 hours is set; a judgment is triggered if no archiving event occurs within 24 hours. The project structure status identifier generated in the previous judgment cycle is read from the system status table. The initial project structure status identifier is in an unfixed state, and the result of this read is also in an unfixed state.

[0074] The first integrity threshold is 0.80, and the second integrity threshold is 0.65. These thresholds are determined based on the statistical distribution of the integrity of the archival data structure of historically archived projects. The first integrity threshold corresponds to the integrity level when the archival structure is basically closed, while the second integrity threshold corresponds to the integrity level when the archival structure has critical missing parts. The second integrity threshold is lower than the first integrity threshold to create a hysteresis interval and prevent frequent state switching. Currently, the archival data structure integrity of 0.222 is less than the first integrity threshold of 0.80 and does not fall within the interval between the first and second integrity thresholds; therefore, the project structure status is set to "unfixed." The archival identifier data set is written to the updatable storage area, which is configured to support overwrite and append writes. Archival data records are stored using file hash values ​​as object identifiers, and also record stage sequence identifiers, data type identifiers, and source node identifiers as object metadata. Currently, all six original records and supplementary records have been written to the updatable storage area.

[0075] After subsequent archiving events are completed, the archive data structure integrity is recalculated, resulting in an archive data structure integrity S=0.82. The project structure status identifier generated in the previous judgment period is still in an unfixed state. The current archive data structure integrity of 0.82 is not less than the first integrity threshold of 0.80, and the project structure status identifier is set to a fixed state. When the project structure status identifier is in a fixed state, a structure mapping index is generated. The structure mapping index contains the project identifier, generation timestamp, stage continuous sequence, and stage mapping set. Each element in the stage mapping set corresponds to a stage sequence identifier. Each element contains the stage sequence identifier and the data type set corresponding to that stage. Each data type set element contains the data type identifier, the number of archive data records, and the file hash value and storage path of the corresponding archive data record.

[0076] The corresponding archive data record in the updatable storage area is read based on the structure mapping index, and then written to the immutable storage area. The immutable storage area uses a write-once-unmodifiable policy; archive data records are stored using file hash values ​​as object identifiers, and overwriting and deletion are prohibited after writing. The structure mapping index is written to the immutable storage area as an independent object.

[0077] In subsequent assessment periods, the archival data structure integrity is recalculated, resulting in an archival data structure integrity S=0.58. The item structure status identifier generated in the previous assessment period is in a fixed state. The current archival data structure integrity of 0.58 is not greater than the second integrity threshold of 0.65, and the item structure status identifier is set to an unfixed state. After the item structure status identifier is set to an unfixed state, newly archived archival data records are written to the updatable storage area, while existing archival data records in the unmodifiable storage area remain unchanged.

[0078] Example 2: Based on the same inventive concept, such as Figure 2 As shown, this embodiment also provides a cloud-based archive data storage and management system, the system comprising:

[0079] The document identification generation module is used to obtain the document data set generated at different implementation stages of the project, and generate a stage sequence identifier, data type identifier, and source node identifier for each document data record, thus obtaining a document identification data set.

[0080] The phase mapping modeling module is used to sort the phase sequence identifiers according to the project implementation process and construct a continuous phase sequence; it maps the data type identifiers in the archive identifier data set to the corresponding positions in the continuous phase sequence to obtain the phase-type distribution matrix.

[0081] The structural integrity calculation module is used to determine whether each stage in a continuous sequence of stages has a data type identifier based on the stage-type distribution matrix, calculate the ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous sequence, and obtain the stage coverage. It then extracts the actual data type set corresponding to each stage based on the stage-type distribution matrix; obtains the expected data type set corresponding to each stage based on preset project process constraints; calculates the type coverage based on the matching degree between the actual data type set and the expected data type set; extracts the data type sets corresponding to any two stages in the continuous sequence of stages, and counts the number of reference relationships between any two stages based on data reference relationship identifiers; calculates the inter-stage connection consistency value based on the matching degree between the number of reference relationships and the preset number of reference relationships; calculates the deviation of stage coverage, type coverage, and inter-stage connection consistency value based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints, and obtains the structural constraint deviation value; combines stage coverage, type coverage, inter-stage connection consistency value, and structural constraint deviation value to obtain the archive data structure integrity; and finally, it determines the structural integrity to obtain the project structure status identifier.

[0082] The hierarchical storage control module is used to write the archive identification data set into the updatable storage area and record the version sequence number when the project structure status is identified as unfixed; when the project structure status is identified as fixed, it generates a structure mapping index based on the stage continuous sequence and stage-type distribution matrix, and writes the archive identification data set into the immutable storage area according to the structure mapping index, while generating multi-node replica data.

[0083] It should be noted that the specific ways in which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0084] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cloud computing-based method for archival data storage and management, characterized in that: The method includes: Obtain the set of archival data generated at different implementation stages of the project, and generate a stage sequence identifier, data type identifier, and source node identifier for each archival data record to obtain an archival identifier data set; The stage sequence identifiers are sorted according to the project implementation process to construct a continuous stage sequence; the data type identifiers in the archive identifier data set are mapped to the corresponding positions in the continuous stage sequence to obtain the stage-type distribution matrix; The following steps are performed to determine the presence of data type identifiers in each stage of a continuous sequence of stages based on the stage-type distribution matrix. The ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous sequence is calculated to obtain the stage coverage. The actual data type set corresponding to each stage is extracted based on the stage-type distribution matrix. The expected data type set corresponding to each stage is obtained based on preset project process constraints. The type coverage is calculated based on the matching degree between the actual data type set and the expected data type set. The data type sets corresponding to any two stages are extracted from the continuous sequence of stages, and the number of reference relationships between any two stages is counted based on data reference relationship identifiers. The inter-stage connection consistency value is calculated based on the matching degree between the number of reference relationships and the preset number of reference relationships. Based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints, the deviations of stage coverage, type coverage, and inter-stage connection consistency value are calculated to obtain the structural constraint deviation value. The completeness of the archive data structure is obtained by combining the stage coverage, type coverage, inter-stage connection consistency value, and structural constraint deviation value. Finally, the structural completeness is judged to obtain the project structure status identifier. When the project structure status is marked as unfixed, the archive identification data set is written to the updatable storage area; when the project structure status is marked as fixed, a structure mapping index is generated based on the phase continuous sequence and the phase-type distribution matrix, and the archive identification data set is written to the immutable storage area according to the structure mapping index.

2. The cloud computing-based archival data storage and management method according to claim 1, characterized in that, The method for mapping data type identifiers in the archive identifier dataset to corresponding positions in a continuous sequence of stages to obtain a stage-type distribution matrix includes: Based on the type-stage constraint relationship between the data type identifier and the stage sequence identifier, determine the candidate stage set for each archival data record; select the stage position that satisfies the node-stage correspondence from the candidate stage set to obtain the first stage position set; select the stage position where the generated timestamp falls within the corresponding stage time interval from the first stage position set to obtain the second stage position set. When the second stage location set contains multiple stage locations, select the stage locations that satisfy the stage dependency relationship from the second stage location set to obtain a candidate stage subset; for the archive data records with multiple stage locations in the candidate stage subset, perform priority comparison processing according to the stage sequence order to obtain a unique stage location identifier; write the data type identifier into the corresponding position of the continuous stage sequence according to the unique stage location identifier to construct a stage-type distribution matrix.

3. The cloud computing-based archival data storage and management method according to claim 2, characterized in that, The method for performing priority comparison processing on archive data records with multiple stage positions in the candidate stage subset to obtain a unique stage position identifier includes: Obtain the data reference relationship identifier in the archival data record; determine the target data record referenced by the archival data record based on the data reference relationship identifier, and obtain the stage position corresponding to the target data record; The stage position that matches the stage position of the target data record in the candidate stage subset is selected as the unique stage position identifier; when there is no stage position that matches the stage position of the target data record in the candidate stage subset, the stage position that is later in the sequence is selected as the unique stage position identifier.

4. The cloud computing-based archival data storage and management method according to claim 1, characterized in that, The method for calculating the structural integrity of archival data by combining phase coverage, type coverage, inter-phase consistency value, and structural constraint deviation value includes: The stage coverage, type coverage, and inter-stage consistency values ​​are normalized to obtain normalized stage coverage, normalized type coverage, and normalized inter-stage consistency values; the deviation suppression coefficient is calculated based on the structural constraint deviation values. ;in, This represents the structural constraint deviation value. The preset deviation suppression parameter; The basic completeness is calculated based on the coverage of the normalization stage, the coverage of the normalization type, and the consistency value of the connection between normalization stages. ;in, Based on completeness, To normalize the coverage, To normalize type coverage, To ensure consistency between normalization stages, , and These are the weighting coefficients, and The structural integrity of the archival data is calculated based on the deviation suppression coefficient and the basic integrity. .

5. The cloud computing-based archival data storage and management method according to claim 4, characterized in that, The weighting coefficient , and The methods for determining this include: Obtain the project type identifier of the current project; query the corresponding baseline weight coefficient combination from the preset weight configuration library based on the project type identifier; count the number of archive data records corresponding to each stage based on the stage-type distribution matrix, and calculate the time distribution dispersion based on the distribution of the number of archive data records corresponding to each stage in the stage continuous sequence; Based on the correspondence between the time distribution dispersion and the preset dispersion threshold range, the combination of weight adjustment coefficients is determined from the preset weight adjustment mapping table; the combination of baseline weight coefficients and the combination of weight adjustment coefficients are multiplied item by item, and the product results are normalized to obtain the dynamically adjusted weight coefficients. , and .

6. The cloud computing-based archival data storage and management method according to claim 1, characterized in that, The method for determining the structural integrity and obtaining the project structural status identifier includes: Within a preset judgment period, obtain the current archive data structure integrity and obtain the project structure status identifier generated in the previous judgment period; When the project structure status identifier generated in the previous judgment period is in an unfixed state and the current archive data structure integrity is not less than the first integrity threshold, the project structure status identifier will be determined to be in a fixed state. When the project structure status identifier generated in the previous judgment period is in a solidified state and the current archive data structure integrity is not greater than the second integrity threshold, the project structure status identifier will be determined to be in an unsolidified state. When the current archive data structure integrity is between the first integrity threshold and the second integrity threshold, the project structure status identifier is maintained as the project structure status identifier generated in the previous judgment period; wherein, the first integrity threshold is greater than the second integrity threshold.

7. A cloud-based archival data storage and management system, characterized in that, The system is used to perform the method according to any one of claims 1 to 6, the system comprising: The document identification generation module is used to obtain the collection of document data generated at different implementation stages of the project, and to generate a stage sequence identifier, data type identifier, and source node identifier for each document data record, thus obtaining a collection of document identification data. The phase mapping modeling module is used to sort the phase sequence identifiers according to the project implementation process and construct a continuous phase sequence; it maps the data type identifiers in the archive identifier data set to the corresponding positions in the continuous phase sequence to obtain the phase-type distribution matrix. The structural integrity calculation module is used to determine whether each stage in a continuous sequence of stages has a data type identifier based on the stage-type distribution matrix, calculate the ratio of the number of stages with existing data type identifiers to the total number of stages in the continuous sequence, and obtain the stage coverage; extract the actual data type set corresponding to each stage based on the stage-type distribution matrix; obtain the expected data type set corresponding to each stage based on preset project process constraints; calculate the type coverage based on the matching degree between the actual data type set and the expected data type set; extract the data type sets corresponding to any two stages in the continuous sequence of stages, and count the number of reference relationships between any two stages based on data reference relationship identifiers; calculate the inter-stage connection consistency value based on the matching degree between the number of reference relationships and the preset number of reference relationships; calculate the deviation of stage coverage, type coverage, and inter-stage connection consistency value based on the expected coverage, expected data type set, and expected number of reference relationships corresponding to each stage in the preset project process constraints, and obtain the structural constraint deviation value; combine the stage coverage, type coverage, inter-stage connection consistency value, and structural constraint deviation value to obtain the archive data structure integrity; and determine the structural integrity to obtain the project structure status identifier. The hierarchical storage control module is used to write the archive identification data set into the updatable storage area and record the version sequence number when the project structure status is identified as unfixed; when the project structure status is identified as fixed, it generates a structure mapping index based on the stage continuous sequence and stage-type distribution matrix, and writes the archive identification data set into the immutable storage area according to the structure mapping index, while generating multi-node replica data.

Images