Database adaptation integration method, system and equipment based on credential storage and medium

Abstract

The invention relates to the technical field of credential storage and database integration, and discloses a database adaptation integration method, system and equipment based on credential storage and a medium, and the database adaptation integration method based on credential storage comprises the following steps: extracting a protocol element definition set and a medium performance characteristic parameter matrix of a source system and a target system; generating a protocol difference vector by using an editing distance algorithm; generating a three-dimensional affinity tensor in a data block-medium-coding format by integrating the storage overhead, the access efficiency score and the medium performance parameters; optimizing, solving and generating a version-aware data coding conversion rule set and a medium distribution mapping table through a genetic algorithm; executing streaming data reading and carrying out online format conversion; and directly writing the converted data stream into the corresponding target storage medium. According to the method, integrated processing of protocol adaptation and medium optimization is realized, a migration window is effectively shortened, and temporary storage of an intermediate format is avoided.

Description

Technical Field

[0001] This invention relates to the field of information technology innovation storage and database integration technology, and more specifically, it relates to a database adaptation and integration method, system, device and medium based on information technology innovation storage. Background Technology

[0002] In the context of generational upgrades of domestically developed storage systems, enterprises face the dual challenges of changing database protocol versions and replacing storage media. For example, when upgrading a system from an older version of database using SATA SSDs to a newer version using NVMe SSDs, it is necessary not only to adapt to the new protocol format but also to fully leverage the performance advantages of the new media.

[0003] Traditional migration methods employ a separate processing strategy: first, data is converted from the old format to an intermediate format, temporarily stored, and then converted back to the new format; only then is optimized allocation of storage media considered. This approach suffers from the following technical problems: First, data undergoes multiple conversions and moves, each incurring I / O overhead and significantly extending the migration window. Second, the temporary storage of the intermediate format not only consumes additional storage space but also accelerates wear and tear on the storage media due to frequent read and write operations. Third, the data structure changes in the new protocol may be incompatible with the access characteristics of the new media; for example, nested data structures introduced by the new protocol may actually lead to performance degradation on media optimized for sequential access, failing to fully leverage the performance advantages of the new generation of storage media. Therefore, there is an urgent need for a database migration method that can uniformly handle protocol adaptation and media optimization to address the technical problems of excessively long migration windows, wasted storage resources, and performance mismatches. Summary of the Invention

[0004] This invention provides a database adaptation and integration method, system, device, and medium based on domestically developed storage, which solves the technical problems in related technologies where the separation of protocol adaptation and media optimization during database generational upgrades leads to extended migration windows, increased I / O overhead, and performance mismatch.

[0005] This invention discloses a database adaptation and integration method based on domestically developed storage, comprising the following steps: obtaining source system configuration information and target system configuration information; extracting a source protocol element definition set, a target protocol element definition set, and a media performance feature parameter matrix, wherein the media performance feature parameter matrix describes the performance indicators of various storage media; calculating the similarity between protocol elements in the source protocol element definition set and protocol elements in the target protocol element definition set using an edit distance algorithm; generating a protocol difference vector based on a matching algorithm that minimizes the overall difference, wherein the protocol difference vector quantifies the conversion complexity between the source protocol and the target protocol; performing comprehensive calculations on the storage overhead and access efficiency scores of data blocks with the media performance feature parameter matrix to generate a three-dimensional affinity tensor of data block-media-encoding format, wherein the three-dimensional affinity tensor reflects the degree of adaptation of a specific combination of data block, encoding format, and storage media; and using a genetic algorithm to analyze the three-dimensional affinity tensor. The force tensor is optimized to maximize the overall affinity value under storage capacity and encoding format compatibility constraints, generating a version-aware data encoding conversion rule set and a media allocation mapping table. Streaming data reading is performed, and each data block is converted online in memory according to the data encoding conversion rule set to generate a data stream conforming to the target protocol specification. The converted data stream is directly written to the corresponding target storage medium according to the media allocation mapping table, completing the integrated migration of protocol adaptation and media optimization. Each element of the three-dimensional affinity tensor is calculated by weighted combination of three indicators: storage overhead, access efficiency score, and performance matching degree. The storage overhead reflects the ratio of the encoded data size to the original data size. The access efficiency score is calculated based on the value of the corresponding element in the protocol difference vector combined with data access mode characteristics. The performance matching degree is calculated by weighted sum of the corresponding rows in the standardized media performance feature parameter matrix.

[0006] Further, the step of calculating similarity and generating protocol difference vectors using the edit distance algorithm includes: calculating the edit distance for each protocol element in the source protocol element definition set and each protocol element in the target protocol element definition set, where the edit distance represents the minimum number of edit operations required to convert one protocol element into another, and generating a protocol difference matrix; constructing a bipartite graph based on the protocol difference matrix, where the left nodes correspond to source protocol elements and the right nodes correspond to target protocol elements, with edge weights being the edit distance values; applying the Hungarian algorithm to solve for minimum weight perfect matching, ensuring that each source protocol element matches a target protocol element; extracting the edit distance between each source protocol element and its matched target protocol element in the matching results, normalizing it, and generating the protocol difference vector, where the normalization is achieved by dividing each edit distance value by the maximum value of all edit distance values, so that the difference values ​​are mapped to the interval between zero and one.

[0007] Further, the step of generating the three-dimensional affinity tensor includes: analyzing the access logs of the data to be migrated, identifying the access pattern characteristics of each data block, and generating an access feature vector, wherein the access feature vector includes access frequency, sequential access ratio, and random access ratio; standardizing the access frequency and scaling it to the range of zero to one, and setting the standardized access frequency to 0.5 when all data blocks have the same access frequency; calculating the comprehensive access efficiency score of each data block under different encoding formats based on the protocol difference vector and the access feature vector, wherein the comprehensive access efficiency score comprehensively considers protocol compatibility, the support efficiency of the encoding format for sequential and random access, and the impact of access frequency; for each combination of data block, encoding format, and storage medium, calculating the affinity value by weighting the storage overhead, the comprehensive access efficiency score, and the performance matching degree, wherein the performance matching degree is calculated by weighted summation of the dependence weight of the encoding format on various performance parameters of the storage medium and the standardized performance parameter values.

[0008] Further, the comprehensive access efficiency score is calculated as follows: the element corresponding to the encoding format in the protocol difference vector is inverted and then incremented by one to obtain the protocol compatibility component; the access mode adaptation component is obtained by multiplying the sequential access ratio of the data block by the sequential access support efficiency of the encoding format, and then multiplying it by the random access ratio of the data block by the random access support efficiency of the encoding format; the frequency adaptation component is obtained by multiplying the normalized access frequency by the high-frequency access support efficiency of the encoding format; and the comprehensive access efficiency score is obtained by weighted summation of the protocol compatibility component, the access mode adaptation component, and the frequency adaptation component.

[0009] Further, the steps of optimizing the solution using a genetic algorithm include: defining the optimization objective as maximizing the overall affinity value of all data blocks, where the overall affinity value is the sum of the affinity values ​​of each data block under its assigned encoding format and storage medium combination; setting storage capacity constraints to ensure that the sum of the encoded sizes of all data blocks allocated to the same storage medium does not exceed the product of the physical capacity and the upper limit of capacity utilization of the storage medium; setting encoding format compatibility constraints to ensure that the encoding format assigned to each data block supports the data type of the data block; setting migration time window constraints to ensure that the sum of the conversion time and transmission time of all data blocks does not exceed the maximum allowed migration time window; using integer encoding to represent chromosomes, where the first half of each chromosome represents the mapping relationship from data block to encoding format, and the second half represents the mapping relationship from data block to storage medium; iteratively optimizing through selection, crossover, and mutation operations, applying a penalty coefficient to individuals that violate the constraints to reduce their fitness value, and outputting the data encoding conversion rule set and the media allocation mapping table corresponding to the optimal allocation scheme.

[0010] Furthermore, the optimization objective employs a time-sensitive fitness function, which comprehensively considers three factors: overall affinity value, data transmission time, and format conversion time. The data transmission time is calculated based on the ratio of the encoded data size to the effective transmission bandwidth of the storage medium. The effective transmission bandwidth is the minimum of the storage medium write bandwidth and the network transmission bandwidth, multiplied by the network utilization coefficient. The format conversion time is calculated based on the original data size, CPU processing speed, and encoding format conversion complexity coefficient. The CPU processing speed is determined based on the product of CPU frequency, number of cores, number of instructions per clock cycle, and CPU utilization. The time-sensitive fitness function is constructed by assigning positive weights to the overall affinity value and negative weights to the data transmission time and format conversion time. The steps for performing streaming data reading and online format conversion include: Establish a streaming data channel from the source system to the target system, and set up a memory buffer for temporary data storage; Read data block content from the source system in fixed block sizes and parse the protocol element structure of the data blocks; The target encoding format corresponding to the current data block is determined according to the data encoding conversion rule set, and the corresponding encoding converter is called to perform format conversion. The encoding converter reorganizes the data fields, adjusts the byte order and updates the metadata identifier according to the target protocol specification. The converted data blocks are written to the output buffer. When the buffer reaches a preset threshold, the data is transferred in batches to the corresponding storage medium of the target system, avoiding temporary storage of intermediate formats.

[0011] Furthermore, it also includes the following steps: Based on the media allocation mapping table, a batch migration execution plan is generated, which aggregates data blocks with the same encoding format and target storage medium into a migration batch, reducing system call overhead through batch processing. Monitor the actual access performance metrics of the migrated data blocks in the target system, standardize each performance metric, compare it with the expected performance predicted based on the three-dimensional affinity tensor, and calculate the deviation value. When the deviation value exceeds the preset threshold, the weight coefficients in the calculation of the three-dimensional affinity tensor are adjusted, the affinity tensor of subsequent batches is recalculated, and an updated conversion rule set and allocation mapping table are generated to achieve dynamic optimization and adjustment.

[0012] The present invention also discloses a computer storage medium, characterized in that it stores computer-readable instructions, which, when read by a computer, enable the execution of the aforementioned database adaptation and integration method based on information technology innovation storage.

[0013] The present invention also discloses a computer device for reading computer-readable instructions stored in the aforementioned computer storage medium.

[0014] This invention also discloses a database adaptation and integration system based on domestically developed storage, used to execute the aforementioned database adaptation and integration method based on domestically developed storage, comprising: a configuration information extraction module for extracting a set of protocol element definitions and a matrix of media performance characteristic parameters from the source system and the target system; a protocol difference analysis module for generating a protocol difference vector based on an edit distance algorithm and a matching algorithm; an affinity tensor calculation module for generating a three-dimensional affinity tensor by comprehensively considering storage overhead, access efficiency scores, and media performance parameters; an optimization decision module for generating a data encoding conversion rule set and a media allocation mapping table through optimization using a genetic algorithm; a streaming conversion module for performing online format conversion and generating a data stream conforming to the target protocol specification; and a data writing module for writing the converted data stream to the corresponding target storage medium according to the media allocation mapping table.

[0015] This invention introduces a three-dimensional affinity tensor mechanism to uniformly model data block characteristics, encoding format requirements, and storage medium performance, enabling version adaptation and media allocation to be completed within the same decision-making framework. This overcomes the problem of multiple data movements caused by the separate processing of version conversion and media optimization in traditional methods, and solves the technical problems of extended migration window, increased I / O overhead, and waste of storage resources. It achieves significant technical effects such as shortening migration time, reducing storage medium wear, and fully leveraging the performance advantages of next-generation storage media. Attached Figure Description

[0016] Figure 1 This is the main flowchart of the database adaptation and integration method based on domestically developed storage of the present invention; Detailed Implementation

[0017] This embodiment provides a database adaptation and integration method based on domestically developed storage. Before execution, the database adaptation and integration method based on domestically developed storage requires configuring the basic environment of the source system and the target system, including the database instance of the source system, the storage media array of the target system, and the network connection channel.

[0018] Step 1: Obtain the source system configuration information and target system configuration information, and extract the protocol element definition set and media performance characteristic parameter matrix.

[0019] Read the protocol version specification file and data storage format definition from the source system, and parse them to obtain the set of source protocol element definitions. ,in Represents the first of the source system The protocol element definition set is obtained by reading the new version of the protocol specification and storage medium configuration file from the target system and parsing them. and the matrix of characteristic parameters of media performance .

[0020] Furthermore, the specific implementation process of protocol element parsing is as follows: First, read the protocol specification file (usually in XML, JSON, or IDL format) and extract basic elements such as data type declarations, table structure definitions, and index configurations through a parser; Second, convert each protocol element into a standardized string representation in the format "element type: element name: attribute list", for example, TABLE: user_info: id (INT), name (VARCHAR), age (INT); Third, construct the syntax tree structure of the protocol elements, record the dependencies and hierarchical relationships between elements, and generate a complete set of protocol element definitions.

[0021] It should be noted that the medium performance characteristic parameter matrix It is The matrix, where Indicates the number of storage media types. Indicates the number of performance parameter types, matrix elements Indicates the first The first type of storage medium The performance parameters include IOPS (Input / Output Operations Per Second), latency, sequential read / write bandwidth, and random read / write bandwidth.

[0022] Furthermore, the dielectric performance characteristic parameter matrix The specific acquisition method is as follows: First, perform standardized performance tests on each storage medium. Use the fio tool to perform a 4KB random read / write test to obtain the IOPS value, use the ioping tool to measure the average access latency, and use the dd command to test the bandwidth values ​​of 1MB sequential read / write and 4KB random read / write respectively. Second, execute each test 10 times and take the average value to eliminate measurement errors. Set the test time window to 60 seconds to ensure the stability of the results. Third, store the test results in a performance parameter matrix, with each row corresponding to a storage medium and each column corresponding to a performance indicator. Fourth, record the test environment parameters, including CPU model, memory size, and operating system version, as the basis for verifying the validity of the performance data.

[0023] Since different performance parameters have different dimensions (IOPS is measured in seconds, latency in milliseconds, and bandwidth in MB / s), to eliminate the influence of these dimensions on subsequent calculations, the media performance characteristic parameter matrix needs to be standardized, scaling the parameter values ​​to a scaled value. The interval, the standardized formula is .

[0024] Furthermore, to avoid the denominator being zero in the standardized formula, when At that time, it indicates that all storage media are in the first... When performance parameters have the same value, set... This ensures the numerical stability of the standardization process.

[0025] Step 2: Calculate the similarity between the new and old protocol elements using the edit distance algorithm to generate a protocol difference vector.

[0026] For the source protocol element set Each element in and the set of target protocol elements Each element in Calculate its edit distance Generate protocol difference matrix , where matrix elements Based on the protocol difference matrix, a protocol difference vector is generated using a matching algorithm that minimizes the overall difference. ,in Indicates the source protocol number The degree of difference between an element and its best-matching target protocol element.

[0027] The aforementioned edit distance algorithm takes as input two string sequences of protocol elements and outputs a non-negative integer value representing the minimum number of edit operations required to convert one protocol element into another. Edit operations include three basic operations: insertion, deletion, and replacement of characters.

[0028] The aforementioned matching algorithm for minimizing overall differences includes the following steps: First, based on the protocol difference matrix... The first step is to construct a bipartite graph, where left-side nodes correspond to source protocol elements and right-side nodes correspond to target protocol elements, with edge weights being the edit distance values. The second step is to apply the Hungarian algorithm to find the minimum weight perfect match, ensuring that each source protocol element matches a target protocol element. The third step is to extract the edit distance between each source protocol element and its matched target protocol element from the matching results, normalize the distance, and generate a protocol difference vector. The normalization formula is This maps the difference values ​​to Interval.

[0029] Furthermore, the reasons for choosing between the edit distance algorithm and the Hungarian algorithm are as follows: The edit distance algorithm is suitable for calculating the similarity of protocol elements because protocol elements can be converted into string sequences, and the edit distance can accurately quantify the structural differences between strings. The smaller the value, the more similar the protocol elements are, and the lower the conversion cost. The Hungarian algorithm is suitable for matching problems that minimize overall differences because this problem is essentially finding the minimum weight perfect match in a bipartite graph, and the Hungarian algorithm can... Find the globally optimal solution within the time complexity, ensuring that each source protocol element can match the target protocol element with the minimum conversion cost.

[0030] Furthermore, when all protocol elements match perfectly, it leads to... When the target protocol is fully compatible, the source protocol is set to 1. For all This ensures the numerical stability of the protocol difference vector.

[0031] Step 3: Perform matrix operations on the storage overhead, access efficiency score and media performance parameter matrix to generate a three-dimensional affinity tensor of data block-media-encoding format.

[0032] This step generates the three-dimensional affinity tensor through the following calculation process. : First, define the data block set. Encoding format set Storage media type set ; Then, for each data block Encoding format and storage media Calculate the affinity value based on the combination of factors: ; in, Represents data block Encoding format The storage overhead is calculated by the ratio of the original size of the data block to the size after encoding; Represents data block Encoding format Access efficiency score based on protocol difference vector Calculate the value of the corresponding element in the middle; Indicates encoding format With storage media The performance matching degree is determined by the standardized medium performance characteristic parameter matrix. The weighted sum of the corresponding rows is calculated. , , For the weighting coefficients, satisfying .

[0033] Furthermore, storage overhead The specific calculation formula is as follows ,in Represents data block The number of raw bytes, Represents data block Encoding format The storage overhead calculation formula, based on the number of bytes after compression, ensures that encoding formats with high compression efficiency achieve lower overhead values. Performance matching. The calculation formula is: ,in Indicates encoding format For storage media The performance parameters are assigned weights, the weight values ​​of which are determined based on the I / O access characteristics of the encoding format.

[0034] Furthermore, the specific method for measuring data block size is as follows: By reading data blocks The metadata obtained includes the total number of bytes in the table header information, row data, and index structure; It is obtained through a simulated encoding process, specifically in the following steps: First, extract the data block. The first step is to obtain sample data (either the first 1000 rows or 10% of the total data, whichever is smaller); the second step is to apply an encoding format to the sample data. The third step involves performing the actual encoding conversion; measuring the number of bytes in the encoded sample and calculating the compression ratio. The fourth step is to calculate the encoded size of the complete data block based on the compression ratio. .

[0035] Furthermore, depending on weights The specific calculation method is as follows: First step, analyze the encoding format. The first step involves analyzing the I / O access patterns and determining their sensitivity to four performance parameters: IOPS, latency, sequential bandwidth, and random bandwidth. The second step involves assigning basic weights based on these sensitivity levels, and setting encoding formats for IOPS-sensitive parameters. For delay-sensitive formatting For sequential bandwidth-sensitive format settings For format settings sensitive to random bandwidth The third step is to adjust the weights based on the data access granularity of the encoding format. For small-granularity access (<4KB), the IOPS weight is increased by 0.1; for large-granularity access (>64KB), the bandwidth weight is increased by 0.1. The fourth step is to perform normalization processing to ensure… .

[0036] Furthermore, weighting coefficients , , The range of values ​​is ,in Used to adjust the importance of storage overhead. The importance of adjusting access efficiency. The recommended typical value for adjusting the importance of performance matching is [value to be filled in]. , , This indicates that access efficiency is given priority, while storage overhead and performance matching are taken into account.

[0037] It should be noted that the three-dimensional affinity tensor Each element It reflects the degree of fit of a specific combination of data blocks, encoding format and storage medium; the larger the value, the better the combination.

[0038] Furthermore, the three-dimensional affinity tensor Each element The theoretical range of values ​​is This is because of its three components , , The range of values ​​after standardization is all And weighting coefficient ,therefore The actual range of values ​​is constrained by Within the range, a value close to 1 indicates that the combination has the best fit.

[0039] In this embodiment of the application, in order to improve the accuracy of affinity calculation, the following steps are also included in step 3: Step 3.1: Analyze the access logs of the data to be migrated, identify the access pattern characteristics of the data blocks, and generate access feature vectors.

[0040] Extract access records for each data block within a past time window from the access logs of the source system, and calculate the access frequency. Sequential access ratio and random access ratio Generate data blocks Access feature vector .

[0041] To eliminate the impact of the dimensional difference between access frequency (times / time unit) and access ratio (dimensionless) on subsequent calculations, the access frequency needs to be standardized and scaled to a scaled value. Interval: .

[0042] Furthermore, when all data blocks are accessed at the same frequency, it leads to... When this time, it indicates that the access load of each data block is balanced, and at this time, the settings are... For all Established to ensure the numerical stability of standardized processing.

[0043] The aforementioned access pattern feature identification includes the following calculations: access frequency ,in For data blocks In the time window Number of visits within; percentage of sequential visits ,in The number of consecutive accesses to adjacent data blocks; random access ratio. .

[0044] Furthermore, time window The range of values ​​is The recommended time window is 8 hours. Typical business cycles should be covered to ensure the accuracy of access pattern recognition.

[0045] Step 3.2: Based on the protocol difference vector and access feature vector, calculate the storage overhead and access efficiency score of each data block under different encoding formats.

[0046] Combined with protocol difference vector and the standardized access feature vector The data block is calculated using the following formula. Encoding format Overall access efficiency score: ; in, It is the protocol difference vector Chinese and encoding formats The corresponding element, and These represent the encoding formats. Efficiency in supporting sequential and random access Indicates encoding format Efficiency in supporting high-frequency access. , and For the weight parameters, satisfying .

[0047] Furthermore, protocol difference vectors Middle elements With encoding format The correspondence is achieved through the format mapping function. Establish, among which Mapping function Return encoding format The index position in the protocol difference vector is determined based on the matching results of protocol elements, and is specifically calculated as follows: ,in Indicates encoding format Protocol signature, Indicates source protocol element The protocol signature.

[0048] Furthermore, weight parameters , , The range of values ​​is The recommended typical value is , , This indicates that protocol compatibility and access pattern adaptability are equally important, while access frequency has a relatively smaller impact. Efficiency parameters are supported. , and The range of values ​​is The value is 1, which indicates full support, and 0 indicates no support. The specific value is determined according to the technical specifications of the encoding format.

[0049] Furthermore, the specific method for obtaining the efficiency parameters is as follows: First, read the encoding format. The first step is to extract the access mode optimization features from the technical specification document; the second step is to calculate the sequential access support efficiency based on the data organization method of the encoding format. The third step is to calculate the random access support efficiency based on the indexing mechanism of the encoding format. For row-oriented storage format, the efficiency is set to 0.9; for column-oriented storage format, it is set to 0.6; and for mixed storage format, it is set to 0.8. The format with a B+ tree index is set to 0.9, the format with a hash index is set to 0.8, and the format without an index is set to 0.3; the fourth step is to calculate the high-frequency access support efficiency based on the caching mechanism of the encoding format. The format with memory caching is set to 0.9, the format with SSD caching is set to 0.7, and the format without caching is set to 0.4.

[0050] Step 4: Use a genetic algorithm to optimize the affinity tensor and generate a version-aware data encoding conversion rule set and a media allocation mapping table.

[0051] Three-dimensional affinity tensor As input to the fitness function, the optimization objective is defined as maximizing the overall affinity value of all data blocks, with constraints including capacity limitations of each storage medium and compatibility requirements of the encoding format. Through selection, crossover, and mutation operations using a genetic algorithm, the optimal allocation scheme is iteratively optimized, outputting a set of data encoding conversion rules. and media allocation mapping table .

[0052] Furthermore, the mathematical formal expression of the optimization problem is as follows: the optimization objective function is... Storage capacity constraint is ,in Indicates storage medium Physical capacity, This represents the upper limit of capacity utilization, with a value range of [value missing]. The recommended value is 0.85; the encoding format compatibility constraint is... ,in This is a compatibility judgment function; when the encoding format... Supported data types Returns 1 if the migration time is met, otherwise returns 0; the migration time window constraint is... ,in This indicates the maximum allowed migration time window.

[0053] Furthermore, migration time window The calculation method is as follows: ,in This indicates the maximum allowable downtime for the business, in hours. It is determined through business requirements analysis. Typical values ​​include 2 hours for critical business systems, 8 hours for general business systems, and 24 hours for non-critical business systems. This represents the time tolerance coefficient, with a value range of [value missing]. The recommended value is 0.8, and the time tolerance coefficient is a buffer time reserved during the migration process to cope with unexpected situations; data type compatibility functions. By parsing data blocks The metadata is obtained, and the return value includes identifiers of basic data types such as integer, floating-point, string, and binary.

[0054] Furthermore, the reasons for choosing a genetic algorithm are as follows: First, the three-dimensional affinity tensor optimization problem is an NP-hard multi-objective combinatorial optimization problem, with a search space size of... First, traditional gradient descent algorithms cannot handle discrete variables and multiple constraints. Second, genetic algorithms are naturally suited for maximization problems, retaining individuals with high fitness through selection operations, which is highly consistent with the goal of maximizing the overall affinity value. Third, genetic algorithms have global search capabilities, which can avoid getting trapped in local optima, and this is crucial for complex three-dimensional affinity tensor optimization. Fourth, genetic algorithms are easy to handle constraints, and can effectively handle storage capacity limitations and format compatibility constraints through penalty function mechanisms.

[0055] The aforementioned genetic algorithm uses integer encoding, where each chromosome is represented by a string of length 1. an integer vector, where the first... Each gene bit represents the mapping relationship from a data block to a coding format, and then... Each gene bit represents the mapping relationship from a data block to the storage medium, and the value range of the gene is as follows: and The algorithm input includes a three-dimensional affinity tensor. The population size, number of iterations, crossover probability, and mutation probability are considered. The output is the encoding conversion rule set and media allocation mapping table corresponding to the optimal allocation scheme. The fitness function is calculated using the following formula: ,in and Representing data blocks The assigned encoding format and storage medium.

[0056] Furthermore, to reflect the role of the time dimension in the optimization objective, a time-sensitive fitness function is introduced. ,in Represents data block Encoding format Transfer to storage medium The required time is calculated using the following formula: ; Represents data block Convert to encoding format The required time is calculated using the following formula: Weighting coefficient , , satisfy Recommended value , , .

[0057] Furthermore, the definitions of the time calculation-related parameters are as follows: Indicates storage medium The effective transmission bandwidth, measured in MB / s, is determined by the media performance characteristic parameter matrix. The bandwidth parameter is obtained from the formula, which is: ,in For storage medium Write bandwidth, For network transmission bandwidth, This is the network utilization coefficient, with a value range of [value range missing]. The recommended value is 0.8; This indicates the CPU processing speed for data conversion and processing, measured in MB / s. It is obtained through system configuration and calculated using the following formula: ,in CPU frequency, Number of CPU cores Number of instructions per clock cycle For CPU utilization, a value of 0.7 is recommended. Indicates encoding format The transformation complexity coefficient is dimensionless and its value ranges from 0 to 1. The simple format conversion (such as type conversion) is set to 1.0, the compressed format conversion is set to 2.5, the encrypted format conversion is set to 4.0, and the comprehensive conversion is set to 5.0.

[0058] Furthermore, the specific implementation of the genetic algorithm is as follows: the selection operation adopts the roulette wheel selection method, based on the fitness value. Calculate the probability of an individual being selected ,in The population size is specified; the crossover operation uses a single-point crossover method, randomly selecting a crossover point to split the two parent chromosomes and exchange the latter half to generate offspring; the mutation operation uses a uniform mutation method, randomly replacing each gene position with other values ​​within its range with a mutation probability; the constraint handling mechanism ensures that the total allocated capacity of each storage medium does not exceed its physical capacity limit, and a penalty coefficient of 0.5 is applied to the fitness value when the constraint is violated.

[0059] Furthermore, the parameter constraints for the genetic algorithm are as follows: the population size ranges from [value missing]. The recommended value is 100; the number of iterations can range from [value missing]. The recommended value is 300; the crossover probability range is [value missing]. The recommended value is 0.8; the mutation probability range is [value missing]. The recommended value is 0.05. These parameter settings can ensure the convergence of the algorithm and the quality of the solution.

[0060] Step 5: Perform streaming data reading, apply the encoding conversion rule set for online format conversion, and output the converted data stream.

[0061] According to the data encoding conversion rule set For each data block Read the raw data stream from the source system and apply the corresponding encoding format in memory. Perform real-time conversion to generate a data stream that conforms to the target protocol specification and transmit it directly to the target system, avoiding temporary storage of intermediate formats.

[0062] Furthermore, the specific implementation process of online format conversion is as follows: First, establish a streaming data channel from the source system to the target system, and set the memory buffer size to 64MB for temporary data storage; Second, read data block content from the source system at fixed block sizes (default 8KB) and parse the protocol element structure of the data blocks; Third, according to the conversion rule set... Determine data blocks Corresponding target encoding format The first step involves calling the corresponding encoding converter to perform format conversion. The converter reorganizes the data fields, adjusts the byte order, and updates the metadata identifier according to the target protocol specification. The fourth step is to write the converted data blocks into the output buffer. When the buffer reaches the threshold, the data is transferred in batches to the corresponding storage medium of the target system.

[0063] Step 6: Write the converted data stream to the corresponding target storage medium according to the media allocation mapping table, and record the migration progress marker.

[0064] According to the media allocation mapping table Write the converted data stream output from step 5 to the specified storage medium. The migration status and timestamp of each data block are recorded in the metadata database to generate a migration progress report.

[0065] In this embodiment of the application, in order to improve migration efficiency, the following steps are also included in addition to steps 5 and 6: Step 7: Generate a batch migration execution plan based on the mapping table, and aggregate data blocks with the same conversion rules and target media into migration batches.

[0066] Scan data encoding conversion rule set and media allocation mapping table They will have the same encoding format and target medium Data blocks are aggregated into a batch to generate a batch migration execution plan, reducing system call overhead through batch processing.

[0067] Step 8: Monitor the actual access performance indicators after migration, calculate the deviation from the expected performance, and dynamically adjust the encoding rules and media allocation strategies for subsequent batches.

[0068] The actual access performance metrics of the migrated data blocks in the new system are collected, including response time and throughput. Since different performance metrics have different units (response time is in milliseconds, throughput is in MB / s), each performance metric needs to be standardized before being compared with the expected performance predicted based on affinity tensor, and the deviation value is calculated. When the deviation exceeds the threshold, the weighting coefficient is adjusted. , , The affinity tensor for subsequent batches is recalculated to generate updated transformation rules and allocation strategies.

[0069] Furthermore, the deviation threshold The range of values ​​is The recommended value is 0.2, and the deviation threshold represents the maximum permissible relative deviation between actual and expected performance. When When this occurs, it indicates that the current allocation strategy's predictive accuracy is insufficient and dynamic adjustments are needed; the adjustment range of the weighting coefficients is set to... This ensures the gradual and stable adjustment of strategies.

[0070] This implementation introduces a three-dimensional affinity tensor mechanism to uniformly model data block characteristics, encoding format requirements, and storage medium performance, enabling version adaptation and media allocation to be completed within the same decision-making framework. This overcomes the problem of multiple data movements caused by the separate processing of version conversion and media optimization in traditional methods, thereby solving the technical problems of extended migration window and increased I / O overhead.

[0071] Specifically, the protocol difference vector quantifies the conversion complexity between new and old version protocol elements, providing a precise basis for the selection of encoding format; the streaming online conversion mechanism enables data to be converted in format while being read and directly written to the target medium, eliminating the extra I / O operations generated by intermediate temporary format storage, thereby solving the problems of storage space waste and accelerated media wear.

[0072] Furthermore, the three-dimensional affinity tensor comprehensively considers three dimensions: data access mode, protocol compatibility, and media performance characteristics. It obtains the globally optimal allocation scheme through genetic algorithm optimization, ensuring that the data structure of the new version protocol can match the access characteristics of the new media, giving full play to the performance advantages of the new generation of storage media, and solving the performance degradation problem caused by the mismatch between protocol structure and media characteristics.

[0073] A city commercial bank needs to upgrade its core business system from an Oracle 11g database and SATA SSD storage to a MySQL 8.0 database and NVMe SSD storage. The upgrade involves three core data tables: a customer information table (1 million records, average 0.8KB per record), a transaction record table (5 million records, average 1.2KB per record), and an account balance table (2 million records, average 0.5KB per record). The source system uses a SATA SSD array (2TB capacity, 8000 IOPS, average latency 0.5ms), while the target system is configured with an NVMe SSD array (4TB capacity, 50000 IOPS, average latency 0.1ms) and a traditional HDD array (10TB capacity, 200 IOPS, average latency 5ms) for cold data storage.

[0074] Step 1: Extraction of configuration information from source and target systems The protocol element definition set extracted from the Oracle 11g system contains the following elements. The target protocol element definition set is extracted from the MySQL 8.0 system, and storage media performance parameters are also obtained: Table 1: Set of source protocol element definitions

[0075] Table 2: Set of Target Protocol Element Definitions

[0076] Table 3 Matrix of Media Performance Characteristic Parameters (Before standardization):

[0077] Table 4 Standardized Media Performance Characteristic Parameter Matrix :

[0078] Step 2: Protocol Difference Calculation Calculate the edit distance between each protocol element and generate a protocol difference vector: Table 5 Protocol Difference Matrix :

[0079] After solving for the minimum weight perfect match using the Hungarian algorithm, the protocol difference vector is obtained: ; in , , , , .

[0080] Step 3: Calculation of the three-dimensional affinity tensor For three data blocks ( Customer Information Form Transaction record table Account balance sheet) and two encoding formats ( Row-based storage format (Columnar storage format), calculate access feature vectors and affinity values: Table 6 Data Block Access Feature Vectors:

[0081] Mapping relationship between encoding format and protocol difference vector: Based on format mapping function , Table 7 calculates the similarity between the protocol signature of the encoded format and the source protocol elements:

[0082] Table 8 Storage overhead matrix:

[0083] Table 9 Supports Efficiency Parameters for Encoding Formats:

[0084] Access efficiency score Matrix calculation (weight parameters) , , ): against (Row-based storage) :

[0085] against (Columnar storage) :

[0086] Table 10 Access Efficiency Score matrix:

[0087] Performance matching calculate: For NVMe SSD media (all standardized performance parameters are 1.0), the performance matching degree of each encoding format is 0. For HDD media (where all standardized performance parameters are 0), the performance matching degree of each encoding format is 0. .

[0088] 3D Affinity Tensor Calculate (weighting coefficients) , , ): According to the formula : Affinity values ​​for NVMe SSD media: against Data blocks:

[0089] against Data blocks:

[0090] against Data blocks:

[0091] Table 11 Affinity value matrix for NVMe SSD media:

[0092] Affinity value under HDD media: because The affinity score is determined solely by storage overhead and access efficiency scores: against Data blocks:

[0093] against Data blocks:

[0094] against Data blocks:

[0095] Table 12 Affinity value matrix for NVMe SSD media:

[0096] Step 4: Genetic Algorithm Optimization Results Based on the three-dimensional affinity tensor, the genetic algorithm parameters are set (population size 100, number of iterations 300, crossover probability 0.8, mutation probability 0.05), and considering storage capacity constraints, the optimal allocation scheme is obtained: Table 13 Data Encoding Conversion Rule Set :

[0097] Table 14 Media Allocation Mapping Table :

[0098] Fitness function optimization process: Based on the updated 3D affinity tensor, the fitness value of the optimal assignment scheme is... : Table 15 Optimal Allocation Scheme:

[0099] Final migration results summary Table 16 Comparison of system configuration and performance after migration:

Claims

1. A database adaptation and integration method based on domestically developed storage, characterized in that, Includes the following steps: Obtain source system configuration information and target system configuration information, extract source protocol element definition set, target protocol element definition set and media performance characteristic parameter matrix, the media performance characteristic parameter matrix describes the performance indicators of various storage media; The similarity between protocol elements in the source protocol element definition set and protocol elements in the target protocol element definition set is calculated using the edit distance algorithm. A protocol difference vector is generated based on a matching algorithm that minimizes the overall difference. The protocol difference vector quantifies the conversion complexity between the source protocol and the target protocol. The storage overhead and access efficiency score of the data block are combined with the media performance characteristic parameter matrix to generate a three-dimensional affinity tensor of data block-media-encoding format. The three-dimensional affinity tensor reflects the degree of fit of a specific combination of data block, encoding format and storage medium. The three-dimensional affinity tensor is optimized and solved using a genetic algorithm to maximize the overall affinity value under storage capacity constraints and encoding format compatibility constraints, thereby generating a version-aware data encoding conversion rule set and a media allocation mapping table. Perform streaming data reading, and perform online format conversion of each data block in memory according to the data encoding and conversion rule set to generate a data stream that conforms to the target protocol specification; The converted data stream is directly written to the corresponding target storage medium according to the media allocation mapping table, thus completing the integrated migration of protocol adaptation and media optimization. Each element of the three-dimensional affinity tensor is calculated by weighted combination of three indicators: storage overhead, access efficiency score, and performance matching degree. The storage overhead reflects the ratio of the encoded data size to the original data size. The access efficiency score is calculated based on the value of the corresponding element in the protocol difference vector combined with the data access mode characteristics. The performance matching degree is calculated by weighted sum of the corresponding rows in the standardized media performance characteristic parameter matrix.

2. The database adaptation and integration method based on domestically developed storage according to claim 1, characterized in that, The steps of calculating similarity and generating protocol difference vectors using the edit distance algorithm include: For each protocol element in the source protocol element definition set and each protocol element in the target protocol element definition set, calculate their edit distance, where the edit distance represents the minimum number of edit operations required to transform one protocol element into another, and generate a protocol difference matrix; A bipartite graph is constructed based on the protocol difference matrix, where the left-side nodes correspond to source protocol elements and the right-side nodes correspond to target protocol elements, with the edge weights being the edit distance values. The Hungarian algorithm is applied to find the minimum weight perfect match, ensuring that each source protocol element matches a target protocol element. The edit distance between each source protocol element and its matched target protocol element in the matching results is extracted, and the protocol difference vector is generated after normalization. The normalization is achieved by dividing each edit distance value by the maximum value of all edit distance values, so that the difference values ​​are mapped to the interval between zero and one.

3. The database adaptation and integration method based on information technology innovation storage according to claim 1, characterized in that, The steps to generate a three-dimensional affinity tensor include: Analyze the access logs of the data to be migrated, identify the access pattern characteristics of each data block, and generate an access feature vector, which includes access frequency, sequential access ratio, and random access ratio. The access frequency is standardized and scaled to the range of zero to one. When the access frequency of all data blocks is the same, the standardized access frequency is set to 0.

5. Based on the protocol difference vector and the access feature vector, a comprehensive access efficiency score for each data block is calculated under different encoding formats. The comprehensive access efficiency score takes into account protocol compatibility, the efficiency of encoding format in supporting sequential and random access, and the impact of access frequency. For each combination of data block, encoding format, and storage medium, an affinity value is calculated by weighting the storage overhead, the overall access efficiency score, and the performance matching degree. The performance matching degree is calculated by weighting and summing the dependence weights of the encoding format on the various performance parameters of the storage medium with the standardized performance parameter values.

4. The database adaptation and integration method based on information technology innovation storage according to claim 3, characterized in that, The overall access efficiency score is calculated in the following way: Invert the element corresponding to the encoding format in the protocol difference vector and add one to obtain the protocol compatibility component. Multiply the sequential access ratio of the data block by the sequential access support efficiency of the encoding format, and then multiply it by the random access ratio of the data block by the random access support efficiency of the encoding format, and sum them to obtain the access mode adaptation component. The frequency adaptation component is obtained by multiplying the standardized access frequency by the high-frequency access support efficiency of the encoding format. The comprehensive access efficiency score is obtained by weighted summation of the protocol compatibility component, the access mode adaptation component, and the frequency adaptation component.

5. The database adaptation and integration method based on information technology innovation storage according to claim 1, characterized in that, The steps for optimization using a genetic algorithm include: The optimization objective is defined as maximizing the overall affinity value of all data blocks, where the overall affinity value is the sum of the affinity values ​​of each data block under its assigned combination of encoding format and storage medium. Set storage capacity constraints to ensure that the sum of the encoded sizes of all data blocks allocated to the same storage medium does not exceed the product of the physical capacity of the storage medium and the upper limit of capacity utilization. Set encoding format compatibility constraints to ensure that the encoding format assigned to each data block supports the data type of that data block; Set migration time window constraints to ensure that the sum of the transformation time and transfer time of all data blocks does not exceed the maximum allowed migration time window; Chromosomes are represented using integer encoding. The first half of each chromosome represents the mapping relationship between data blocks and encoding format, and the second half represents the mapping relationship between data blocks and storage medium. Through iterative optimization using selection, crossover, and mutation operations, a penalty coefficient is applied to individuals that violate the constraints to reduce their fitness value, and the data encoding conversion rule set and the media allocation mapping table corresponding to the optimal allocation scheme are output.

6. The database adaptation and integration method based on domestically developed storage according to claim 5, characterized in that, The optimization objective employs a time-sensitive fitness function, which comprehensively considers three factors: overall affinity value, data transmission time, and format conversion time. The data transmission time is calculated based on the ratio of the encoded data size to the effective transmission bandwidth of the storage medium. The effective transmission bandwidth is the minimum of the storage medium write bandwidth and the network transmission bandwidth, multiplied by the network utilization coefficient. The format conversion time is calculated based on the original data size, CPU processing speed, and encoding format conversion complexity coefficient. The CPU processing speed is determined based on the product of CPU frequency, number of cores, number of instructions per clock cycle, and CPU utilization. The time-sensitive fitness function is constructed by assigning positive weights to the overall affinity value and negative weights to the data transmission time and format conversion time. The steps for performing streaming data reading and online format conversion include: Establish a streaming data channel from the source system to the target system, and set up a memory buffer for temporary data storage; Read data block content from the source system in fixed block sizes and parse the protocol element structure of the data blocks; The target encoding format corresponding to the current data block is determined according to the data encoding conversion rule set, and the corresponding encoding converter is called to perform format conversion. The encoding converter reorganizes the data fields, adjusts the byte order and updates the metadata identifier according to the target protocol specification. The converted data blocks are written to the output buffer. When the buffer reaches a preset threshold, the data is transferred in batches to the corresponding storage medium of the target system, avoiding temporary storage of intermediate formats.

7. The database adaptation and integration method based on information technology innovation storage according to claim 1, characterized in that, It also includes the following steps: Based on the media allocation mapping table, a batch migration execution plan is generated, which aggregates data blocks with the same encoding format and target storage medium into a migration batch, reducing system call overhead through batch processing. Monitor the actual access performance metrics of the migrated data blocks in the target system, standardize each performance metric, compare it with the expected performance predicted based on the three-dimensional affinity tensor, and calculate the deviation value. When the deviation value exceeds the preset threshold, the weight coefficients in the calculation of the three-dimensional affinity tensor are adjusted, the affinity tensor of subsequent batches is recalculated, and an updated conversion rule set and allocation mapping table are generated to achieve dynamic optimization and adjustment.

8. A computer storage medium, characterized in that, It stores computer-readable instructions, which, when read by a computer, enable the execution of the database adaptation and integration method based on domestically developed storage as described in any one of claims 1 to 7.

9. A computer device, characterized in that, Used to read computer-readable instructions stored in the computer storage medium as described in claim 8.

10. A database adaptation and integration system based on domestically developed storage, used to execute the database adaptation and integration method based on domestically developed storage as described in any one of claims 1 to 7, characterized in that, include: The configuration information extraction module is used to extract the protocol element definition set and the medium performance characteristic parameter matrix from the source system and the target system. The protocol difference analysis module is used to generate protocol difference vectors based on edit distance and matching algorithms. The affinity tensor calculation module is used to generate a three-dimensional affinity tensor by comprehensively considering storage overhead, access efficiency scores, and media performance parameters. The optimization decision module is used to generate a data encoding conversion rule set and a media allocation mapping table by optimizing the solution through a genetic algorithm. The streaming conversion module is used to perform online format conversion and generate data streams that conform to the target protocol specifications. The data writing module is used to write the converted data stream to the corresponding target storage medium by allocating a mapping table according to the medium.

Images