Database adaptation integration method, system and device based on xinchuang storage and medium

Abstract

The present application relates to the technical field of Xinchuang storage and database integration, and discloses a database adaptive integration method, system, device and medium based on Xinchuang storage, wherein the database adaptive integration method based on Xinchuang storage comprises the following steps: extracting a protocol element definition set and a medium performance characteristic parameter matrix of a source system and a target system; generating a protocol difference vector by using an edit distance algorithm; generating a three-dimensional affinity tensor of a data block-medium-encoding format by comprehensively storing overhead, access efficiency score and medium performance parameters; generating a version-aware data encoding conversion rule set and a medium allocation mapping table by solving and generating through a genetic algorithm optimization; performing streaming data reading and online format conversion; and directly writing the converted data stream into a corresponding target storage medium. The method realizes integrated processing of protocol adaptation and medium optimization, effectively shortens the migration window, and avoids temporary storage of an intermediate format.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Xinchuang storage and database integration, and more particularly to a database adaptation integration method, system, device and medium based on Xinchuang storage. BACKGROUND

[0002] In the cross-generation upgrade scenario of the Xinchuang storage system, the enterprise needs to handle the dual challenges of database protocol version change and storage medium replacement. For example, when the system is upgraded from the old version of the database using SATA SSD to the new version of the database using NVMe SSD, not only the new protocol format needs to be adapted, but also the performance advantages of the new medium need to be fully utilized.

[0003] The traditional migration method adopts a separate processing strategy, first converts the data from the old version format to an intermediate format, temporarily stores it, and then converts it to the new version format, and finally considers the optimization allocation of the storage medium. This method has the following technical problems: first, the data needs to go through multiple conversions and movements, and each conversion will generate I / O overhead, significantly extending the migration window; second, the temporary storage of the intermediate format not only occupies additional storage space, but also aggravates the wear and tear of the storage medium due to frequent read and write operations; third, the data structure change of the new version protocol may not match the access characteristics of the new medium, for example, the nested data structure introduced by the new protocol on the sequentially accessed optimized medium actually leads to performance degradation, and cannot fully exert the performance advantages of the new generation of storage medium. Therefore, there is an urgent need for a database migration method that can uniformly handle protocol adaptation and medium optimization to solve the technical problems of long migration window, storage resource waste and performance mismatch. SUMMARY

[0004] The present application provides a database adaptation integration method, system, device and medium based on Xinchuang storage, which solves the technical problems of long migration window, increased I / O overhead and performance mismatch caused by separation of protocol adaptation and medium optimization in the database cross-generation upgrade process in the related art.

[0005] The application discloses a database adaptation integration method based on Xinchuang storage, comprising the following steps: obtaining source system configuration information and target system configuration information, extracting a source protocol element definition set, a target protocol element definition set and a medium performance characteristic parameter matrix, wherein the medium performance characteristic parameter matrix describes performance indexes of multiple storage media; calculating the similarity between protocol elements in the source protocol element definition set and protocol elements in the target protocol element definition set by using an edit distance algorithm, generating a protocol difference vector based on a matching algorithm for minimizing overall difference, wherein the protocol difference vector quantifies the conversion complexity between the source protocol and the target protocol; comprehensively operating the storage overhead of a data block, an access efficiency score and the medium performance characteristic parameter matrix, generating a three-dimensional affinity tensor of a data block-medium-coding format, wherein the three-dimensional affinity tensor reflects the adaptation degree of a specific data block, a coding format and a storage medium combination; optimizing and solving the three-dimensional affinity tensor by using a genetic algorithm, maximizing the overall affinity value under the constraints of storage capacity and coding format compatibility, generating a version-aware data coding conversion rule set and a medium allocation mapping table; performing streaming data reading, performing online format conversion on each data block in the memory according to the data coding conversion rule set, and generating a data stream conforming to the target protocol specification; directly writing the converted data stream into the corresponding target storage medium according to the medium allocation mapping table, completing the integrated migration of protocol adaptation and medium optimization; wherein each element of the three-dimensional affinity tensor is obtained by weighted combination of the three indexes of storage overhead, access efficiency score and performance matching degree, the storage overhead reflects the ratio relationship between the size of the coded data and the size of the original data, the access efficiency score is calculated based on the value of the corresponding element in the protocol difference vector combined with the data access mode characteristics, and the performance matching degree is calculated by the weighted sum of the corresponding row in the standardized medium performance characteristic parameter matrix.

[0006] Further, the step of calculating the similarity by using the edit distance algorithm and generating the protocol difference vector comprises: for each protocol element in the source protocol element definition set and each protocol element in the target protocol element definition set, calculating the edit distance, wherein the edit distance represents the minimum number of editing operations required to convert one protocol element into another protocol element, and generating a protocol difference matrix; constructing a bipartite graph based on the protocol difference matrix, wherein the left side nodes correspond to the source protocol elements, the right side nodes correspond to the target protocol elements, and the edge weight is the edit distance value; applying the Hungarian algorithm to solve the minimum weight perfect matching, ensuring that each source protocol element is matched to a target protocol element; extracting the edit distance between each source protocol element and the target protocol element matched thereto in the matching result, and generating the protocol difference vector after normalization processing, wherein the normalization processing is realized by dividing each edit distance value by the maximum value of all edit distance values, so that the difference value is mapped to the interval of zero to one.

[0007] Further, the step of generating the three-dimensional affinity tensor comprises: analyzing the access log of the data to be migrated, identifying the access pattern characteristics of each data block, generating an access feature vector, the access feature vector comprising access frequency, sequential access proportion and random access proportion; performing standardization processing on the access frequency, scaling it to the interval of zero to one, and setting the standardized access frequency to zero point five when the access frequencies of all data blocks are the same; based on the protocol difference vector and the access feature vector, calculating the comprehensive access efficiency score of each data block under different encoding formats, which comprehensively considers the protocol compatibility, the support efficiency of encoding formats for sequential access and random access, and the influence of access frequency; for each combination of data block, encoding format and storage medium, the affinity value is calculated by weighted combination of the storage overhead, the comprehensive access efficiency score and the performance matching degree, and the performance matching degree is calculated by weighted summation of the dependency weight of encoding format on each performance parameter of storage medium and the standardized performance parameter value.

[0008] Further, the comprehensive access efficiency score is calculated by: taking the inverse of the element corresponding to the encoding format in the protocol difference vector and adding one to obtain the protocol compatibility component; multiplying the sequential access proportion of the data block by the sequential access support efficiency of the encoding format, and then summing the product of the random access proportion of the data block and the random access support efficiency of the encoding format to obtain the access pattern adaptation component; multiplying the standardized access frequency by the high-frequency access support efficiency of the encoding format to obtain the frequency adaptation component; and performing weighted summation on the protocol compatibility component, the access pattern adaptation component and the frequency adaptation component to obtain the comprehensive access efficiency score.

[0009] Further, the step of optimizing and solving by using the genetic algorithm comprises: defining the optimization target as maximizing the total affinity value of all data blocks, the total affinity value being the sum of the affinity values of each data block under the combination of the encoding format and the storage medium assigned to it; setting a storage capacity constraint to ensure that the sum of the sizes of all data blocks assigned to the same storage medium after encoding does not exceed the product of the physical capacity of the storage medium and the upper limit of the capacity utilization rate; setting an encoding format compatibility constraint to ensure that the encoding format assigned to each data block supports the data type of the data block; setting a migration time window constraint to ensure that the sum of the conversion time and the transmission time of all data blocks does not exceed the allowed maximum migration time window; using integer coding to represent chromosomes, the first half of each chromosome representing the mapping relationship of data blocks to encoding formats, and the second half representing the mapping relationship of data blocks to storage media; iteratively optimizing through selection, crossover and mutation operations, applying a penalty coefficient to individuals that violate the constraint conditions to reduce their fitness values, and outputting the data encoding conversion rule set and the medium allocation mapping table corresponding to the optimal allocation scheme.

[0010] Further, the optimization target adopts a time-sensitive fitness function which comprehensively considers three factors of the overall affinity value, data transmission time and format conversion time:

[0011] The data transmission time is calculated according to the ratio of the size of the encoded data to the effective transmission bandwidth of the storage medium, the effective transmission bandwidth being the minimum of the write bandwidth of the storage medium and the network transmission bandwidth multiplied by the network utilization coefficient;

[0012] The format conversion time is calculated according to the size of the original data, the CPU processing speed and the encoding format conversion complexity coefficient, the CPU processing speed being determined based on the product of the CPU frequency, the number of cores, the number of instructions per clock cycle and the CPU utilization rate;

[0013] The time-sensitive fitness function is constructed by assigning a positive weight to the overall affinity value and a negative weight to the data transmission time and the format conversion time;

[0014] The step of performing streaming data reading and online format conversion includes:

[0015] A streaming data channel is established from the source system to the target system, and a memory buffer is set up for data storage;

[0016] Data blocks are read from the source system in fixed block sizes, and the protocol element structure of the data blocks is parsed;

[0017] The target encoding format corresponding to the current data block is determined according to the data encoding conversion rule set, and a corresponding encoding converter is called to perform format conversion, which reorganizes data fields, adjusts byte order and updates metadata identifiers according to the target protocol specification;

[0018] The converted data blocks are written to the output buffer, and when the buffer reaches a preset threshold, they are batch-transferred to the corresponding storage medium of the target system, avoiding temporary storage of intermediate formats.

[0019] Further, the following steps are included:

[0020] A batch migration execution plan is generated based on the medium allocation mapping table, data blocks with the same encoding format and target storage medium are aggregated into one migration batch, and the system call overhead is reduced through batch processing;

[0021] The actual access performance indicators of the migrated data blocks in the target system are monitored, and after standardization processing, the deviation value is calculated by comparing the expected performance predicted based on the three-dimensional affinity tensor with the actual performance.

[0022] When the deviation value exceeds a preset threshold value, the weight coefficient in the three-dimensional affinity tensor calculation is adjusted, the affinity tensor of the subsequent batch is recalculated, the updated conversion rule set and the allocation mapping table are generated, and dynamic optimization adjustment is realized.

[0023] The application further discloses a computer storage medium, characterized by storing computer readable instructions, which can execute the database adaptation integration method based on the XGStorage when the computer reads the computer readable instructions.

[0024] The application further discloses a computer device for reading the computer readable instructions stored in the computer storage medium.

[0025] The application further discloses a database adaptation integration system based on the XGStorage, which is used for executing the database adaptation integration method based on the XGStorage and comprises a configuration information extraction module, a protocol difference analysis module, an affinity tensor calculation module, an optimization decision module, a streaming conversion module and a data writing module.

[0026] The application introduces a three-dimensional affinity tensor mechanism, uniformly models data block characteristics, encoding format requirements and storage medium performance, enables version adaptation and medium allocation to be completed in the same decision framework, overcomes the problem of multiple data movements caused by the separation of version conversion and medium optimization in the traditional method, solves the technical problems of migration window extension, I / O overhead increase and storage resource waste, and achieves the technical effects of significantly shortening migration time, reducing storage medium wear and fully exerting the performance advantages of a new generation of storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the main flowchart of the database adaptation integration method based on the XGStorage of the application; DETAILED DESCRIPTION

[0028] The embodiment provides a database adaptation integration method based on the XGStorage, which needs to configure the basic environment of a source system and a target system before execution, including a database instance of the source system, a storage medium array of the target system and a network connection channel.

[0029] Step 1: Obtain source system configuration information and target system configuration information, extract protocol element definition set and medium performance characteristic parameter matrix.

[0030] Read the protocol version specification file and data storage format definition from the source system, and parse the source protocol element definition set , wherein represents the th protocol element of the source system; read the new version protocol specification and storage medium configuration file from the target system, and parse the target protocol element definition set and the medium performance characteristic parameter matrix .

[0031] Further, the specific implementation process of protocol element parsing is as follows: first, read the protocol specification file (usually in XML, JSON or IDL format), and extract data type declaration, table structure definition, index configuration and other basic elements through the parser; second, convert each protocol element into a standardized string representation, in the format of "element type: element name: attribute list", for example, TABLE: user_info: id (INT), name (VARCHAR), age (INT); third, build the syntax tree structure of the protocol element, record the dependency relationship and hierarchical relationship between elements, and generate the complete protocol element definition set.

[0032] It should be noted that the medium performance characteristic parameter matrix is a matrix, wherein represents the number of storage medium types, represents the number of performance parameter categories, and the matrix element represents the th performance parameter value of the th storage medium, and the performance parameters include IOPS (input / output operations per second), delay, sequential read / write bandwidth, random read / write bandwidth and other indicators.

[0033] Further, the medium performance characteristic parameter matrix The specific acquisition method is as follows: first, standardized performance test is performed on each storage medium, 4KB random read / write test is performed using the fio tool to obtain the IOPS value, the ioping tool is used to measure the average access delay, and the dd command is used to test the bandwidth value of 1MB sequential read / write and 4KB random read / write respectively; second, each test is performed 10 times to take the average value to eliminate measurement error, and the test time window is set to 60 seconds to ensure the stability of the results; third, the test results are stored in the performance parameter matrix according to rows, each row corresponds to a storage medium, and each column corresponds to a performance index; fourth, the test environment parameters including CPU model, memory size and operating system version are recorded as the basis for verifying the validity of the performance data.

[0034] Since different performance parameters have different dimensions (IOPS is in units of times / second, delay is in units of milliseconds, and bandwidth is in units of MB / s), in order to eliminate the influence of dimensions on subsequent operations, the medium performance characteristic parameter matrix needs to be standardized, and each parameter value is scaled to the interval , and the standardization formula is .

[0035] Further, in order to avoid the case that the denominator is zero in the standardization formula, when , it indicates that all storage media have the same value on the th performance parameter, and at this time is set to ensure the numerical stability of the standardization process.

[0036] Step 2: Calculate the similarity between the new and old protocol elements using the edit distance algorithm to generate the protocol difference vector.

[0037] For each element in the source protocol element set and each element in the target protocol element set , calculate the edit distance to generate the protocol difference matrix , wherein the matrix element ; based on the protocol difference matrix, the protocol difference vector is generated by the matching algorithm of minimizing the total difference, wherein represents the difference degree between the th element of the source protocol and its best matching target protocol element.

[0038] The input of the aforementioned edit distance algorithm is the string representation of two protocol elements, and the output is a non-negative integer value representing the minimum number of editing operations required to convert one protocol element into another, including three basic operations of inserting, deleting and replacing characters.

[0039] The aforementioned matching algorithm for minimizing overall differences includes the following steps: First, based on the protocol difference matrix... The first step is to construct a bipartite graph, where left-side nodes correspond to source protocol elements and right-side nodes correspond to target protocol elements, with edge weights being the edit distance values. The second step is to apply the Hungarian algorithm to find the minimum weight perfect match, ensuring that each source protocol element matches a target protocol element. The third step is to extract the edit distance between each source protocol element and its matched target protocol element from the matching results, normalize the distance, and generate a protocol difference vector. The normalization formula is This maps the difference values ​​to Interval.

[0040] Furthermore, the reasons for choosing between the edit distance algorithm and the Hungarian algorithm are as follows: The edit distance algorithm is suitable for calculating the similarity of protocol elements because protocol elements can be converted into string sequences, and the edit distance can accurately quantify the structural differences between strings. The smaller the value, the more similar the protocol elements are, and the lower the conversion cost. The Hungarian algorithm is suitable for matching problems that minimize overall differences because this problem is essentially finding the minimum weight perfect match in a bipartite graph, and the Hungarian algorithm can... Find the globally optimal solution within the time complexity, ensuring that each source protocol element can match the target protocol element with the minimum conversion cost.

[0041] Furthermore, when all protocol elements match perfectly, it leads to... When the source protocol is fully compatible with the target protocol, the following settings are used: For all This ensures the numerical stability of the protocol difference vector.

[0042] Step 3: Perform matrix operations on the storage overhead, access efficiency score and media performance parameter matrix to generate a three-dimensional affinity tensor of data block-media-encoding format.

[0043] This step generates the three-dimensional affinity tensor through the following calculation process. :

[0044] First, define the data block set. Encoding format set Storage media type set ;

[0045] Then, for each data block Encoding format and storage media Calculate the affinity value based on the combination of factors:

[0046] ;

[0047] in, Represents data block Encoding format The storage overhead is calculated by the ratio of the original size of the data block to the size after encoding; Represents data block Encoding format Access efficiency score based on protocol difference vector Calculate the value of the corresponding element in the middle; Indicates encoding format With storage media The performance matching degree is determined by the standardized medium performance characteristic parameter matrix. The weighted sum of the corresponding rows is calculated. , , For the weighting coefficients, satisfying .

[0048] Furthermore, storage overhead The specific calculation formula is as follows ,in Represents data block The number of raw bytes, Represents data block Encoding format The storage overhead calculation formula, based on the number of bytes after compression, ensures that encoding formats with high compression efficiency achieve lower overhead values. Performance matching. The calculation formula is ,in Indicates encoding format For storage media The performance parameters are assigned weights, the weight values ​​of which are determined based on the I / O access characteristics of the encoding format.

[0049] Furthermore, the specific method for measuring data block size is as follows: By reading data blocks The metadata obtained includes the total number of bytes in the table header information, row data, and index structure; It is obtained through a simulated encoding process, specifically in the following steps: First, extract the data block. The first step is to obtain sample data (either the first 1000 rows or 10% of the total data, whichever is smaller); the second step is to apply an encoding format to the sample data. The third step involves performing the actual encoding conversion; measuring the number of bytes in the encoded sample and calculating the compression ratio. The fourth step is to calculate the encoded size of the complete data block based on the compression ratio. .

[0050] Furthermore, depending on weights The specific calculation method is as follows: first, analyze the I / O access mode of the encoding format , and count the sensitivity of the encoding format to four types of performance parameters, IOPS, delay, sequential bandwidth, and random bandwidth; second, according to the sensitivity, assign a basic weight, for the encoding format sensitive to IOPS, set , for the format sensitive to delay, set , for the format sensitive to sequential bandwidth, set , and for the format sensitive to random bandwidth, set ; third, according to the data access granularity of the encoding format, adjust the weight, increase the IOPS weight of the format with small granularity access (<4KB) by 0.1, and increase the bandwidth weight of the format with large granularity access (>64KB) by 0.1; fourth, normalize to ensure .

[0051] Further, the value range of the weight coefficients , , is , wherein is used to adjust the importance of storage overhead, is used to adjust the importance of access efficiency, is used to adjust the importance of performance matching degree, and the recommended typical value is , , , which indicates that the access efficiency is given priority, and the storage overhead and performance matching degree are considered.

[0052] It should be noted that each element of the three-dimensional affinity tensor reflects the adaptation degree of a specific data block, encoding format, and storage medium combination, and the larger the value is, the better the combination is.

[0053] Further, the theoretical value range of each element of the three-dimensional affinity tensor is , because the three components , , have a standardized value range of , and the weight coefficient , so the actual value range of is constrained in the interval , and a value close to 1 indicates that the combination has the best adaptability.

[0054] In the embodiments of the present application, in order to improve the accuracy of affinity calculation, the following steps are further included on the basis of step 3:

[0055] Step 3.1: Analyze the access log of the data to be migrated, identify the access pattern characteristics of the data block, and generate the access feature vector.

[0056] Extract the access records of each data block in the past time window from the access log of the source system, calculate the access frequency , sequential access ratio , and random access ratio , and generate the access feature vector of the data block .

[0057] To eliminate the dimensional difference between the access frequency (times / time unit) and the access ratio (dimensionless) that affects the subsequent calculation, the access frequency needs to be standardized to scale it to the interval: .

[0058] Further, when the access frequency of all data blocks is the same, it means that the access load of each data block is balanced, and at this time, the is set to for all , ensuring the numerical stability of the standardization processing.

[0059] The aforementioned access pattern feature identification includes the following calculations: access frequency , where is the number of accesses of the data block in the time window ; sequential access ratio , where is the number of times of continuously accessing adjacent data blocks; random access ratio .

[0060] Further, the value range of the time window is , and the recommended value is 8 hours. The time window should cover the typical business cycle to ensure the accuracy of access pattern identification.

[0061] Step 3.2: Based on the protocol difference vector and the access feature vector, calculate the storage overhead and access efficiency score of each data block under different encoding formats.

[0062] Combine the protocol difference vector and the standardized access feature vector , and calculate the comprehensive access efficiency score of the data block using encoding format :

[0063] ;

[0064] wherein, is the protocol difference vector and the encoding format corresponding element, and respectively represent the encoding format support efficiency for sequential access and random access, represent the encoding format support efficiency for high frequency access, , and are weight parameters satisfying .

[0065] Further, the protocol difference vector and the corresponding relationship between the element and the encoding format is established by a format mapping function , wherein , the mapping function returns the index position of the encoding format in the protocol difference vector, and the index position is determined based on the matching result of the protocol element, and is specifically calculated as , wherein represents the protocol signature of the encoding format , represents the protocol signature of the source protocol element .

[0066] Further, the value range of the weight parameters , , is , and the recommended typical value is , , , indicating that the protocol compatibility and the access mode adaptability are equally important, and the influence of the access frequency is relatively small. The value range of the support efficiency parameters , and is , wherein 1 represents complete support, and 0 represents no support, and the specific value is determined according to the technical specification of the encoding format.

[0067] Further, the specific acquisition method of the support efficiency parameters is as follows: first, read the technical specification document of the encoding format , and extract its access mode optimization characteristics; second, calculate the sequential access support efficiency For row storage format, set to 0.9, column storage format set to 0.6, mixed storage format set to 0.8; third step, according to the index mechanism of encoding format calculation random access support efficiency , the format with B+ tree index is set to 0.9, the format with hash index is set to 0.8, and the format without index is set to 0.3; fourth step, according to the cache mechanism of encoding format calculation high frequency access support efficiency , the format with memory cache is set to 0.9, the format with SSD cache is set to 0.7, and the format without cache is set to 0.4.

[0068] Step 4: use genetic algorithm to optimize the solution of affinity tensor, generate version-aware data encoding conversion rule set and medium allocation mapping table.

[0069] The three-dimensional affinity tensor is input as the fitness function, the optimization goal is to maximize the total affinity value of all data blocks, and the constraint conditions include the capacity limit of each storage medium and the compatibility requirement of encoding format; Through the selection, crossover and mutation operation of genetic algorithm, the optimal allocation scheme is obtained by iteration optimization, and the data encoding conversion rule set and medium allocation mapping table are output.

[0070] Further, the mathematical formalization expression of optimization problem is as follows: the optimization objective function is ; the storage capacity constraint is , wherein represents the physical capacity of the storage medium , represents the upper limit of capacity utilization, the value range is , and the recommended value is 0.85; the encoding format compatibility constraint is , wherein is the compatibility judgment function, which returns 1 when the encoding format supports the data type , otherwise returns 0; the migration time window constraint is , wherein represents the maximum allowed migration time window.

[0071] Further, the calculation method of migration time window is as follows: , wherein represents the maximum downtime allowed by the business, in hours, which is determined by business demand analysis, typical values include 2 hours for critical business systems, 8 hours for general business systems, and 24 hours for non-critical business systems; represents the time tolerance coefficient, the value range is , recommended value is 0.8, time tolerance coefficient is to reserve buffer time for migration process to deal with unexpected situations; data type compatibility function The metadata of the data block is parsed to obtain, and the return value includes basic data type identifiers such as integer, floating point, string, and binary.

[0072] Further, the reasons for choosing the genetic algorithm are as follows: first, the three-dimensional affinity tensor optimization problem belongs to the NP-hard multi-objective combinatorial optimization problem, and the search space size is , the traditional gradient descent algorithm cannot handle discrete variables and multiple constraints; second, genetic algorithm is naturally suitable for maximization problem, and through selection operation, high fitness individuals are reserved, which is highly consistent with the goal of maximizing the overall affinity value; third, genetic algorithm has global search capability and can avoid falling into local optimal solution, which is crucial for complex three-dimensional affinity tensor optimization; fourth, genetic algorithm is easy to handle constraints, and through the penalty function mechanism, it can effectively handle storage capacity constraints and format compatibility constraints.

[0073] The aforementioned genetic algorithm coding method uses integer coding, and each chromosome is represented as an integer vector with a length of , where the first gene bits represent the mapping relationship of the data block to the encoding format, and the last gene bits represent the mapping relationship of the data block to the storage medium, and the value range of the gene value is and ; the algorithm input includes three-dimensional affinity tensor , population size, iteration number, crossover probability and mutation probability, and the output is the optimal allocation scheme corresponding to the encoding conversion rule set and medium allocation mapping table; the fitness function calculation formula is , where and represent the encoding format and storage medium allocated to the data block .

[0074] Further, to reflect the role of time dimension in the optimization target, a time-sensitive fitness function is introduced, where represents the time required for the data block to be transmitted to the storage medium using the encoding format , and the calculation formula is ; represents the time required for the data block to be converted to the encoding format , and the calculation formula is ; the weight coefficient , , satisfy Recommended value , , .

[0075] Furthermore, the definitions of the time calculation-related parameters are as follows: Indicates storage medium The effective transmission bandwidth, measured in MB / s, is determined by the media performance characteristic parameter matrix. The bandwidth parameter is obtained from the formula, which is: ,in For storage medium Write bandwidth, For network transmission bandwidth, This is the network utilization coefficient, with a value range of [value range missing]. The recommended value is 0.8; This indicates the CPU processing speed for data conversion and processing, measured in MB / s. It is obtained through system configuration and calculated using the following formula: ,in CPU frequency, Number of CPU cores Number of instructions per clock cycle For CPU utilization, a value of 0.7 is recommended. Indicates encoding format The transformation complexity coefficient is dimensionless and its value ranges from 0 to 1. The simple format conversion (such as type conversion) is set to 1.0, the compressed format conversion is set to 2.5, the encrypted format conversion is set to 4.0, and the comprehensive conversion is set to 5.0.

[0076] Furthermore, the specific implementation of the genetic algorithm is as follows: the selection operation adopts the roulette wheel selection method, based on the fitness value. Calculate the probability of an individual being selected ,in The population size is specified; the crossover operation uses a single-point crossover method, randomly selecting a crossover point to split the two parent chromosomes and exchange the latter half to generate offspring; the mutation operation uses a uniform mutation method, randomly replacing each gene position with other values ​​within its range with a mutation probability; the constraint handling mechanism ensures that the total allocated capacity of each storage medium does not exceed its physical capacity limit, and a penalty coefficient of 0.5 is applied to the fitness value when the constraint is violated.

[0077] Furthermore, the parameter constraints for the genetic algorithm are as follows: the population size ranges from [value missing]. The recommended value is 100; the number of iterations can range from [value missing]. The recommended value is 300; the crossover probability range is [value missing]. , the recommended value is 0.8; the variation probability is in the range of , the recommended value is 0.05, and these parameter settings can ensure the convergence of the algorithm and the quality of the solution.

[0078] Step 5: Perform streaming data reading, apply the coding conversion rule set for online format conversion, and output the converted data stream.

[0079] According to the data coding conversion rule set , for each data block , read the original data stream from the source system, apply the corresponding coding format in memory for real-time conversion, generate a data stream conforming to the target protocol specification, and directly transmit to the target system to avoid temporary storage of intermediate formats.

[0080] Further, the specific implementation process of online format conversion is as follows: first, establish a streaming data channel from the source system to the target system, set the memory buffer size to 64MB for data staging; second, read the data block content from the source system by fixed block size (default 8KB), parse the protocol element structure of the data block; third, determine the corresponding target coding format of the data block according to the conversion rule set , call the corresponding coding converter for format conversion, and the converter reorganizes the data fields according to the target protocol specification, adjusts the byte sequence, and updates the metadata identifier; fourth, write the converted data block to the output buffer, and when the buffer reaches the threshold, batch transfer to the corresponding storage medium of the target system.

[0081] Step 6: Write the converted data stream to the corresponding target storage medium according to the media allocation mapping table, and record the migration progress marker.

[0082] According to the media allocation mapping table , write the converted data stream output by step 5 to the specified storage medium , record the migration status and timestamp of each data block in the metadata database, and generate a migration progress report.

[0083] In the embodiments of the present application, in order to improve the migration efficiency, the following steps are further included based on steps 5 and 6:

[0084] Step 7: Generate a batch migration execution plan based on the mapping table, and aggregate data blocks with the same conversion rule and target medium into a migration batch.

[0085] Scan the data coding conversion rule set and the media allocation mapping table , aggregate data blocks with the same coding format and target media The data blocks of the target media are aggregated into a batch, a batch migration execution plan is generated, and the system call overhead is reduced through batch processing.

[0086] Step 8: Monitor the actual access performance indicators after migration, calculate the deviation value from the expected performance, and dynamically adjust the encoding rules and media allocation strategy of the subsequent batch.

[0087] The actual access performance indicators of the migrated data blocks in the new system are collected, including response time, throughput, etc. Since different performance indicators have different dimensions (response time unit: millisecond, throughput unit: MB / s), the performance indicators need to be standardized before being compared with the expected performance based on the affinity tensor prediction to calculate the deviation value ; when the deviation value exceeds the threshold, the weight coefficient 、 、 is adjusted, and the affinity tensor of the subsequent batch is recalculated to generate updated conversion rules and allocation strategies.

[0088] Further, the deviation value threshold is in the range of , and the recommended value is 0.2. The deviation value threshold represents the maximum relative deviation allowed between the actual performance and the expected performance. When , it means that the prediction accuracy of the current allocation strategy is insufficient and needs to be dynamically adjusted. The adjustment range of the weight coefficient is set to to ensure the gradualness and stability of the strategy adjustment.

[0089] The embodiment introduces a three-dimensional affinity tensor mechanism to unify the modeling of data block characteristics, encoding format requirements, and storage medium performance, enabling version adaptation and media allocation to be completed under the same decision framework, overcoming the multiple data movement problems caused by the separation of version conversion and media optimization in traditional methods, and thus solving the technical problems of migration window extension and I / O overhead increase.

[0090] Specifically, the protocol difference vector quantifies the conversion complexity between new and old version protocol elements, providing accurate decision basis for encoding format selection; the streaming online conversion mechanism enables data to be converted in format while being read and directly written to the target medium, eliminating the additional I / O operations caused by temporary storage of intermediate formats, and thus solving the problems of storage space waste and accelerated media wear.

[0091] In addition, the three-dimensional affinity tensor comprehensively considers the data access mode, protocol compatibility and medium performance characteristics in three dimensions, and obtains the globally optimal allocation scheme through genetic algorithm optimization solution, ensures that the data structure of the new version protocol can match the access characteristics of the new medium, fully plays the performance advantages of the new generation of storage medium, and solves the performance degradation problem caused by the mismatch between protocol structure and medium characteristics.

[0092] A city commercial bank needs to upgrade its core business system from Oracle 11g database and SATA SSD storage to MySQL 8.0 database and NVMe SSD storage. The upgrade involves three core data tables: customer information table (1 million records, average 0.8 KB per record), transaction record table (5 million records, average 1.2 KB per record), and account balance table (2 million records, average 0.5 KB per record). The source system uses SATA SSD array (capacity 2TB, IOPS 8000, average delay 0.5ms), and the target system is configured with NVMe SSD array (capacity 4TB, IOPS 50000, average delay 0.1ms) and traditional HDD array (capacity 10TB, IOPS 200, average delay 5ms) as cold data storage.

[0093] Step 1: Source system and target system configuration information extraction

[0094] The protocol element definition set extracted from the Oracle 11g system contains the following elements, and the target protocol element definition set is extracted from the MySQL 8.0 system, and the storage medium performance parameters are obtained:

[0095] Table 1 Source protocol element definition set:

[0096]

[0097] Table 2 Target protocol element definition set:

[0098]

[0099] Table 3 Medium performance characteristic parameter matrix Before standardization:

[0100]

[0101] Table 4 Standardized medium performance characteristic parameter matrix :

[0102]

[0103] Step 2: Protocol difference calculation

[0104] Calculate the edit distance between each protocol element and generate a protocol difference vector:

[0105] Table 5 Protocol difference matrix :

[0106]

[0107] After solving the minimum weight perfect matching by the Hungarian algorithm, the protocol difference vector is obtained:

[0108] ;

[0109] Where , , , , .

[0110] Step 3: Three-dimensional affinity tensor calculation

[0111] For three data blocks ( : customer information table, : transaction record table, : account balance table) and two encoding formats ( : row storage format, : column storage format), calculate the access feature vector and affinity value:

[0112] Table 6 Data block access feature vector:

[0113]

[0114] Mapping relationship between encoding format and protocol difference vector:

[0115] According to the format mapping function ,

[0116] Table 7 Calculate the similarity between the protocol signature of the encoding format and the source protocol element:

[0117]

[0118] Table 8 Storage overhead Matrix:

[0119]

[0120] Table 9 Encoding format support efficiency parameter:

[0121]

[0122] Access efficiency score Matrix calculation (weight parameter , , ):

[0123] For (row storage), :

[0124]

[0125] For (column storage), :

[0126]

[0127] Table 10 Access efficiency score Matrix:

[0128]

[0129] Performance matching degree Calculation:

[0130] For NVMe SSD medium (all standardized performance parameters are 1.0), the performance matching degrees of various encoding formats are ; for HDD medium (all standardized performance parameters are 0), the performance matching degrees of various encoding formats are .

[0131] Three-dimensional affinity tensor Calculation (weight coefficient , , ):

[0132] According to the formula :

[0133] Affinity value under NVMe SSD medium:

[0134] For data block:

[0135]

[0136] For data block:

[0137]

[0138] For data block:

[0139]

[0140] Table 11 Affinity value matrix under NVMe SSD medium:

[0141]

[0142] Affinity values under HDD media:

[0143] Since , the affinity value is determined only by the storage overhead and access efficiency score:

[0144] For data blocks:

[0145]

[0146] For data blocks:

[0147]

[0148] For data blocks:

[0149]

[0150] Table 12 Affinity value matrix under NVMe SSD media:

[0151]

[0152] Step 4: Genetic algorithm optimization results

[0153] Based on the three-dimensional affinity tensor, set the genetic algorithm parameters (population size 100, iteration number 300, crossover probability 0.8, mutation probability 0.05), consider the storage capacity constraint, and get the optimal allocation scheme:

[0154] Table 13 Data encoding conversion rule set :

[0155]

[0156] Table 14 Media allocation mapping table :

[0157]

[0158] Fitness function optimization process:

[0159] Based on the updated three-dimensional affinity tensor, the fitness value of the optimal allocation scheme is :

[0160] Table 15 Optimal allocation scheme:

[0161]

[0162] Final migration result summary

[0163] Table 16 System configuration and performance comparison after migration is completed:

[0164]

Claims

1. A database adaptation integration method based on Xinchuang storage, characterized in that, The method comprises the following steps: Obtaining source system configuration information and target system configuration information, extracting a source protocol element definition set, a target protocol element definition set, and a medium performance characteristic parameter matrix, the medium performance characteristic parameter matrix describing performance indicators of multiple storage media; Using an edit distance algorithm to calculate the similarity between protocol elements in the source protocol element definition set and protocol elements in the target protocol element definition set, and generating a protocol difference vector based on a matching algorithm for minimizing overall difference, the protocol difference vector quantifying the conversion complexity between the source protocol and the target protocol; Comprehensively operating the storage overhead, access efficiency score, and the medium performance characteristic parameter matrix of the data block to generate a data block-medium-encoding format three-dimensional affinity tensor, the three-dimensional affinity tensor reflecting the adaptation degree of a specific data block, encoding format, and storage medium combination; Using a genetic algorithm to optimize and solve the three-dimensional affinity tensor, maximizing the overall affinity value under the constraints of storage capacity and encoding format compatibility to generate a version-aware data encoding conversion rule set and a medium allocation mapping table; Performing streaming data reading, performing online format conversion on each data block in the memory according to the data encoding conversion rule set to generate a data stream conforming to the target protocol specification; Writing the converted data stream into the corresponding target storage medium according to the medium allocation mapping table to complete the integrated migration of protocol adaptation and medium optimization; Wherein each element of the three-dimensional affinity tensor is calculated by weighted combination of storage overhead, access efficiency score, and performance matching degree, the storage overhead reflects the ratio relationship between the size of the encoded data and the size of the original data, the access efficiency score is calculated based on the value of the corresponding element in the protocol difference vector combined with the data access mode characteristics, and the performance matching degree is calculated by the weighted sum of the corresponding row in the standardized medium performance characteristic parameter matrix.

2. The database adaptation integration method based on Xinchuang storage according to claim 1, characterized in that, The step of calculating the similarity using the edit distance algorithm and generating the protocol difference vector comprises: For each protocol element in the source protocol element definition set and each protocol element in the target protocol element definition set, calculate the edit distance, the edit distance representing the minimum number of editing operations required to convert one protocol element into another protocol element, and generate a protocol difference matrix; Based on the protocol difference matrix, a bipartite graph is constructed, the left side nodes corresponding to the source protocol elements and the right side nodes corresponding to the target protocol elements, and the edge weight being the edit distance value; Applying the Hungarian algorithm to solve the minimum weight perfect matching to ensure that each source protocol element is matched to a target protocol element; Extracting the edit distance between each source protocol element and the target protocol element matched in the matching result, and generating the protocol difference vector after normalization processing, the normalization processing being realized by dividing each edit distance value by the maximum value of all edit distance values, so that the difference value is mapped to the interval of zero to one.

3. The database adaptation integration method based on the Xinchuang storage according to claim 1, characterized in that, The step of generating the three-dimensional affinity tensor comprises: The access log of the data to be migrated is analyzed to identify access pattern features of each data block, and an access feature vector is generated, the access feature vector including access frequency, sequential access proportion, and random access proportion; The access frequency is standardized to scale it to a range of zero to one, and when the access frequencies of all data blocks are the same, the standardized access frequency is set to 0.5; Based on the protocol difference vector and the access feature vector, a comprehensive access efficiency score of each data block in different encoding formats is calculated, which comprehensively considers the protocol compatibility, the support efficiency of encoding formats for sequential access and random access, and the influence of access frequency; For each combination of data block, encoding format, and storage medium, an affinity value is calculated by weighted combination of the storage overhead, the comprehensive access efficiency score, and the performance matching degree, which is calculated by weighted summation of the encoding format-dependent weights of various performance parameters of the storage medium and the standardized performance parameter values.

4. The database adaptation integration method based on the Xinchuang storage according to claim 3, characterized in that, The comprehensive access efficiency score is calculated by: Taking the inverse of the element corresponding to the encoding format in the protocol difference vector and adding 1 to obtain the protocol compatibility component; Multiplying the sequential access proportion of the data block by the sequential access support efficiency of the encoding format, and then summing the product of the random access proportion of the data block and the random access support efficiency of the encoding format to obtain the access pattern adaptation component; Multiplying the standardized access frequency by the high-frequency access support efficiency of the encoding format to obtain the frequency adaptation component; Weighted summation of the protocol compatibility component, the access pattern adaptation component, and the frequency adaptation component obtains the comprehensive access efficiency score.

5. The database adaptation integration method based on Xinchuang storage according to claim 1, characterized in that, The step of optimizing and solving by using a genetic algorithm includes: Defining the optimization target as maximizing the total affinity value of all data blocks, which is the sum of the affinity values of each data block under the combination of the encoding format and the storage medium assigned to it; Setting a storage capacity constraint to ensure that the sum of the sizes of all data blocks assigned to the same storage medium after encoding does not exceed the product of the physical capacity of the storage medium and the upper limit of the capacity utilization rate; Setting an encoding format compatibility constraint to ensure that the encoding format assigned to each data block supports the data type of the data block; Setting a migration time window constraint to ensure that the sum of the conversion time and the transmission time of all data blocks does not exceed the maximum allowed migration time window; Using integer coding to represent chromosomes, with the first half of each chromosome representing the mapping relationship of data blocks to encoding formats and the second half representing the mapping relationship of data blocks to storage media; Through selection, crossover, and mutation operations, individuals that violate the constraint conditions are penalized by reducing their fitness values, and the data encoding conversion rule set and the medium allocation mapping table corresponding to the optimal allocation scheme are output.

6. The database adaptation integration method based on the Xinchuang storage according to claim 5, characterized in that, The optimization target uses a time-sensitive fitness function that considers the total affinity value, data transmission time, and format conversion time: The data transmission time is calculated according to the ratio of the size of the encoded data to the effective transmission bandwidth of the storage medium, the effective transmission bandwidth being the minimum of the write bandwidth of the storage medium and the network transmission bandwidth multiplied by a network utilization coefficient; The format conversion time is calculated according to the size of the original data, the CPU processing speed and the encoding format conversion complexity coefficient, the CPU processing speed being determined based on the product of the CPU frequency, the number of cores, the number of instructions per clock cycle and the CPU utilization rate; The time-sensitive fitness function is constructed by assigning a positive weight to the overall affinity value and a negative weight to the data transmission time and the format conversion time; The step of performing streaming data reading and online format conversion comprises: A streaming data channel is established from the source system to the target system, and a memory buffer is set up for data storage; Data blocks are read from the source system in fixed block sizes, and the protocol element structure of the data blocks is parsed; The target encoding format corresponding to the current data block is determined according to the data encoding conversion rule set, and a corresponding encoding converter is called to perform format conversion, which reorganizes the data fields, adjusts the byte sequence and updates the metadata identifier according to the target protocol specification; The converted data blocks are written to the output buffer, and when the buffer reaches a preset threshold, they are batch-transferred to the corresponding storage medium of the target system, avoiding temporary storage of intermediate formats.

7. The database adaptation integration method based on the Xinchuang storage according to claim 1, characterized in that, Further comprising the following steps: A batch migration execution plan is generated based on the medium allocation mapping table, data blocks with the same encoding format and target storage medium are aggregated into one migration batch, and the system call overhead is reduced through batch processing; The actual access performance indicators of the migrated data blocks in the target system are monitored, and after standardization processing, the performance indicators are compared with the expected performance predicted based on the three-dimensional affinity tensor, and a deviation value is calculated; When the deviation value exceeds a preset threshold, the weight coefficient in the calculation of the three-dimensional affinity tensor is adjusted, the affinity tensor of the subsequent batches is recalculated, an updated conversion rule set and allocation mapping table are generated, and dynamic optimization and adjustment are realized.

8. A computer storage medium, characterized in that The computer readable instructions stored in the computer storage medium can be read by a computer to perform the database adaptation integration method based on the Xinhua storage according to any one of claims 1-7.

9. A computer device, comprising: A computer storage medium for storing computer readable instructions as claimed in claim 8.

10. A database adaptation integration system based on Xinchuang storage, configured to perform the database adaptation integration method based on Xinchuang storage in any of claims 1 to 7. Comprise: A configuration information extraction module for extracting protocol element definition sets and medium performance characteristic parameter matrices from source systems and target systems; A protocol difference analysis module for generating a protocol difference vector based on an edit distance algorithm and a matching algorithm; An affinity tensor calculation module for generating a three-dimensional affinity tensor by integrating storage overhead, access efficiency score and medium performance parameters; An optimization decision module for generating a data encoding conversion rule set and a medium allocation mapping table by genetic algorithm optimization solution; A streaming conversion module for performing online format conversion and generating a data stream conforming to the target protocol specification; A data writing module for writing the converted data stream to the corresponding target storage medium according to the medium allocation mapping table. A configuration information extraction module for extracting protocol element definition sets and medium performance characteristic parameter matrices from source systems and target systems; A protocol difference analysis module for generating a protocol difference vector based on an edit distance algorithm and a matching algorithm; An affinity tensor calculation module for generating a three-dimensional affinity tensor by integrating storage overhead, access efficiency score and medium performance parameters; An optimization decision module for generating a data encoding conversion rule set and a medium allocation mapping table by genetic algorithm optimization solution; A streaming conversion module for performing online format conversion and generating a data stream conforming to the target protocol specification; A data writing module for writing the converted data stream to the corresponding target storage medium according to the medium allocation mapping table.

Images