Big data distributed storage method and system based on cloud computing

Abstract

The invention relates to the field of quantum storage and cloud computing fusion, and discloses a big data distributed storage method and system based on cloud computing, and the method comprises the steps: revealing a data intrinsic geometric structure through manifold embedding, and generating a manifold coordinate; secondly, constructing an Ising physical model based on the geometric structure, and solving a model ground state through quantum annealing to determine an optimal data dynamic fragmentation scheme; thirdly, homomorphic encryption is carried out on the data fragments, a Lie group optimization algorithm is used for solving on a special orthogonal group manifold, and a resource mapping matrix considering safety and efficiency is generated; and finally, instructing a fragmentation scheme, constructing a transmission tensor load fusing manifold information, data content and security mapping for each fragment, and optimizing transmission configuration. According to the method, multiple frontier technologies are fused, dynamic fragmentation of data structure sensing and safe and efficient resource mapping are achieved, and the overall performance and safety of the system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of the integration of quantum storage and cloud computing, and specifically to a big data distributed storage method and system based on cloud computing. Background Art

[0002] With the rapid development of quantum computing technology, quantum storage systems face the dual challenges of explosive growth in data dimensions and large-scale access to heterogeneous devices. Traditional storage architectures typically use Euclidean space clustering algorithms for resource allocation when processing high-dimensional unstructured data. However, such methods struggle to capture the inherent manifold structure of the data, leading to a mismatch between storage resources and the spatial geometry of data distribution, resulting in storage fragmentation and accumulated access delays. In quantum computing environments, in particular, the entanglement between quantum bits causes the data correlation dimension to grow exponentially, and traditional sharding strategies based on scalar distance metrics are no longer able to meet the storage requirements of quantum state data.

[0003] Existing encrypted storage solutions often combine homomorphic encryption with static resource mapping. While this approach ensures data privacy, the fixed-topology resource allocation model struggles to adapt to dynamically changing storage loads. In distributed quantum storage scenarios, the quantum properties of storage nodes differ fundamentally from those of classical hardware. Traditional scheduling algorithms, lacking mathematical modeling of orthogonal group space constraints, lead to dimensionality collapse and orthogonality loss during resource mapping, severely limiting the scalability and security of storage systems.

[0004] Current cross-modal data transmission mechanisms generally employ an arbitration strategy combining time-division multiplexing and priority queues, but this fails to effectively resolve the spatiotemporal conflicts between quantum instruction streams and classical data packets. The real-time control instructions generated by quantum computing processes are highly time-sensitive and interdependent, while the statistically multiplexed transmission model of classical storage systems lacks the ability to model the spatiotemporal characteristics of quantum states. This results in low bus bandwidth utilization and increased transmission jitter, becoming a key bottleneck restricting the performance of quantum storage systems.

[0005] Therefore, the present invention proposes a big data distributed storage method and system based on cloud computing to address the deficiencies of the existing technology. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a big data distributed storage method and system based on cloud computing. Through the integration of manifold space modeling, quantum dynamic sharding, Lie group encryption scheduling and cross-modal collaborative transmission, it effectively solves the three core problems of high-dimensional data storage mismatch, dynamic resource scheduling imbalance and hybrid data transmission conflict.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cloud computing-based big data distributed storage method and system, comprising the following steps:

[0008] Step S1: perform manifold embedding on the original data block to generate a manifold space coordinate matrix;

[0009] Step S2: constructing a quantum Ising model based on the geometric characteristics of the manifold space coordinate matrix, solving the spin configuration through a quantum annealing algorithm, and dynamically dividing the data into slices;

[0010] Step S3: Based on the result of the data sharding, storage resources are allocated in the orthogonal group space by using a Lie group optimization algorithm to generate a resource mapping matrix;

[0011] Step S4: According to the resource mapping matrix, the data slices and quantum instructions are transmitted to the target storage node in a coordinated manner through the cross-modal bus protocol.

[0012] Preferably, the step S1 includes

[0013] The step S1 comprises:

[0014] S1.1: For each data block d in N data blocks i , where i = 1, ..., N, extract its m-dimensional feature vector f i =[φ1(d i ),...,φ m (d i )] T , where φ m (·) represents the mth predefined feature extraction function, which is used to capture the key attributes of the original data block;

[0015] S1.2: Construct a similarity matrix based on the feature vectors of each extracted data block Its elements are expressed by the formula Calculated, where ||f i -f j ||2 is the eigenvector f i With f j The Euclidean distance between them is used to quantify the similarity between data blocks, N is the total number of data blocks, and σ is the kernel width parameter, which adjusts the rate at which similarity decays with distance;

[0016] S1.3: Calculate the angle matrix The angle matrix The elements on the diagonal of Represents data block d i The total similarity of the angle matrix All non-diagonal elements of are zero, and the similarity matrix W and degree matrix D are used to construct the transfer matrix P = D -1 W, the transfer matrix is the probability transfer of random walks between data blocks;

[0017] S1.4: Perform eigendecomposition on the transfer matrix P to obtain its first k largest eigenvalues {λ1,...,λ k}, where, and the set of eigenvectors corresponding to the maximum eigenvalue Among them, the maximum eigenvalue λ1≥λ2≥....≥λ k >0, reflects the importance of the data manifold in the corresponding direction, corresponds to the eigenvalue λ ι The lth eigenvector of represents a main direction of the data in the manifold space;

[0018] Based on the maximum eigenvalue and each component in the eigenvector set, for each data block d i Construct its k-dimensional coordinates in low-dimensional manifold space The k-dimensional coordinate m i The projections of the data blocks in each main direction are combined and weighted by the eigenvalues, so that the manifold coordinates of all data blocks are combined to form the manifold space coordinate matrix The coordinate matrix The i-th behavior A low-dimensional manifold embedding representation of the original high-dimensional data block is achieved.

[0019] Preferably, the value of the kernel width parameter σ used to calculate the similarity matrix W in step S1.2 is determined by calculating the average of the squares of the Euclidean distances between the eigenvectors of all pairs of data blocks. The specific calculation formula is as follows:

[0020]

[0021] Wherein, N is the total number of data blocks in step S1.1, f i and f j are the feature vectors extracted from the i-th and j-th data blocks in step S1.1, represents the eigenvector f i With f j The square of the Euclidean distance between them.

[0022] Preferably, in order to realize dynamic quantum sharding based on the geometric characteristics of the data manifold, the step S2 specifically includes:

[0023] S2.1: Based on the manifold space coordinate matrix M obtained in step S1, calculate the Riemannian metric tensor g that describes the local intrinsic geometric properties of the data manifoldμv , where μ and ν are indices of the local coordinate system of the manifold;

[0024] S2.2: According to the Riemann metric tensor g μv and the manifold space coordinate matrix M, for any pair of data blocks d i and d j Derivation of the coupling coefficient J in the lsing model ij , the coupling coefficient J ij Quantitatively describes the data block d i and d j geometric correlation strength or interaction energy on the data manifold;

[0025] S2.3: Construct Hamiltonian H of target lsing model = -∑ i<j J ij σ i σ j -∑ i h i σ i , where σ i Assigned to data block d i The spin variable, σ i The value of indicates the shard to which the data block belongs, h i is applied to the spin σ i External local field parameters;

[0026] The Hamiltonian is optimized and solved to obtain a set of optimal spin configurations The optimal spin configuration The dynamic sharding results of all data blocks are determined.

[0027] Preferably, in step S2.3, the step of optimizing and solving the Hamiltonian includes the step of solving the optimal spin configuration of the Ising model Hamiltonian using quantum annealing technology, specifically:

[0028] a) Initialize the system as a system containing N qubits, where each qubit corresponds to the spin variable σ of a data block i , its initial quantum state |ψ(0)> is all 2 N Uniform re-addition of possible computational ground states |s>: This initial state is the ground state of the initial Hamiltonian H0;

[0029] b) The state of the quantum system |ψ(t)> evolves with time according to the time-dependent Schrödinger equation: Among them, Planck constant h is taken as 1;

[0030] c) The time-dependent Hamiltonian H(t) slowly and continuously adiabatically evolves from the initial Hamiltonian H0 to the final target Hamiltonian H f ;

[0031] The final target Hamiltonian H f That is the Ising model Hamiltonian of the optimal data sharding scheme after ground state encoding As a result, the system is in the final target Hamiltonian H with high probability at the end of the evolution f The ground state of the final target Hamiltonian H f The optimal spin configuration is obtained by measuring the ground state

[0032] Preferably, in order to achieve secure, optimized, and structure-preserving mapping of encrypted data shards to storage resources, step S3 includes:

[0033] S3.1: Constructing an Aggregate Algebraic Space for Describing Storage Resource Transformation Characteristics Among them, so(n l ) is n l ×n l The special orthogonal group SO(n l ), where L is the number of aggregated Lie algebra components;

[0034] S3.2: Slice the data obtained in step S2 p Apply the fully homomorphic encryption algorithm to encrypt and obtain ciphertext data fragments that can be calculated in the encrypted state The encryption process can be expressed as Enc(x) = (A, b), and b = As + e + x (modq), where x is the plaintext unit, A is the public matrix, s is the private key, e is the error term, and q is the modulus;

[0035] S3.3: Construct a cost matrix in the encryption domain Among them, α p is the data shard weight, For encrypted data sharding, r j is a resource request vector and solves the optimization problem min U∈SO(n) Tr(U T CU) obtaining an optimal n×n special orthogonal resource mapping matrix U, wherein the n×n special orthogonal resource mapping matrix U is used to align encrypted data slices with storage resources;

[0036] S3.4: Based on the optimal n×n special orthogonal resource mapping matrix U obtained by the optimization solution, generate the final n×n resource mapping matrix D, the final n×n resource mapping matrix D is an element of the special orthogonal group SO(n) and satisfies the orthogonal constraint DT D=I n , where I n is the identity matrix and can be expressed as the exponential mapping from the Lie algebra so(n) to the Lie group SO(n) Among them, A k is the antisymmetric basis matrix of so(n), θ k Parameters determined for optimization.

[0037] Preferably, the resource mapping matrix D in step S3.4 is represented by a set of scalar parameters and scalar parameters The corresponding n×n real antisymmetric matrix of the predefined Lie algebra so(n) When parameterization is generated, step S3.3 is used to solve the optimization problem min U Tr(U T The rules for iteratively updating the resource mapping matrix by CU include the following operations:

[0038] In the i-th iteration, if the parameter Θ k To make adjustments, the resource mapping matrix is changed from the current state U i Update to the next state U i+1 , where k is a parameter index, 1≤k≤P, and the resource status mapping formula is:

[0039] U i+1 =exp(-ηA (k) )U i ;

[0040] in,

[0041] U i ∈SO(n) is the n×n special orthogonal resource mapping matrix at the beginning of the i-th iteration;

[0042] U i+1 ∈SO(n) is the updated n×n special orthogonal resource mapping matrix after this parameter adjustment;

[0043] η(eta) is a positive scalar learning rate that controls the iteration step size;

[0044] exp(·) is the matrix exponential function;

[0045] A (k) is a parameter Θ k The n×n real antisymmetric matrix A is constructed based on the gradient information of the Lie algebra so(n). (k) The specific calculation steps are as follows:

[0046] First, calculate the cost function In the current resource mapping matrix U i When the kth scalar parameter Θ k The partial derivative value is recorded as the scalar gradient value g k , Where C is the cost matrix;

[0047] Then, using the scalar gradient value g k and the scalar parameter Θ k The corresponding Lie algebra basis matrix B k To construct the n×n real antisymmetric matrix A (k) , A (k) =g k B k .

[0048] Preferably, in order to construct a cross-modal transmission channel based on tensor manifold encapsulation, step S4 further includes:

[0049] S4.1: The optimal spin configuration obtained by quantum annealing in step S2.3 Encoded as a quantum instruction sequence Q cmd , the encoding is a quantum instruction sequence Q cmd The quantum tag QTAG corresponding to each data block i Aggregated, expressed as Where N is the total number of data blocks, is the final spin state of the i-th data block, Represents the aggregation operation, each quantum tag QTAG i The data structure is defined as:

[0050] QTAG i =[opcode(8),qubit a ddr(16),spin v al(32)];

[0051] in,

[0052] opcode(8) is an 8-bit operation code field used to indicate the type of the tag or subsequent operation; qubit a ddr(16) is a 16-bit quantum bit address field, which is used to uniquely identify the quantum bit corresponding to the i-th data block;

[0053] spin v al(32) is a 32-bit field used to store or represent the final spin state of the i-th data block

[0054] S4.2: For the optimal spin configuration Determine the data block index set S for each data shard p , first construct the tensor vector representation vec(S p ), the tensor product of the manifold coordinates of all data blocks in each data slice and the data block itself is accumulated through the tensor accumulation model. The formula of the tensor accumulation model is:

[0055]

[0056] in,

[0057] m j is the manifold coordinate vector of the j-th data block;

[0058] d j is the jth original data block or its predetermined representation;

[0059] Represents the tensor product operation;

[0060] Then, using the transpose D of the resource mapping matrix D T , combined with the manifold eigenvector The generated manifold basis space tensor e φ , the tensor quantization vector is represented by vec(S p ) is transformed and encapsulated to construct the transmission tensor load P tensor , in, Represents matrix-vector multiplication or generalized linear transformation.

[0061] Preferably, a big data distributed storage system based on cloud computing includes:

[0062] A data manifold feature extraction module is used to extract feature vectors from data blocks, calculate the similarity matrix between data blocks, and perform feature decomposition on the similarity matrix to obtain the manifold coordinates and manifold feature vectors of the data manifold;

[0063] A dynamic quantum sharding module is used to construct an Ising physical model based on the geometric characteristics of the data manifold and solve the ground state of the model through quantum annealing technology to determine the optimal solution for allocating data blocks to different data shards;

[0064] a secure resource mapping module for homomorphically encrypting the data shards and constructing and solving an optimization problem defined on a special orthogonal group manifold to generate an optimal special orthogonal mapping matrix for securely mapping the encrypted data shards to storage resources;

[0065] The transmission encapsulation module is used to encode the optimal sharding scheme into an instruction sequence and construct a transmission tensor load for each data shard that encapsulates manifold geometry information, data content and is transformed by the optimal special orthogonal mapping matrix.

[0066] The present invention provides a method and system for distributed storage of big data based on cloud computing. It has the following beneficial effects:

[0067] 1. This paper constructs a dynamic data organization architecture optimized for quantum storage through the deep coupling of manifold embedding and quantum sharding. By revealing the intrinsic manifold structure of high-dimensional data based on diffusion mapping and combining it with the spin coupling effect of the quantum Ising model, it overcomes the bottleneck of traditional storage systems in processing unstructured data. It achieves adaptive matching of storage resources with the geometric characteristics of data distribution, significantly improving the quantum storage system's ability to handle complex data patterns.

[0068] 2. This invention innovatively designs a Lie group space encryption scheduling mechanism to achieve secure and efficient dynamic allocation of storage resources under orthogonal group constraints. Lattice-based fully homomorphic encryption ensures sharded data privacy, while Lie algebra parameterization techniques maintain the orthogonality of resource mapping matrices. This effectively resolves the conflict between data security and system scalability in distributed storage, providing an optimized scheduling solution for quantum storage environments that combines cryptographic security with topological structure preservation.

[0069] 3. This paper proposes a cross-modal spatiotemporal collaborative transmission model, achieving efficient transmission of quantum-classical hybrid data through tensor manifold encapsulation and variational temporal arbitration. Based on a geometric consistency-preserving mechanism for manifold basis vectors and an intelligent bandwidth allocation algorithm based on a Lagrangian dynamics model, a bus transmission system with self-feedback regulation is constructed. This overcomes the challenges of protocol differences and transmission conflicts between heterogeneous quantum storage devices, forming an intelligent storage infrastructure for future quantum computing architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a big data distributed storage method based on cloud computing;

[0071] Figure 2 This is an architecture diagram of a big data distributed storage system based on cloud computing. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] Please see the attached Figure 1 The embodiment of the present invention provides a big data distributed storage method based on cloud computing. The following describes each step of the method of the present invention in detail.

[0074] Step S1: Perform manifold embedding on the original data block to generate a manifold space coordinate matrix.

[0075] In this embodiment of a cloud computing-based distributed big data storage method, the core task of the first step (S1) is to reveal the nonlinear intrinsic geometric structure hidden within the original data block set and express this structure in a numerical form. The final output of this step is a manifold space coordinate matrix, which provides the key geometric basis for subsequent dynamic quantum sharding.

[0076] Specifically, for a data set containing multiple original data blocks, the present invention first performs a i Using the preset feature extraction algorithm, it is converted into a multidimensional feature vector f that can describe its core characteristics i These eigenvectors constitute the numerical representation of the original data set and are the basis for subsequent geometric analysis.

[0077] After obtaining the eigenvectors of all data blocks, the present invention further constructs a similarity matrix W to quantify the local similarity between any two data blocks in the feature space. Each element W of the similarity matrix W is ij It is calculated by the Gaussian kernel function, and its calculation formula is:

[0078]

[0079] Among them, W ij represents the similarity between the i-th and j-th data blocks; f i and f j are the feature vectors of the i-th and j-th data blocks respectively; represents the square of the Euclidean distance between the two eigenvectors; σ is the kernel width parameter of the Gaussian kernel function, which controls the rate at which the similarity decays with distance.

[0080] To ensure that the similarity matrix accurately reflects the inherent scale of the data set and to avoid distortion of the manifold structure information due to improper parameter selection, the value of the kernel width parameter σ is specifically limited in this invention. Its value is determined by calculating the average of the squares of the Euclidean distances between the eigenvectors of all data block pairs. The specific calculation formula is as follows:

[0081]

[0082] Where N is the total number of data blocks. This method provides a deterministic and adaptive calculation basis for the selection of kernel width parameters.

[0083] In order to perform subsequent spectral analysis to reveal the manifold structure, it is necessary to construct a graph Laplacian matrix based on the calculated similarity matrix W First, construct a diagonal matrix, namely the metric matrix definition D, whose diagonal elements D ii It is defined as the sum of all elements in the i-th row of the similarity matrix W, that is, Then, the graph Laplacian matrix It can be defined as

[0084] Graph Laplacian Matrix The spectrum of (i.e., its eigenvalues and eigenvectors) contains the intrinsic geometric information of the data manifold.

[0085] This step extracts this information by solving the following generalized eigenvalue problem: Among them, λ is the eigenvalue, The corresponding eigenvector is obtained by solving this problem. And sort by eigenvalue size, that is, 0=λ0≤λ1≤…≤λ N-1 .

[0086] Finally, in order to embed the data from the high-dimensional original feature space into the low-dimensional intrinsic manifold space, the present invention discards the trivial solution corresponding to the minimum eigenvalue λ0=0 (its corresponding eigenvector is a constant vector) and selects the subsequent d smallest non-zero eigenvalues λ1,...,λ d The corresponding eigenvector Combining these d eigenvectors by column forms an N×d. In order to perform subsequent spectral analysis to reveal the manifold structure, it is necessary to construct a graph Laplacian matrix L based on the calculated similarity matrix W.

[0087] First, construct a diagonal matrix, namely the metric matrix D, whose diagonal elements D ii It is defined as the sum of all elements in the i-th row of the similarity matrix W, that is,

[0088] Then, the graph Laplacian matrix It can be defined as the L = DW.-dimensional manifold space coordinate matrix

[0089] The i-th row vector of the matrix That is, the new coordinate of the i-th data block in the embedded d-dimensional manifold space, which accurately reflects the geometric position of the data block in the entire data set.

[0090] At this point, step S1 is completed, and the manifold space coordinate matrix M generated thereby will serve as the basis for the next step of dynamic quantum sharding.

[0091] Step S2: Construct a quantum Ising model based on the geometric characteristics of the manifold space coordinate matrix, solve the spin configuration through the quantum annealing algorithm, and dynamically divide the data into shards.

[0092] Specifically, the present invention first calculates the metric matrix g of the data manifold based on the manifold space coordinate matrix M obtained in step S1. μv The metric tensor g μv It accurately describes the measurement of distances and angles within an infinitesimal neighborhood of any point in a low-dimensional space defined by manifold coordinates, thereby capturing the inherent local geometric properties of the data set.

[0093] where μ and ν are indices of the local manifold coordinate system, ranging from 1 to the dimension d of the manifold space. This metric tensor can be calculated using numerical methods such as differencing or differentiation of the manifold coordinates and is the fundamental basis for establishing geometric relationships between data blocks.

[0094] After obtaining the metric tensor g μv Afterwards, the present invention further converts this geometric information into interaction parameters of the Ising physical model. i and d j Define a coupling coefficient J ij ,This coefficient is used to quantify the interaction strength of the two data blocks in the sing model.

[0095] Coupling coefficient J ij The values of are directly determined by their geometric proximity on the data manifold. A specific implementation is to ij is defined as the data block d i and d j Functions related to geodesic distances on manifolds.

[0096] When the geometric relationship between two data blocks on the manifold is closer, the corresponding coupling coefficient J ij It is set to a larger positive value, otherwise it is set to a smaller or negative value, thereby providing physical constraints for the subsequent energy minimization process.

[0097] Subsequently, the present invention constructs an Ising model Hamiltonian H that describes the total energy of the system, and its mathematical expression is:

[0098] H=-∑ i<j J ij σ i σ j -∑i h i σ i ;

[0099] Among them, σ i is related to each data block d i The associated spin variable has a value of {+1,-1}. The value of this variable directly determines the data block d i The data shard to which it belongs, for example, σ i = +1 means that data block i is divided into the first fragment, σ i =-1 means it is divided into the second shard; J ij This is the coupling coefficient defined above, which reflects the geometric relationship between data blocks i and j; i It is an external local field parameter, which is a bias parameter that can be set according to specific business requirements or system constraints to impose bias guidance on the division of specific data blocks.

[0100] Find the spin configuration that can minimize the above Hamiltonian H is a combinatorial optimization problem. The present invention uses quantum annealing technology to efficiently solve this problem. The process first maps the Ising model to a quantum system containing N quantum bits, where the state of the 60th quantum bit corresponds to the self-applied variable σ i .

[0101] In the initial stage of quantum annealing, the system is prepared in a ground state of an initial Hamiltonian H0, where |ψ(0)> is a ground state for all 2 N The uniform superposition state of possible classical spin configurations is expressed as:

[0102]

[0103] Where |s> represents a specific classical spin configuration (i.e., a complete data sharding scheme), Σ s Indicates that all 2 N The sum of possible configurations is calculated, where N is the total number of qubits, that is, the total number of data blocks. This initial state means that at the beginning of the solution, all possible sharding schemes are treated equally.

[0104] Then, the Hamiltonian H(t) followed by the system will slowly and continuously evolve adiabatically from the initial Hamiltonian H0 to the final target Hamiltonian H with time t. f Here H f This is the quantum mechanical representation of the Ising model that needs to be solved, and its specific form is:

[0105]

[0106] in is the Pauli Z operator acting on the i-th quantum bit, and its eigenvalue is the classical spin value ±1. The evolution of the quantum state |ψ(t)> of the entire system follows the time-dependent Schrödinger equation Where h is the reduced Lenck constant.

[0107] According to the quantum adiabatic theorem, if the evolution process is slow enough, the system will always remain in the ground state of the instantaneous Hamiltonian H(t) with extremely high probability.

[0108] Therefore, at the end of the evolution, the quantum state of the system is the target Hamiltonian H f At this point, by measuring all the quantum bits in the system on the computational basis, a unique set of spin configurations can be obtained.

[0109] This configuration is the global optimal solution or high-quality approximate solution of the Ising model Hamiltonian H, which ultimately determines the dynamic partitioning scheme for all data blocks.

[0110] Step S3: Based on the result of data sharding, storage resources are allocated in the orthogonal group space by using the Lie group optimization algorithm to generate a resource mapping matrix.

[0111] In this embodiment of the present invention, after completing the dynamic partitioning of data blocks in step S2, the process proceeds to step S3. This step aims to allocate specific storage resources to each acquired data shard in a manner that balances data privacy and security with resource utilization efficiency. By introducing the theory of Lie groups and Lie algebras, this step transforms the resource mapping problem into an optimization problem on a special orthogonal group manifold, and solves it to obtain a resource mapping matrix that maintains the integrity of the data structure.

[0112] In order to accurately describe the transformation characteristics of storage resources, the present invention first constructs a polymer algebraic space J. This space is composed of multiple Lie algebraic components through tensor product operations, and its mathematical expression is Among them, so(n l ) is the special orthogonal group SO(n l ) corresponds to the Lie algebra, which consists of all n l ×n l dimensional real antisymmetric matrix; L is the specific number of Lie algebra components that constitute this aggregation space; Represents the tensor product operation.

[0113] The establishment of this algebraic structure provides a rigorous mathematical framework for subsequent resource mapping. In order to protect the confidentiality of data content during resource allocation, the present invention performs the following steps on the data fragment S obtained in step S2: pApply fully homomorphic encryption algorithm.

[0114] Through this encryption operation, each plaintext data fragment S p Converted into a ciphertext data fragment It can directly perform subsequent mathematical operations without decryption. This encryption process can be abstractly represented as Enc(x) = (A, b), and one specific implementation is b = As + e + x (modq).

[0115] In this expression, x is the plaintext unit to be encrypted; A is a public matrix in the encryption scheme; s is the private key used in the encryption scheme; e is a random error term introduced in the encryption process; and q is a large integer modulus on which the encryption operation depends.

[0116] On the basis that the data has been encrypted and protected, the present invention constructs and solves an optimization problem in the encryption domain. First, a cost matrix C is constructed, which is expressed as In this definition, α p is the preset weight coefficient of the p-th data shard; is the pth data shard after homomorphic encryption; r ' is a vector representing the characteristics of the jth type of available storage resources or specific resource requests. Subsequently, the present invention proposes the following optimization goals:

[0117] min U∈SO(n) Tr(U T CU) where U is an n×n dimensional special orthogonal matrix to be solved, representing the rotation transformation from data slices to storage resources. SO(n) is the group space of all n×n dimensional special orthogonal matrices; Tr(·) is the trace operation of the matrix.

[0118] The goal of this optimization problem is to find an optimal rotation transformation U that minimizes the cost of a certain association between encrypted data shards and storage resources.

[0119] In order to solve the optimization problem defined on the SO(n) manifold, the present invention adopts an iterative optimization algorithm. In the kth iteration of the algorithm, the current estimate of the optimal matrix U k It is updated to U by the following update rules k+j :U k+j =exp(-ηX k )U k , in this iterative formula, U k with U k+1 are the n×n special orthogonal matrices at the beginning and after the update of the kth iteration respectively; η is a positive scalar learning rate used to control the step size of each iteration; exp(·) represents the matrix exponential function, which ensures that the updated matrix Uk+1 It remains a special orthogonal matrix, thus maintaining the validity of the solution throughout the optimization process.

[0120] Matrix X k is an n×n dimensional real antisymmetric matrix, that is, X k ∈so(n), which represents the current point U k Cost function The optimal descent direction of is represented in Lie algebra space. The matrix X k The value of is based on the cost function About the matrix U at point U k The gradient at is calculated, which ensures that the iteration proceeds along the geodesic direction on the manifold, approaching the optimal solution in the most efficient way. By repeatedly performing the above iterative update steps until the preset convergence conditions are met, an optimal special orthogonal transformation matrix U is finally obtained.

[0121] This matrix is determined as the final resource mapping matrix D. As an element of the special orthogonal group SO(n), this matrix D naturally satisfies the orthogonal constraint D T D=I n (where I n is the n×n identity matrix). In addition, the matrix D can also be parameterized by the exponential mapping from Lie algebra to Lie group, which is in the form of where {A k} is a set of basis matrices of the Lie algebra so(n), and {θ k} is the scalar parameter finally determined by the above optimization process. This process ensures that the generated resource mapping is both optimal and structure-preserving.

[0122] Step S4: Constructing the transmission package structure and optimizing the transmission configuration

[0123] In this embodiment of the present invention, after steps S1 to S3 complete the discovery of the data's intrinsic geometric structure, optimal data sharding based on that structure, and secure mapping to storage resources, the process proceeds to step S4. This step aims to ultimately encapsulate and integrate the results of the previous steps, constructing a structured data payload suitable for cross-modal transmission and determining the optimal operating parameters for the transmission channel carrying that payload.

[0124] Specifically, the present invention first solidifies and directs the optimal data sharding scheme determined by quantum annealing in step S2. The optimal scheme is embodied as a set of spin configurations Each of these The value of represents the final ownership of the i-th data block. To facilitate subsequent processing and recording, the present invention encodes this configuration into a quantum instruction sequence Q emd .

[0125] The sequence is achieved by assigning a quantum tag QTAG to each data block. i The polymerization operation is performed and is expressed as In this aggregation formula, N is the total number of data blocks, Represents a predefined aggregation operation, such as concatenating the data structures of each quantum tag in sequence. Each quantum tag QTAG i It is a unit with a specific data structure, which is defined as: QTAG i =[opcode(8),qubit_addr(16),spin_val(32)], where opcode(8) is an 8-bit opcode field used to indicate the type of the tag or subsequent operation instructions related to it; qubit_addr(16) is a 16-bit address field, whose value uniquely identifies the quantum bit associated with the i-th data block during the quantum annealing process; and spinval(32) is a 32-bit data field used to store the spin state corresponding to the i-th data block finally measured by step S2.3.

[0126] While directing the sharding scheme, the present invention provides each data shard S determined by the optimal spin configuration. p Construct a transmission tensor payload that encapsulates its core information. First, for a specific data slice S p , construct its tensorized vector representation vec(S p This construction process is achieved by accumulating the tensor product of the manifold coordinates of all data blocks in the shard and the data itself, and its calculation formula is:

[0127]

[0128] In this formula, the sum is traversed over the shards S p All data block indexes j in m j is the manifold coordinate vector of the jth data block; d j is the jth original data block or its pre-extracted numerical representation; Represents a tensor product operation. This operation converts the geometric position information m of the data block into j With data content d j Organically integrated into a unified tensor structure.

[0129] Then, using the finalized resource mapping matrix D and manifold eigenvector For the above tensor vector representation vec(S p ) performs the final transformation and packaging to generate the transmission tensor load Ptensor The calculation formula for this process is:

[0130]

[0131] Among them, D T is the transpose of the resource mapping matrix D, which acts on vec(S p ) realizes the mapping of data to resources in a secure and structure-preserving encryption domain; represents the multiplication of matrices and vectors or generalized linear transformations; e φ is the manifold eigenvector obtained The generated manifold basis space tensor provides a reference basis describing the global geometric background for the transmitted load.

[0132] Finally, the present invention is to carry the transmission tensor load P constructed above tensor The bus library or transmission channel determines an optimal operating parameter τ * , the parameter τ * It is also quantified as an improvement level of data processing. The optimal operating parameter τ * Obtained by solving a specific maximization problem:

[0133]

[0134] Where τ is the operating parameter to be optimized and T is its predefined valid value range. The objective function L(τ) is defined as The value of the potential energy function V(τ) in this function is proportional to the aggregate strength of the transmission tensor load of all data shards and is dynamically adjusted by the data promotion level τ. It is specifically expressed as:

[0135]

[0136] In the definition of this potential energy function, κ(τ) is a predefined non-negative scalar function that is dynamically adjusted according to τ; N S is the total number of data shards; α p is a predefined positive real weight assigned to the p-th data shard; is the transmission tensor payload corresponding to the p-th data slice; It represents the square norm of the transmission tensor load. By finding the optimal τ * ,The present invention configures an optimal working state for the entire data transmission process,taking into account both operating efficiency and load cost.

[0137] Please see the attached Figure 2 The present invention also provides a big data distributed storage system based on cloud computing, the specific contents of which are as follows:

[0138] A big data distributed storage system based on cloud computing, comprising:

[0139] The data manifold feature extraction module is used to extract feature vectors from data blocks, calculate the similarity matrix between data blocks, and perform feature decomposition on the similarity matrix to obtain the manifold coordinates and manifold feature vectors of the data manifold;

[0140] Dynamic quantum sharding module, which constructs an Ising physics model based on the geometric characteristics of the data manifold and solves the ground state of the model through quantum annealing technology to determine the optimal solution for allocating data blocks to different data shards;

[0141] The secure resource mapping module is used to homomorphically encrypt data shards and construct and solve an optimization problem defined on a special orthogonal group manifold to generate an optimal special orthogonal mapping matrix for securely mapping encrypted data shards to storage resources.

[0142] The transmission encapsulation module is used to encode the optimal sharding scheme into an instruction sequence and construct a transmission tensor payload for each data shard that encapsulates manifold geometry information, data content, and undergoes optimal special orthogonal mapping matrix transformation.

[0143] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0144] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A big data distributed storage method based on cloud computing, characterized in that: The following steps are involved: Step S1: perform manifold embedding on the original data block to generate a manifold space coordinate matrix; Step S2: constructing a quantum Ising model based on the geometric characteristics of the manifold space coordinate matrix, solving the spin configuration through a quantum annealing algorithm, and dynamically dividing the data into slices; Step S3: Based on the result of the data sharding, storage resources are allocated in the orthogonal group space by using a Lie group optimization algorithm to generate a resource mapping matrix; Step S4: According to the resource mapping matrix, the data slices and quantum instructions are transmitted to the target storage node in a coordinated manner through the cross-modal bus protocol.

2. The method for distributed storage of big data based on cloud computing according to claim 1, characterized in that: The step S1 comprises: S1.1: For each data block d in N data blocks i , where i = 1, ..., N, extract its m-dimensional feature vector f i =[φ1(d i ),...,φ m (d i )] T , where φ m (·) represents the mth predefined feature extraction function, which is used to capture the key attributes of the original data block; S1.2: Construct a similarity matrix based on the feature vectors of each extracted data block Its elements are expressed by the formula Calculated, where ||f i -f j ||2 is the eigenvector f i With f j The Euclidean distance between them is used to quantify the similarity between data blocks, N is the total number of data blocks, and σ is the kernel width parameter, which adjusts the rate at which similarity decays with distance; S1.3: Calculate the angle matrix The angle matrix The elements on the diagonal of Represents data block d i The total similarity of the angle matrix All non-diagonal elements of are zero, and the similarity matrix W and degree matrix D are used to construct the transfer matrix P = D -1 W, the transfer matrix is the probability transfer of random walks between data blocks; S1.4: Perform eigendecomposition on the transfer matrix P to obtain its first k largest eigenvalues {λ1,...,λ k }, where, and the set of eigenvectors corresponding to the maximum eigenvalue Among them, the maximum eigenvalue λ1≥λ2≥....≥λ k >0, reflects the importance of the data manifold in the corresponding direction, corresponds to the eigenvalue λ ι The lth eigenvector of represents a main direction of the data in the manifold space; Based on the maximum eigenvalue and each component in the eigenvector set, for each data block d i Construct its k-dimensional coordinates in low-dimensional manifold space The k-dimensional coordinate m i The projections of the data blocks in each main direction are combined and weighted by the eigenvalues, so that the manifold coordinates of all data blocks are combined to form the manifold space coordinate matrix The coordinate matrix The i-th behavior A low-dimensional manifold embedding representation of the original high-dimensional data block is achieved.

3. The method for distributed storage of big data based on cloud computing according to claim 2, characterized in that: The value of the kernel width parameter σ used to calculate the similarity matrix W in step S1.2 is determined by calculating the average of the squares of the Euclidean distances between the eigenvectors of all data block pairs. The specific calculation formula is as follows: Wherein, N is the total number of data blocks in step S1.1, f i and f j are the feature vectors extracted from the i-th and j-th data blocks in step S1.1, represents the eigenvector f i With f j The square of the Euclidean distance between them.

4. The method for distributed storage of big data based on cloud computing according to claim 1, characterized in that: To achieve dynamic quantum sharding based on the geometric characteristics of the data manifold, step S2 specifically includes: S2.1: Based on the manifold space coordinate matrix M obtained in step S1, calculate the Riemannian metric tensor g that describes the local intrinsic geometric properties of the data manifold μv , where μ and ν are indices of the local coordinate system of the manifold; S2.2: According to the Riemann metric tensor g μv and the manifold space coordinate matrix M, for any pair of data blocks d i and d j Derivation of the coupling coefficient J in the lsing model ij , the coupling coefficient J ij Quantitatively describes the data block d i and d j geometric correlation strength or interaction energy on the data manifold; S2.3: Construct Hamiltonian H of target lsing model = -∑ i<j J ij σ i σ j -∑ i h i σ i , where σ i Assigned to data block d i The spin variable, σ i The value of indicates the shard to which the data block belongs, h i is applied to the spin σ i External local field parameters; The Hamiltonian is optimized and solved to obtain a set of optimal spin configurations The optimal spin configuration The dynamic sharding results of all data blocks are determined.

5. The method for distributed storage of big data based on cloud computing according to claim 4, characterized in that: In step S2.3, the step of optimizing and solving the Hamiltonian includes the step of solving the optimal spin configuration of the Ising model Hamiltonian using quantum annealing technology, specifically: a) Initialize the system as a system containing N qubits, where each qubit corresponds to the spin variable σ of a data block i , its initial quantum state |ψ(0)> is all 2 N Uniform re-addition of possible computational ground states |s>: This initial state is the ground state of the initial Hamiltonian H0; b) The state of the quantum system |ψ(t)> evolves with time according to the time-dependent Schrödinger equation: Among them, Planck constant h is taken as 1; c) The time-dependent Hamiltonian H(t) slowly and continuously adiabatically evolves from the initial Hamiltonian H0 to the final target Hamiltonian H f ; The final target Hamiltonian H f That is the Ising model Hamiltonian of the optimal data sharding scheme after ground state encoding As a result, the system is in the final target Hamiltonian H with high probability at the end of the evolution f The ground state of the final target Hamiltonian H f The optimal spin configuration is obtained by measuring the ground state 6. The method for distributed storage of big data based on cloud computing according to claim 1, characterized in that: To achieve secure, optimized, and structure-preserving mapping of encrypted data shards to storage resources, step S3 includes: S3.1: Constructing an Aggregate Algebraic Space for Describing Storage Resource Transformation Characteristics Among them, so(n l ) is n l ×n l The special orthogonal group SO(n l ), where L is the number of aggregated Lie algebra components; S3.2: Slice the data obtained in step S2 p Apply the fully homomorphic encryption algorithm to encrypt and obtain ciphertext data fragments that can be calculated in the encrypted state The encryption process can be expressed as Enc(x) = (A, b), and b = As + e + x (modq), where x is the plaintext unit, A is the public matrix, s is the private key, e is the error term, and q is the modulus; S3.3: Construct a cost matrix in the encryption domain Among them, α p is the data shard weight, For encrypted data sharding, r j is a resource request vector and solves the optimization problem min U∈SO(n) Tr(U T CU) obtaining an optimal n×n special orthogonal resource mapping matrix U, wherein the n×n special orthogonal resource mapping matrix U is used to align encrypted data slices with storage resources; S3.4: Based on the optimal n×n special orthogonal resource mapping matrix U obtained by the optimization solution, generate the final n×n resource mapping matrix D, the final n×n resource mapping matrix D is an element of the special orthogonal group SO(n) and satisfies the orthogonal constraint D T D=I n , where I n is the identity matrix and can be expressed as the exponential mapping from the Lie algebra so(n) to the Lie group SO(n) Among them, A k is the antisymmetric basis matrix of so(n), θ k Parameters determined for optimization.

7. The method for distributed storage of big data based on cloud computing according to claim 6, characterized in that: The resource mapping matrix D in step S3.4 is represented by a set of scalar parameters and scalar parameters The corresponding n×n real antisymmetric matrix of the predefined Lie algebra so(n) When parameterization is generated, step S3.3 is used to solve the optimization problem min U Tr(U T The rules for iteratively updating the resource mapping matrix by CU include the following operations: In the i-th iteration, if the parameter Θ k To make adjustments, the resource mapping matrix is changed from the current state U i Update to the next state U i+1 , where k is a parameter index, 1≤k≤P, and the resource status mapping formula is: IN i+1 =exp(-ηA (k) )IN i ; in, U i ∈SO(n) is the n×n special orthogonal resource mapping matrix at the beginning of the i-th iteration; U i+1 ∈SO(n) is the updated n×n special orthogonal resource mapping matrix after this parameter adjustment; η(eta) is a positive scalar learning rate that controls the iteration step size; exp(·) is the matrix exponential function; A (k) is a parameter Θ k The n×n real antisymmetric matrix A is constructed based on the gradient information of the Lie algebra so(n). (k) The specific calculation steps are as follows: First, calculate the cost function In the current resource mapping matrix U i When the kth scalar parameter Θ k The partial derivative value is recorded as the scalar gradient value g k , Where C is the cost matrix; Then, using the scalar gradient value g k and the scalar parameter Θ k The corresponding Lie algebra basis matrix B k To construct the n×n real antisymmetric matrix A (k) , A (k) =g k B k .

8. The method for distributed storage of big data based on cloud computing according to claim 1, characterized in that: To construct a cross-modal transmission channel based on tensor manifold encapsulation, step S4 further includes: S4.1: The optimal spin configuration obtained by quantum annealing in step S2.3 Encoded as a quantum instruction sequence Q cmd , the encoding is a quantum instruction sequence Q cmd The quantum tag QTAG corresponding to each data block i Aggregated, expressed as Where N is the total number of data blocks, is the final spin state of the i-th data block, ⊕ represents the aggregation operation, and each quantum tag QTAG i The data structure is defined as: QTAG i =[opcode(8),qubit a ddr(16),spin v al(32)]; in, opcode(8) is an 8-bit operation code field used to indicate the type of the tag or subsequent operation; qubit a ddr(16) is a 16-bit quantum bit address field, which is used to uniquely identify the quantum bit corresponding to the i-th data block; spin v al(32) is a 32-bit field used to store or represent the final spin state of the i-th data block S4.2: For the optimal spin configuration Determine the data block index set S for each data shard p , first construct the tensor vector representation vec(S p ), the tensor product of the manifold coordinates of all data blocks in each data slice and the data block itself is accumulated through the tensor accumulation model. The formula of the tensor accumulation model is: in, m j is the manifold coordinate vector of the j-th data block; d j is the jth original data block or its predetermined representation; Represents the tensor product operation; Then, using the transpose D of the resource mapping matrix D T , combined with the manifold eigenvector The generated manifold basis space tensor e φ , the tensor quantization vector is represented by vec(S p ) is transformed and encapsulated to construct the transmission tensor load P tensor , in, Represents matrix-vector multiplication or generalized linear transformation.

9. A system for implementing the method according to any one of claims 1 to 8, characterized in that: include: A data manifold feature extraction module is used to extract feature vectors from data blocks, calculate the similarity matrix between data blocks, and perform feature decomposition on the similarity matrix to obtain the manifold coordinates and manifold feature vectors of the data manifold; A dynamic quantum sharding module is used to construct an Ising physical model based on the geometric characteristics of the data manifold and solve the ground state of the model through quantum annealing technology to determine the optimal solution for allocating data blocks to different data shards; a secure resource mapping module for homomorphically encrypting the data shards and constructing and solving an optimization problem defined on a special orthogonal group manifold to generate an optimal special orthogonal mapping matrix for securely mapping the encrypted data shards to storage resources; The transmission encapsulation module is used to encode the optimal sharding scheme into an instruction sequence and construct a transmission tensor load for each data shard that encapsulates manifold geometry information, data content and is transformed by the optimal special orthogonal mapping matrix.

Images