Satellite distributed cluster-oriented data placement optimization method and system

Abstract

The invention belongs to the technical field of distributed systems and satellite communication, and discloses a satellite distributed cluster-oriented data placement optimization method and system. The method comprises the steps that a working node on a satellite distributed cluster generates a historical retrieval log, obtains state information and network topology of the satellite distributed cluster and sends the state information and the network topology to a computing node on the satellite distributed cluster; the computing node extracts a data feature vector from the historical retrieval log, inputs the data feature vector into a pre-trained association model, and predicts data correlation; determining a data placement scheme according to the current state information of the satellite distributed cluster, the network topology and the data correlation, and sending the data placement scheme to the working node; and the working node executes data migration according to the data placement scheme to realize optimized placement of the data. According to the method, accurate prediction of the new data relevance is realized, high-relevance data copies are aggregated and stored in the same node or adjacent nodes, and the total time delay of a multi-data retrieval task is reduced.

Description

Technical Field

[0001] This invention belongs to the field of distributed systems and satellite communication technology, and relates to a data placement optimization method and system for satellite distributed clusters. Background Technology

[0002] In satellite-to-ground communication scenarios, satellite constellations generate a large amount of task result data as they perform various tasks. Ground communication stations need to frequently send data requests to the satellites to obtain these task results. When making data retrieval requests, ground terminals often exhibit "locality of interest" towards the target data. For example, when a task requests observation data at 8:00 AM Beijing time, it is highly likely to simultaneously request observation data from the same location at 9:00 AM Beijing time. This locality of interest results in data frequently appearing in the same retrieval request being relevant to the request.

[0003] Existing distributed data placement methods typically rely on historical records. By parsing historical data access records, the number of times data pairs appear in the same retrieval request (co-access count) is counted. Data pairs with high co-access counts are placed on the same node to reduce the latency of future requests for these data.

[0004] However, existing technologies face the following problems and challenges in dynamic edge scenarios such as distributed low-orbit satellite constellations.

[0005] (1) New data lacks access records: The cluster continuously generates new data during operation. This new data lacks historical access records, so it is impossible to use existing technologies that rely on access history to calculate the number of co-accesses and guide placement. Users still face high request latency when making multiple data requests for this data.

[0006] (2) Complex cluster environment and limited resources: The previously mentioned methods for placing request-related data are mostly applied in data centers where network latency and topology are relatively stable. However, the network status between nodes in a satellite cluster is unstable, and the computing power and storage resources of satellite nodes are heterogeneous and limited. If highly correlated data is simply placed on the same node, it is easy to cause too much data to be stored on a few nodes, exacerbating the load on some nodes. If the network status of the node is poor or the storage capacity is insufficient, it is easy to cause a single node failure, thereby reducing the service performance of the entire cluster.

[0007] In summary, there is an urgent need for an optimization method that adapts to the characteristics of satellite distributed clusters and guides data placement accordingly, in order to reduce the latency of multi-data retrieval in satellite distributed clusters. Summary of the Invention

[0008] To address the problems existing in the prior art, the purpose of this invention is to provide a data placement optimization method and system for satellite distributed clusters.

[0009] This invention provides a data placement optimization method for satellite distributed clusters, the steps of which include: The worker nodes on the satellite distributed cluster generate historical retrieval logs and obtain the status information and network topology of the satellite distributed cluster, which are then sent to the computing nodes on the satellite distributed cluster. The computing node extracts data feature vectors from the historical retrieval logs and inputs them into a pre-trained association model to predict data relevance; and determines a data placement scheme based on the current status information, network topology, and data relevance of the satellite distributed cluster and sends it to the working node. The working node performs data migration according to the data placement scheme to achieve optimized data placement.

[0010] Preferably, the step of the computing node extracting data feature vectors from the historical retrieval logs is as follows: The historical retrieval logs are cleaned to obtain a set of effective retrieval tasks; based on the task identifier and request data identifier in the log information, a data subset associated with the retrieval task is generated for each task in the task set. Extract a set of features from each data point in the subset of data to obtain the data feature vector for that data.

[0011] Preferably, the step of generating the data correlation matrix is ​​as follows: For any pair of data in the data subset, count the number of times they co-occur in the effective retrieval task set; calculate the total number of occurrences for each pair of data respectively; The retrieval relevance of this data to the two data is determined by the co-occurrence frequency and the total occurrence frequency of each data. Based on the retrieval relevance of each data pair, a data relevance matrix is ​​obtained.

[0012] Preferably, the step of predicting the data correlation using a pre-trained association model is as follows: Construct a sample dataset, which includes multiple samples. Each sample includes a data feature vector matrix composed of data feature vectors of multiple data and a corresponding data correlation matrix. The sample dataset is divided into a training set and a validation set. Construct a correlation model with non-zero value focus; The association model is trained using the training set, and validated and optimized using the validation set.

[0013] Preferably, the association model includes a data feature encoder, a similarity detector, and a relevance strength predictor; the association model is trained using the training set, and the training method is as follows: The data feature vector matrix of the sample is encoded using a data feature encoder to generate low-dimensional data features; Calculate the spatial distance and temporal difference of all data pairs in the sample as spatial-temporal features; fuse the feature differences of data pairs with the spatial-temporal features to form the comprehensive features of the data pairs; The similarity probability of data pairs is calculated based on the comprehensive features using a similarity detector; The correlation strength predictor outputs the basic strength of the data pairs based on the comprehensive features, and then weights them with similarity probability as the weight to obtain the preliminary correlation strength matrix of the data pairs. Set the diagonal elements of the preliminary correlation strength matrix to zero, and map the output range to a preset interval to obtain the data correlation matrix; Calculate the non-zero prediction loss based on the non-zero elements in the data correlation matrix, calculate the classification loss based on the target matrix and the sample labels, calculate the range constraint loss based on the non-zero prediction values ​​in the target matrix, and calculate the zero penalty loss based on the elements in the target matrix where the target is zero. A total loss value is calculated based on the non-zero prediction loss, classification loss, range constraint loss, and zero-value penalty loss, and the association model is optimized based on the total loss value.

[0014] Preferably, the step of generating the data placement scheme is as follows: Let the set of data objects to be placed be business-related satellite data, and the set of satellite nodes be the satellite cluster nodes participating in the storage; Extract the data feature vectors from the satellite data to construct a data correlation matrix, which is then input into the correlation model M to predict the data correlation of the satellite data. The optimization objective in the genetic algorithm is set according to the requirements of the mission scenario, and the constraints in the genetic algorithm are set according to the characteristics of the satellite cluster. The fitness function used to evaluate the merits of the data placement scheme is constructed according to the data correlation of the satellite data. Each gene of the chromosome in the genetic algorithm corresponds to the storage allocation of a data object. The data placement scheme is generated using the genetic algorithm.

[0015] Preferably, a fitness function is constructed based on data correlation aggregation, load balancing, and constraint satisfaction.

[0016] Preferably, the step of performing data migration is as follows: Compare the current data placement plan with the latest received data placement plan to determine the data that needs to be moved in and out; For data that needs to be moved in, a request is sent to the data storage node in the latest data placement scheme to retrieve the data; For data that needs to be moved out, a request is sent to the target node to transfer the data to the target node.

[0017] Preferably, the satellite distributed cluster is a low-Earth orbit constellation network.

[0018] Another aspect of the present invention provides a data placement optimization system for satellite distributed clusters, characterized in that it includes a preprocessing module, a correlation calculation module, a model building module, and a placement optimization module deployed on the computing nodes of the satellite distributed cluster; and a status acquisition module and a data migration module deployed on the working nodes of the satellite distributed cluster. The status acquisition module is used to collect the execution status of data retrieval tasks on the current working node, generate historical retrieval logs, and obtain the status information and network topology of the satellite distributed cluster. The preprocessing module is used to extract data feature vectors from the historical retrieval logs; The correlation prediction module is used to input the data feature vector into a pre-trained correlation model to predict the data correlation. The placement scheme generation module is used to generate a data placement scheme based on the current status information and data correlation of the satellite distributed cluster. The data migration module is used to perform data migration according to the data placement scheme to achieve optimized data placement.

[0019] The advantages of this invention are as follows: (1) By constructing a correlation model between the data feature labels and the correlation of historical access data, accurate prediction of the correlation of new data can be achieved; (2) When placing data, adaptive load balancing and reliability are taken into account to ensure stable system operation; (3) Based on the relevance prediction results, highly relevant data copies are aggregated and stored on the same or adjacent nodes to reduce the total latency of multiple data retrieval tasks. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a system module diagram.

[0022] Figure 3 This is a system flowchart. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0024] like Figure 1 As shown, the data placement optimization method for satellite distributed clusters of the present invention includes the following steps: The worker nodes on the satellite distributed cluster generate historical retrieval logs and obtain the status information and network topology of the satellite distributed cluster, which are then sent to the computing nodes on the satellite distributed cluster. The computing node extracts data feature vectors from the historical retrieval logs and inputs them into a pre-trained association model to predict data relevance; and determines a data placement scheme based on the current status information, network topology, and data relevance of the satellite distributed cluster and sends it to the working node. The working node performs data migration according to the data placement scheme to achieve optimized data placement.

[0025] An optional embodiment of the present invention provides a data placement optimization system for satellite distributed clusters, the module composition of which is as follows: Figure 2 As shown, the deployment, main responsibilities, and interaction methods of each module are as follows: (1) The satellite distributed cluster consists of several computing nodes and several working nodes. The computing nodes are used to analyze data and generate optimized placement decisions, while the working nodes are used to handle specific data retrieval tasks and apply the data placement decisions. The preprocessing module, correlation calculation module, model building module, and placement optimization module are deployed on the computing nodes, while the status acquisition module and data migration module are deployed on the working nodes. When performing data placement optimization, the computing nodes that are closer to each other can coordinate with the relevant working nodes to complete the task. (2) The preprocessing module interacts with the status acquisition module to collect historical retrieval logs, clean up logs and extract features; (3) The correlation calculation module interacts with the preprocessing module to calculate the data correlation matrix based on the characteristics of the processed logs; (4) The model building module interacts with the preprocessing module and the correlation calculation module to use a neural network model to predict data correlation; (5) The placement optimization module interacts with the status acquisition module, data migration module, and model building module. Based on the current cluster status and data correlation, the genetic algorithm is used to calculate the optimized data placement scheme. (6) The status acquisition module interacts with the preprocessing module and the placement optimization module to provide cluster information and relevant information about the data retrieval tasks being performed; (7) The data migration module interacts with the placement optimization module, receives the optimized data placement plan, and performs the data migration.

[0026] In an optional embodiment, the specific steps of the preprocessing module are as follows: (1) Collection and cleaning of historical retrieval logs: 1. The preprocessing module continuously interacts with the status acquisition module of each working node in the satellite cluster to collect the historical retrieval logs of each working node in the satellite cluster. The log fields include "retrieval task ID, list of requested data IDs, retrieval time, and requesting satellite node ID". 2. Clean up invalid logs (such as missing data IDs or failed retrieval records) and retain valid retrieval task sets. , where each task T k Includes data subset ; k = 1 ~ m; The elements in the data subset are the data associated with the retrieval task, which are aggregated from the task ID and request data ID list in the log information collected in step 1.

[0027] (2) Data feature extraction: 1. Extract the data feature labels from all data and define the data feature set. , characterizing the data One characteristic, The p-th data label corresponds to data generation time, latitude and longitude coordinates, etc., and the specific characteristics are determined according to the satellite data type; 2. Standardize the data features, corresponding to the data feature set. Each feature is defined for each data element in the subset of data associated with each task obtained after data cleaning in step (1). Extract features and form a feature vector corresponding to the data. , For data The One characteristic.

[0028] In an optional embodiment, the specific steps of the correlation calculation module are as follows: (1) Historical co-occurrence frequency statistics: 1. For each data pair Count the number of times it co-occurs in the effective retrieval task set. (i.e., appearing simultaneously in the same search task) (number of times in) 2. Calculate the data separately. , Total number of occurrences , .

[0029] (2) Quantification of retrieval relevance: 1. Define data , Search relevance The calculation is performed using a weighted model of "co-occurrence frequency - total frequency", as shown in the following formula: ,in , The closer to 1, the better. , The more likely they are to appear in the same search task (the higher the relevance); This indicates that there are no co-occurrence records. 2. Calculate the retrieval relevance of all data pairs to obtain a... Data correlation matrix .

[0030] In an optional embodiment, the specific steps of the model building module are as follows: (1) Construction of training dataset: 1. Using historical data as samples, each sample includes a data feature vector matrix. and the corresponding data correlation matrix obtained through step (2) of the above correlation calculation module. ; 2. Divide the sample set into a training set and a validation set in a 7:3 ratio for model training and performance evaluation.

[0031] (2) Training of association models with non-zero value focusing: Since the correlation matrix used for validation contains a large number of 0 values ​​(most of the data are not correlated), the model can achieve high accuracy even if it outputs 0 values ​​without any processing during training. This results in low prediction performance of the model for non-zero values. Therefore, it is necessary to build a non-zero value-focused association model for sparse scenarios, which mainly includes a data feature encoder, a similarity detector, and a correlation strength predictor.

[0032] 1. Model Building: This model employs a two-stage neural network architecture. Its core lies in first identifying the correlation between data pairs (whether they are non-zero values), and then accurately predicting the strength of the non-zero correlation. Specifically, this includes: 1.1 Data Feature Encoder: By combining multi-level linear transformation with nonlinear activation function (ReLU), and using batch normalization and dropout mechanisms, the input data feature vector matrix (including spatial coordinates and temporal features) is mapped to a low-dimensional hidden space, thereby achieving effective feature extraction and dimensionality reduction.

[0033] 1.2 Similarity Detector: Taking the feature difference of the data pair (from the output of the data feature encoder in step 1.1) and the spatial and temporal distance features as input, the similarity probability of the data pair is output by the multilayer perceptron to determine whether the data pair has a non-zero correlation.

[0034] 1.3 Correlation Strength Predictor: Based on the comprehensive features of data pairs (feature difference and spatial-temporal distance), the basic correlation strength of data pairs is output through a deeper perceptron structure. The output is weighted and adjusted by combining similarity probability, so that the strength prediction of highly similar data pairs is more significant.

[0035] 1.4 Dynamic Threshold: Introduce a learnable similarity threshold parameter to adaptively adjust the judgment criteria of data on similarity and improve the model's adaptability to different scenarios.

[0036] 2. Data preprocessing: Standardize the input data, including: 2.1 Convert the raw features of the data (spatial coordinates, timestamps, and other labels) into a format suitable for model input; 2.2 Align the format of the relevance target matrix to ensure that it matches the dimension of the input features, thus providing a consistent input-output structure for model training.

[0037] 3. Model Training: Model training is achieved through iterative optimization, and the core steps include: 3.1 Forward Propagation Process: 3.1.1 Data Feature Encoding: The data feature vector matrix of the sample is encoded using a data feature encoder to generate low-dimensional features; 3.1.2 Data Pair Feature Construction: The spatial distance and temporal difference of all data pairs in the sample are calculated using L2 Norm as spatial-temporal features; the feature differences of data pairs are fused with the spatial-temporal features to form the comprehensive features of the data pairs; 3.1.3 Similarity Detection: Based on comprehensive features, the similarity probability of data pairs is output through the similarity detector in 1.2; 3.1.4 Intensity Prediction and Adjustment: The basic intensity of the data pairs is output by the correlation intensity predictor based on comprehensive features, and then weighted by similarity probability to obtain the preliminary correlation intensity matrix of the data pairs; 3.1.5 Output Correction: Force the diagonal elements (i.e., autocorrelation) of the preliminary correlation matrix obtained in step 2 to zero, and map the output range to a preset interval (e.g., [0, 1.2]) to match the target value distribution, thus obtaining the final data correlation matrix. .

[0038] 3.2 Loss Function Optimization: A multi-objective fusion loss function is adopted, focusing on the prediction accuracy of non-zero values, including: 3.2.1 Non-zero value prediction loss For data correlation matrix For non-zero elements in the data, mean squared error loss is used to optimize the prediction accuracy of non-zero values; an indicator function is defined. For non-zero thresholds, Time indicates (If the value is non-zero, then it is 0; otherwise, it is 0). The loss for non-zero values ​​is calculated as follows: in This represents the number of non-zero value samples.

[0039] 3.2.2 Classification Loss The relevance prediction is transformed into a binary classification task of "non-zero" and "zero," and the model's ability to distinguish between zero and non-zero values ​​is improved by using weighted cross-entropy loss; target labels are set. Predicted probability We employ weighted binary cross-entropy (BCE) loss: in , which represents the weight of positive samples (to balance the uneven distribution of positive and negative samples). ( (Total number of data).

[0040] 3.2.3 Range Constraint Loss The range constraint loss is calculated using the following formula, penalizing non-zero predicted values ​​that exceed a reasonable range (e.g., [0.1, 1.5]) to ensure output stability: 3.2.4 Zero-value penalty loss For elements with a target value of zero, penalize the portion of their predicted value that is too high, thus suppressing misjudgments in the zero-value region. The specific formula for calculating the loss value is as follows: 3.2.5 The total loss is formed by weighted fusion of the sub-losses and the loss values ​​are truncated to avoid gradient explosion caused by outliers.

[0041] 3.3 Parameter Update: The Adam gradient descent optimization algorithm is adopted, and all parameters of the model (including encoder, detector, predictor and learnable threshold) are updated based on backpropagation of total loss; the performance is monitored through the validation set during training, and an early stopping strategy is adopted to prevent overfitting; 3.4 Train the aforementioned non-zero value focusing association model, save the trained association model M, and use it for new data correlation prediction.

[0042] In an optional embodiment, the specific steps of the placement optimization module are as follows: (1) Satellite constellation and data model construction: 1. Define the data and node set: Let the set of data objects to be placed be business-related satellite data (such as observation data, mission data, etc.), and the set of satellite nodes be the satellite cluster nodes participating in storage, including information such as the remaining storage capacity, current load and communication link status of each node.

[0043] 2. Data Correlation Quantification: Extract the data feature vectors from satellite data and input them into the correlation model M to predict data correlation, thus obtaining the data correlation matrix. .

[0044] 3. Optimize the target definition: Based on the task scenario requirements, the optimization target is defined as: maximizing the correlation clustering between data (data replicas with higher correlation are placed closer together).

[0045] 4. Constraint Definition: Based on the characteristics of the satellite cluster, the constraints include: node storage capacity constraint (data placement must not exceed the remaining storage resources of the satellite node), communication latency constraint (the communication latency between nodes containing relevant data must be lower than the threshold to adapt to the characteristics of the satellite link), and load balancing constraint (avoiding overload caused by excessive data storage on a single node).

[0046] (2) Placement optimization based on genetic algorithm: 1. Chromosome coding design: A multi-layer coding structure is used to represent a possible solution (data placement scheme). Each gene in the chromosome corresponds to the storage allocation of a data object, including the satellite node number where the primary copy and backup copy of the data are located, to ensure data reliability and access redundancy; 2. Initial Population Generation: Taking into account the current load of satellite nodes and the initial distribution of data, an initial population of a preset size is randomly generated to cover the placement patterns of preliminary aggregation of highly relevant data and balanced distribution, thereby improving the diversity of global search.

[0047] 3. Fitness function construction: The fitness function is used to evaluate the merits of placement schemes, comprehensively considering data correlation aggregation, load balancing, and constraint satisfaction, and is defined as follows: in: This is a correlation clustering index, calculated as the proportion of highly correlated data pairs on the same satellite node or adjacent low-latency link nodes. A higher value indicates a better correlation data aggregation effect. The calculation method is as follows: in Representing data Up to the most recently stored data The average distance to the nearest node. Distance matrix. Represents nodes in the cluster The shortest distance between them (including jump distance).

[0048] The load balancing index is calculated based on the uniformity of the distribution of the amount of data stored in each satellite node. The higher the value, the more balanced the load. The load balancing index is calculated using the standard deviation formula. To constrain violations of penalties, solutions that exceed storage capacity or latency thresholds are penalized to ensure the feasibility of the solutions. , , The weighting coefficient is dynamically adjusted adaptively based on the mission priority of the satellite constellation.

[0049] 4. Genetic operations and iterative optimization: 4.1 Selection Operation: A tournament selection strategy is adopted, in which several individuals are randomly selected from the population, and the individual with the highest fitness is selected as the parent generation, thus preserving high-quality gene fragments; 4.2 Crossover operation: Crossover is performed on the selected parent individuals. By exchanging node allocation information of data copies through single-point or multi-point crossover, offspring schemes are generated to strengthen the gene combination of high fitness schemes. 4.3 Mutation Operation: Randomly select offspring genes with a preset probability, adjust the node allocation of data replicas, introduce new placement patterns, and avoid the population from getting trapped in local optima; 4.4 Infeasible solution repair: Repair schemes that violate storage capacity or delay constraints after crossover or mutation to ensure the feasibility of offspring schemes; 5. Output of the optimal solution The iteration terminates when the preset number of iterations is reached, or when the optimal fitness stabilizes over multiple generations. The chromosome with the highest fitness in the population is selected as the optimal data placement scheme, and the primary and backup copies of the data objects are allocated to the corresponding satellite nodes according to this scheme.

[0050] In an optional embodiment, the specific steps of the status acquisition module are as follows: (1) Continuously collect the execution status of data retrieval tasks on the current working node and generate historical retrieval logs; (2) Continuously collect node information, network topology, and other data on the current working node; (3) Respond to the interaction request of the preprocessing module and return the corresponding data and historical search logs.

[0051] In an optional embodiment, the specific steps of the data migration module are as follows: (1) Interact with the placement optimization module to continuously obtain the current data placement plan; (2) Compare the current data placement scheme with the latest data placement scheme, filter the data entries of the current node, and determine the data entries that need to be moved in and out; (3) For each data entry that needs to be moved in, send a request to the data storage node in the latest placement scheme to obtain the data; (4) For each data entry that needs to be removed, send a request to the target node to transfer the data to the target node; (5) After confirming that all removed data entries have been successfully placed, delete the local copy.

[0052] An optional embodiment of the present invention provides a data placement optimization method for satellite distributed clusters, the specific steps of which are as follows: (1) The working nodes continuously execute data retrieval tasks, and the status acquisition module acquires and generates node and cluster status information and historical retrieval logs; (2) The preprocessing module on the computing node extracts data feature vectors from historical retrieval logs; (3) The correlation calculation module on the computing node uses data feature vectors to generate a data correlation matrix; (4) The model building module continuously trains the model based on the training set obtained in steps (2) and (3) to predict data correlation. (5) For the new input data feature vector, predict the data correlation matrix; (6) The placement optimization module uses a genetic algorithm to calculate the optimized placement scheme; (7) The data migration module continuously adjusts the data placement according to the optimized placement plan; (8) Continue to execute steps (1) to (7) to continuously adjust data placement in response to changes in the data retrieval task load.

[0053] like Figure 3 As shown, in an optional embodiment of the present invention, node A is a computing node, and nodes B to F are working nodes. Nodes A to F constitute a low-orbit constellation network.

[0054] (1) Detect relevant information about existing low-Earth orbit constellation networks, including satellite nodes and inter-satellite links, in order to map This represents the network topology formed by inter-satellite clusters. This represents a set of satellite nodes, which includes the current load attributes of each node. Let be the set of edges of the graph, where the weight of each edge represents the communication delay between the nodes at both ends of the edge; (2) Select the data to be processed as target detection result data, and the relevant data features are the spatial coordinates and timestamps (observation location, observation time) of the data. Satellite nodes B to F generate and save the data when performing observation tasks. When the ground data center requests data from nodes B to F, it queries the spatial coordinates and timestamps of the target data and generates the corresponding retrieval logs; (3) Node A and nodes B through F continuously interact. Based on the aforementioned retrieval logs, the status acquisition module and preprocessing module deployed on nodes B through F collect the log retrieval information of each node and filter to obtain historical data access records. The corresponding historical data features are normalized to obtain data feature vectors; (4) Based on its historical data access records, Node A uses the relevance calculation module to statistically analyze the frequency of data appearing in the same retrieval task, generating a historical data relevance matrix (e.g., ...). Figure 3 (as shown in step 1). (5) Node A, based on the historical data feature vector and the historical data correlation matrix, generates a training set through the model building module to train the association model (e.g., Figure 3 (See step 2 in the middle); after training is completed, the data is sent to the satellite edge computing unit (e.g., ...). Figure 3 (as shown in step 3). (6) For newly generated observation data of satellite nodes Node A continuously collects log data from nodes B to F, and calculates its corresponding data feature vectors (such as...) through a preprocessing module. Figure 3 (as shown in step 4). (7) Node A, based on the new data feature vector and the trained association model, uses the new data feature vector as the model input and outputs the prediction result - the new data association matrix (e.g., ...) through the model building module. Figure 3 (See steps 5 and 6 in the middle). (8) Based on the low-orbit constellation network structure and the new data correlation matrix, node A establishes a maximum correlation clustering optimization problem under constraints by placing optimization modules with the maximum correlation clustering as the optimization objective and latency and load balancing as constraints. ; (9) Node A solves the optimization problem using a multi-copy placement method based on genetic algorithm. Find the optimal data-node mapping relationship with the highest correlation clustering (e.g., Figure 3 (as shown in step 7). (10) The data migration modules deployed on nodes B through F receive the decision results from the placement optimization module on node A and perform data placement. In this example, the data... , The correlation index between them is high, and the data With data The correlation index between them is low, therefore the calculation result is to use the data , The corresponding replicas are placed on the same node. , And data The placement should consider load balancing and latency constraints, and should be placed on the node. , (like Figure 3 (See step 8).

[0055] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A data placement optimization method for satellite distributed clusters, comprising the following steps: The worker nodes on the satellite distributed cluster generate historical retrieval logs and obtain the status information and network topology of the satellite distributed cluster, which are then sent to the computing nodes on the satellite distributed cluster. The computing node extracts data feature vectors from the historical retrieval logs and inputs them into a pre-trained association model to predict data relevance; The data placement scheme is determined based on the current status information, network topology, and data correlation of the satellite distributed cluster and then sent to the working node. The working node performs data migration according to the data placement scheme to achieve optimized data placement.

2. The method according to claim 1, characterized in that, The steps for the computing node to extract data feature vectors from the historical retrieval logs are as follows: The historical retrieval logs are cleaned to obtain a set of valid retrieval tasks; based on the task identifier and request data identifier in the log information, a data subset associated with the retrieval task is generated for each task in the task set. Extract a set of features from each data point in the subset of data to obtain the data feature vector for that data.

3. The method according to claim 2, characterized in that, The steps for generating the data correlation matrix are as follows: For any pair of data in the data subset, count the number of times they co-occur in the effective retrieval task set; calculate the total number of occurrences for each pair of data respectively; The retrieval relevance of this data to the two data is determined by the co-occurrence frequency and the total occurrence frequency of each data. Based on the retrieval relevance of each data pair, a data relevance matrix is ​​obtained.

4. The method according to claim 1, characterized in that, The step of predicting the data correlation using a pre-trained correlation model is as follows: Construct a sample dataset, which includes multiple samples. Each sample includes a data feature vector matrix composed of data feature vectors of multiple data and a corresponding data correlation matrix. The sample dataset was divided into a training set and a validation set; Construct a correlation model with non-zero value focus; The association model is trained using the training set, and validated and optimized using the validation set.

5. The method according to claim 4, characterized in that, The association model includes a data feature encoder, a similarity detector, and a relevance strength predictor; the association model is trained using the training set, and the training method is as follows: The data feature vector matrix of the sample is encoded using a data feature encoder to generate low-dimensional data features; Calculate the spatial distance and temporal difference of all data pairs in the sample as spatial-temporal features; By integrating the feature differences and spatial and temporal features of data pairs, a comprehensive feature of the data pairs is formed; The similarity probability of data pairs is calculated based on the comprehensive features using a similarity detector; The correlation strength predictor outputs the basic strength of the data pairs based on the comprehensive features, and then weights them with similarity probability as the weight to obtain the preliminary correlation strength matrix of the data pairs. Set the diagonal elements of the preliminary correlation strength matrix to zero, and map the output range to a preset interval to obtain the data correlation matrix; Calculate the non-zero prediction loss based on the non-zero elements in the data correlation matrix, calculate the classification loss based on the target matrix and the sample labels, calculate the range constraint loss based on the non-zero prediction values ​​in the target matrix, and calculate the zero penalty loss based on the elements in the target matrix where the target is zero. A total loss value is calculated based on the non-zero prediction loss, classification loss, range constraint loss, and zero-value penalty loss, and the association model is optimized based on the total loss value.

6. The method according to claim 1, characterized in that, The steps for generating the data placement scheme are as follows: Let the set of data objects to be placed be business-related satellite data, and the set of satellite nodes be the satellite cluster nodes participating in the storage; Extract the data feature vectors from the satellite data to construct a data correlation matrix, which is then input into the correlation model M to predict the data correlation of the satellite data. The optimization objective in the genetic algorithm is set according to the requirements of the mission scenario, and the constraints in the genetic algorithm are set according to the characteristics of the satellite cluster. The fitness function used to evaluate the merits of the data placement scheme is constructed according to the data correlation of the satellite data. Each gene of the chromosome in the genetic algorithm corresponds to the storage allocation of a data object. The data placement scheme is generated using the genetic algorithm.

7. The method according to claim 6, characterized in that, A fitness function is constructed based on data correlation aggregation, load balancing, and constraint satisfaction.

8. The method according to claim 1, characterized in that, The steps for performing the data migration are as follows: Compare the current data placement plan with the latest received data placement plan to determine the data that needs to be moved in and out; For data that needs to be moved in, a request is sent to the data storage node in the latest data placement scheme to retrieve the data; For data that needs to be moved out, a request is sent to the target node to transfer the data to the target node.

9. The method according to claim 1, characterized in that, The satellite distributed cluster is a low-Earth orbit constellation network.

10. A data placement optimization system for satellite distributed clusters, characterized in that, It includes a preprocessing module, a correlation calculation module, a model building module, and a placement optimization module deployed on satellite distributed cluster computing nodes; Status acquisition module and data migration module deployed on the working nodes of the satellite distributed cluster; The status acquisition module is used to collect the execution status of data retrieval tasks on the current working node, generate historical retrieval logs, and obtain the status information and network topology of the satellite distributed cluster. The preprocessing module is used to extract data feature vectors from the historical retrieval logs; The correlation prediction module is used to input the data feature vector into a pre-trained correlation model to predict the data correlation. The placement scheme generation module is used to generate a data placement scheme based on the current status information and data correlation of the satellite distributed cluster. The data migration module is used to perform data migration according to the data placement scheme to achieve optimized data placement.

Images