A cross-border e-commerce trade data distributed storage management system and method

Abstract

The application belongs to the technical field of data storage management, and discloses a cross-border e-commerce trade data distributed storage management system and method; comprising: collecting cross-border e-commerce trade data; formulating data classification rules based on the cross-border e-commerce trade data, and dividing the cross-border e-commerce trade data into several data subsets according to the data classification rules; constructing a heterogeneous coding graph for each data subset, coloring the heterogeneous coding graph, and generating a corresponding heterogeneous coding data subset; distributing and storing the heterogeneous coding data subset in several heterogeneous storage nodes according to a preset distribution rule, grouping the heterogeneous storage nodes, and obtaining n heterogeneous storage node groups; and allocating storage space to each heterogeneous storage node group to complete distributed storage; the application not only optimizes the data management process, but also provides a more reliable data basis for deep data mining and business intelligence analysis.

Description

Technical Field

[0001] This invention relates to the field of data storage management technology, and more specifically, to a distributed storage management system and method for cross-border e-commerce trade data. Background Technology

[0002] With the acceleration of globalization and the development of Internet technology, the cross-border e-commerce industry has ushered in a period of vigorous development. According to statistics, it is expected to maintain an annual growth rate of over 20% in the next five years. Along with the expansion of transaction volume, the amount of data generated by cross-border e-commerce has experienced explosive growth, and the data of cross-border e-commerce has significant heterogeneity and complexity.

[0003] The cross-border e-commerce industry currently faces numerous challenges in data storage management. Due to the rapid development of cross-border e-commerce, data volume has exploded, and traditional centralized storage solutions can no longer meet the demands of large-scale data processing, leading to slow system response and severely impacting operational efficiency. Simultaneously, cross-border e-commerce data exhibits significant heterogeneity, including various types such as product information, transaction records, and logistics data. These data have complex interrelationships, and traditional storage systems lack effective data classification and organization mechanisms, resulting in data redundancy and management chaos. Regarding data security, existing systems do not adequately protect high-value data and sensitive information, easily leading to data leaks and security incidents, causing significant economic losses to enterprises. Furthermore, cross-border e-commerce data exhibits distinct geographical distribution characteristics, with significant differences in data access needs across different regions. Existing storage systems lack intelligent data distribution strategies and cannot specifically optimize data storage locations, resulting in high latency and costs for cross-regional data access. Existing solutions are also ill-suited to flexibly respond to rapid changes in business scale.

[0004] In view of this, the present invention proposes a distributed storage management system and method for cross-border e-commerce trade data to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a distributed storage management system for cross-border e-commerce trade data, comprising: a data acquisition module, a preliminary segmentation module, a comprehensive encoding module, and a distributed storage module; the modules are connected to each other via wired and / or wireless means;

[0006] The data acquisition module is used to collect cross-border e-commerce trade data;

[0007] The modules are initially divided, and data classification rules are formulated based on cross-border e-commerce trade data. The cross-border e-commerce trade data is then divided into several data subsets according to the data classification rules.

[0008] The integrated coding module is used to construct a heterogeneous coding graph for each data subset, color the heterogeneous coding graph, and generate the corresponding heterogeneous coded data subset.

[0009] The distributed storage module is used to distribute and store a subset of heterogeneous encoded data across several heterogeneous storage nodes according to a preset distribution rule, and to group the heterogeneous storage nodes into n heterogeneous storage node groups; and to allocate storage space to each heterogeneous storage node group to complete the distributed storage.

[0010] Furthermore, the methods for formulating data classification rules include:

[0011] Extract data features from cross-border e-commerce trade data. These features include product category, transaction amount, shipping location, delivery location, payment method, customer credit, product attributes, logistics method, order compliance, and time attribute. Standardize numerical data features and encode categorized data features to obtain standard data features.

[0012] All standard data features are constructed into a feature dataset. For the i-th standard data feature in the feature dataset, the feature dataset is divided into several subsets according to the value range of the standard data feature. The comprehensive entropy H(D) of each subset D is calculated, and the weighted average entropy is calculated according to the size of the subset. The original entropy of the feature dataset is also calculated. Based on the weighted average entropy of the subset and the original entropy of the feature dataset, the gain of the corresponding subset is calculated.

[0013] H(D)=-∑ j (w1×cost(j)+w2×sim(j,D))×p(j|D)×log2(p(j|D)); where j is the index of the category of the standard data feature in subset D, w1 and w2 are weighting parameters, cost(j) is the cost of misclassifying a data instance as category j; sim(j,D) is the average similarity between category j and data instances in subset D; and p(j|D) is the probability that a data instance in subset D belongs to category j.

[0014] Based on the calculated gain, the corresponding standard data features are sorted in descending order. From the sorted standard data features, the dimensions of the first L standard data features are selected as the initial dimensions. A grid space is constructed based on the initial dimensions. For each grid cell in the grid space, the number of data instances falling into it is counted. For each category, the number of data instances of the category in the corresponding cell is counted, and then the proportion of each category in the cell is calculated. Based on the calculated proportions, a data classification rule is fitted.

[0015] Furthermore, the method for fitting data classification rules based on the calculated proportions includes:

[0016] Calculate the rule quality score CN for the corresponding cell: CN = 1 - ∑ I (s_I×(p_I) 2 ); where p_I is the proportion of category I in the grid, and s_I is the average similarity score of data instances in the I-th category;

[0017] Calculate the rule quality score for each cell and select the cell with the smallest rule quality score as the initial rule. From the adjacent cells of the cell corresponding to the initial rule, select the cell with the smallest rule quality score and merge the selected cell into the initial rule to form a new rule. Repeat the selection and merging until the rule quality score no longer improves, and obtain the preliminary classification rules.

[0018] For the initial classification rules, calculate the change in rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to their condition contributions, with the smaller the condition contribution, the further back it is, to obtain a condition list; traverse the condition list from back to front, and remove conditions whose condition contributions are less than the preset condition threshold to obtain the data classification rules.

[0019] Furthermore, the method of constructing the lattice space based on the initial dimension includes:

[0020] For each dimension of the standard data feature, it is divided into a grids at equal intervals or with equal frequencies, and the grids are encoded. The grids of all dimensions of the standard data feature are combined by Cartesian product to generate all the grids in the grid space. Each grid is represented by m1 integers, which correspond to the grid numbers of the m1 dimensions of the standard data feature. That is, each dimension is divided into several intervals or grids, and each data record corresponds to one grid in the grid space.

[0021] Methods for dividing a feature dataset into several subsets include:

[0022] For each standard data feature, a Gaussian kernel is used to calculate its probability density function. A probability density function curve is constructed based on the probability density function. At the local peak points of the probability density function curve, the detected local peak points are used as dividing points to divide each standard data feature into several intervals. The intervals of all standard data features are combined using the Cartesian product method, and each combination corresponds to a subset.

[0023] Furthermore, the method of constructing a heterogeneous coding graph for each data subset includes:

[0024] For each data subset, iterate through each data instance and initialize a node graph; represent each data instance as a node in the node graph and use the corresponding unique identifier as the node label; calculate the fuzzy distance dist(x1,x2) between two data instances x1 and x2.

[0025] The calculation methods for the fuzzy distance dist(x1,x2) include:

[0026] Each data instance is considered as a fuzzy set, where different features correspond to different membership degrees; data instances x1 and x2 correspond to fuzzy sets x1' and x2', respectively; the membership degrees of x1' and x2' on the q-th feature are defined as x1'(q) and x2'(q), respectively.

[0027] Then fuzzy distance

[0028] For any two data instances x1 and x2 in the data subset, if the distance dist(x1,x2) between them is greater than 0, then add an edge connecting x1 and x2 in the node graph; the weight of the edge is set to the fuzzy distance between x1 and x2, thus completing the construction of the heterogeneous coding graph.

[0029] Furthermore, the method of coloring the heterogeneous coding graph to generate the corresponding heterogeneous coding data subset includes:

[0030] Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node initially; each membrane object contains the node's label information and the node's color state; all membrane objects constitute a membrane structure.

[0031] Based on the topology of the heterogeneous coding graph, the neighbor set of each membrane object is determined. That is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes. In the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor nodes. The color update rule of the membrane object is defined, and the color of the membrane object is iteratively updated based on the color update rule. In each iteration, the color state of the membrane object is updated. The iteration continues until the color state of all membrane objects is stable, that is, there are no more conflicts. Each membrane object is assigned a color state. The nodes corresponding to the membrane objects with the same color state are grouped together, and each group of nodes corresponds to a subset of heterogeneous coding data.

[0032] Furthermore, the defined color update rules for the membrane object, and the method of iteratively updating the color of the membrane object based on the color update rules, include:

[0033] Define a conflict if two neighboring nodes are colored with the same color state. Define a conflict counter for any membrane object. Traverse the neighbor list and compare the current color state of any membrane object with the color state of its neighboring nodes. If a conflict exists, increment the value of the corresponding conflict counter. This value is recorded as the number of conflicts for the corresponding membrane object.

[0034] Based on the number of conflicts within a membrane object, the activity level of that object is calculated. The formula for calculating the activity level is as follows:

[0035] GH_z=(1-exp(-k1×DH_z))×(1-exp(-k2×DS_z))×

[0036] (1+tanh(k3×(SQ_z-x0)))+(1+k4×sin(β2×π×LB_z); where k1, k2, k3, k4, x0, and β2 are all adjustable parameters, GH_z is the activity of membrane object z; DH_z is the number of conflicts of membrane object z; DS_z is the degree of the node corresponding to membrane object z; SQ_z is the attribute score of membrane object z; and LB_z is the average activity of the neighboring nodes of the node corresponding to membrane object z.

[0037] A preset activity threshold is set, and membrane objects with an activity level greater than the activity threshold are recorded as objects to be updated. The set of currently available color states is determined and denoted as C2. For each color state c' belonging to set C2, all neighboring nodes of the object to be updated are traversed, and the number of conflicts with neighboring nodes when updating its color state to c' is counted, i.e., the number of conflicts.

[0038] Find the color state that minimizes the number of conflicts in C2, denoted as c_min, and update the color state of the object to be updated to c_min; update the conflict counter and activity level accordingly.

[0039] Furthermore, the method of distributing and storing the heterogeneous coded data subset across several heterogeneous storage nodes according to a preset distribution rule includes:

[0040] The distribution rule is as follows: calculate the hash value based on any attribute of the heterogeneous encoded data subset, and map it to the corresponding heterogeneous storage node to determine the number N of heterogeneous storage nodes;

[0041] According to the distribution rules, calculate the corresponding hash value for each heterogeneous encoded data subset, and record it as the storage node number; store the mapping relationship between data subset encoding and storage node number in the routing table;

[0042] Traverse all heterogeneous encoded data subsets and, according to the routing table, transmit each heterogeneous encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes the data subset encoding, storage node information, and timestamp.

[0043] Furthermore, the method of grouping heterogeneous storage nodes includes:

[0044] Define a network coordinate structure and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the latency with the reference nodes; determine the coordinate position of each heterogeneous storage node.

[0045] Define the number of levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell. Calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes in the cell, the center point of the coordinate position of all heterogeneous storage nodes in the cell, and the variance of the coordinate position of the heterogeneous storage nodes in the cell relative to the center point.

[0046] The division index for each cell is calculated based on comprehensive statistical information; if the cell's division index is greater than the preset division threshold, it is divided into 2. d2 Each sub-cell is divided into several sub-cells, and the division is repeated until the level L1 is reached.

[0047] Define a hierarchy as the clustering hierarchy. At the defined clustering hierarchy, traverse all non-empty cells and mark each non-empty cell as an independent cluster, resulting in an initial n3 clusters, each cluster corresponding to one cell. For each pair of adjacent clusters, calculate their splitting index and find the difference between them. If the difference is less than a preset merging threshold, merge the pair of adjacent clusters into one cluster. Repeat this process until the difference calculated for all adjacent clusters is greater than or equal to the merging threshold, finally obtaining n heterogeneous storage node groups.

[0048] A method for distributed storage management of cross-border e-commerce trade data, which is based on the aforementioned distributed storage management system for cross-border e-commerce trade data, includes: S1, collecting cross-border e-commerce trade data;

[0049] S2. Develop data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0050] S3. Construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate the corresponding heterogeneous coded data subset;

[0051] S4. Distribute the heterogeneous encoded data subset into several heterogeneous storage nodes according to the preset distribution rules, and group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; allocate storage space to each heterogeneous storage node group to complete distributed storage.

[0052] The technical effects and advantages of the distributed storage management system and method for cross-border e-commerce trade data of this invention are as follows:

[0053] This invention significantly improves the processing efficiency and storage quality of massive heterogeneous data through refined data classification and coding strategies, effectively capturing subtle relationships and differences between data, laying a solid foundation for subsequent analysis. The construction of heterogeneous coding graphs preserves the inherent structure of the data, significantly improving the speed and accuracy of data retrieval. Based on storage node clustering methods, a more intelligent data distribution is achieved, effectively balancing system load and improving overall performance. Multi-layered data processing and storage strategies enhance system security and privacy protection capabilities. The distributed storage architecture has excellent scalability, flexibly responding to dynamic changes in data scale and meeting ever-growing storage demands. It effectively improves the utilization rate of storage resources, reduces the risk of system downtime, ensures the continuous and stable operation of business, and provides strong protection for the secure management of cross-border e-commerce data. Furthermore, it can flexibly handle cross-border e-commerce data of different scales and types. It not only optimizes the data management process but also provides a more reliable data foundation for deep data mining and business intelligence analysis, helping to formulate more accurate cross-border e-commerce strategies. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a distributed storage management system for cross-border e-commerce trade data according to the present invention;

[0055] Figure 2 This is a schematic diagram of a distributed storage management method for cross-border e-commerce trade data according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] Please see Figure 1As shown in this embodiment, a distributed storage management system for cross-border e-commerce trade data includes: a data acquisition module, a preliminary segmentation module, a comprehensive encoding module, and a distributed storage module; the modules are connected to each other via wired and / or wireless means to realize data transmission between modules;

[0059] The data acquisition module is used to collect cross-border e-commerce trade data;

[0060] The modules are initially divided, and data classification rules are formulated based on cross-border e-commerce trade data. The cross-border e-commerce trade data is then divided into several data subsets according to the data classification rules.

[0061] The integrated coding module is used to construct a heterogeneous coding graph for each data subset, color the heterogeneous coding graph, and generate the corresponding heterogeneous coded data subset.

[0062] The distributed storage module is used to distribute and store heterogeneous encoded data subsets across several heterogeneous storage nodes according to preset distribution rules, and to group the heterogeneous storage nodes into n heterogeneous storage node groups; n is an integer greater than 1; to allocate storage space to each heterogeneous storage node group to complete distributed storage; when a certain data subset needs to be accessed, the required data subset is retrieved and decoded from the corresponding heterogeneous storage node group according to the encoding characteristics of the required data subset.

[0063] Cross-border e-commerce trade data sources include e-commerce platforms and third-party data providers. Based on the characteristics of the data sources, corresponding data collection interfaces should be established, such as API interfaces, web crawlers, and database connections. The frequency of cross-border e-commerce trade data collection should be determined, such as real-time collection, hourly collection, or daily collection, to meet business needs. The scope of data collection should also be defined, such as collecting data from specific regions, specific product categories, or specific time periods. Data collection should be executed according to the set frequency and scope, obtaining raw cross-border e-commerce trade data from various data sources. Necessary preprocessing should be performed on the collected raw cross-border e-commerce trade data, such as deduplication, format conversion, and handling of missing values.

[0064] The methods for formulating data classification rules include:

[0065] Data features are extracted from cross-border e-commerce trade data. These features include product category, transaction amount, shipping location, receiving location, payment method, customer credit (buyer's past transaction completion rate and payment timeliness), product attributes (such as product value, size, weight, etc., affecting the ease of transportation and customs clearance), logistics method, order compliance, and time attributes. Time attributes include order creation time, payment time, shipping time, customs clearance time, delivery time, and final delivery time. Order creation time reflects the start of the order lifecycle; payment time reflects transaction timeliness; shipping time shows logistics processing speed; customs clearance time reflects customs clearance efficiency; delivery time reflects logistics timeliness; and final delivery time shows order completion status. Numerical data features are standardized, and categorical data features are coded to obtain standard data features.

[0066] All standard data features are constructed into a feature dataset. For the i-th standard data feature in the feature dataset, the feature dataset is divided into several subsets according to the value range of the standard data feature. The comprehensive entropy H(D) of each subset D is calculated, and the weighted average entropy is calculated according to the size of the subset (the amount of data). The original entropy of the feature dataset is also calculated. Based on the weighted average entropy of the subset and the original entropy of the feature dataset, the gain of the corresponding subset (original entropy minus weighted average entropy) is calculated.

[0067] H(D)=-∑ j (w1×cost(j)+w2×sim(j,D))×p(j|D)×log2(p(j|D)); where j is the index of the category of the standard data feature in subset D, w1 and w2 are weighting parameters used to control the weight between cost and similarity, cost(j) is the cost of misclassifying a data instance as category j, sim(j,D) is the average similarity between category j and data instances in subset D, and p(j|D) is the probability that a data instance in subset D belongs to category j.

[0068] It should be noted that a data instance typically contains multiple data features. Each cross-border e-commerce order is considered a data instance, which contains various feature information of that order. For example, a data instance might be: {Product Category: Electronics, Transaction Amount: $1,500, Shipping Location: China, Delivery Location: USA, Payment Method: Credit Card, Customer Credit: Good, Logistics Method: Air Freight, Order Date: 2023-05-01}.

[0069] Specifically, for any data instance x, define a cost matrix M, where the element M(I,j) represents the cost of incorrectly classifying a data instance x belonging to class I (different from j) as class j; then cost(j) = ∑ IM(I,j)×p(I); where p(I) is the prior probability of class I in the entire feature dataset.

[0070] The cost matrix M is defined as follows:

[0071] Setting a higher cost for misclassifying high-value goods (such as electronics and jewelry) as low-value goods, and a lower cost for misclassifying low-value goods as high-value goods, can reduce misclassification of high-value goods and avoid potential economic losses. Setting a higher cost for misclassifying high-value transactions as low-value transactions, and a lower cost for misclassifying low-value transactions as high-value transactions, can better identify and protect high-value transactions and prevent fraud. Setting a higher cost for misclassifying domestically shipped orders as internationally shipped orders, and a higher cost for misclassifying long-distance (such as intercontinental) orders as short-distance shipped orders.

[0072] The cost of misclassifying orders using high-risk payment methods such as credit cards as low-risk payment methods is set to a higher value; the cost of misclassifying orders from customers with good credit as orders from customers with poor credit is set to a higher value; the cost of misclassifying orders that may violate relevant regulations (such as prohibited items) as compliant orders is set to a higher value; both the higher and lower values ​​fluctuate between the range (0, 1) and are set according to the actual situation; based on the above aspects, a cost matrix is ​​constructed.

[0073] Based on the calculated gain, the corresponding standard data features are sorted in descending order. From the sorted standard data features, the dimensions of the first L standard data features are selected as the initial dimensions. A lattice space is constructed based on the initial dimensions. For each dimension of the standard data features, it is divided into a lattice cells at equal intervals or with equal frequencies. The lattice cells are encoded, for example, by using integer indices. The lattice cells of all dimensions of the standard data features are combined by Cartesian product to generate all the lattice cells in the lattice space. Each lattice cell is represented by m1 integers, corresponding to the cell numbers of the m1 standard data features in each dimension. That is, each dimension is equally divided into several intervals or lattices. This completes the construction of the lattice space.

[0074] Methods for dividing a feature dataset into several subsets include:

[0075] For each standard data feature, a Gaussian kernel is used to calculate its probability density function. A probability density function curve is constructed based on the probability density function. At the local peak points of the probability density function curve, the detected local peak points are used as dividing points to divide each standard data feature (corresponding data instance) into several intervals. The intervals of all standard data features are combined using the Cartesian product method, and each combination corresponds to a subset.

[0076] For each cell in the lattice space, count the number of data instances falling into it. For each category, count the number of data instances of that category in the corresponding cell, and then calculate the proportion of each category in the cell. Based on the calculated proportions, fit the data classification rules.

[0077] Specifically, the rule quality score CN for the corresponding grid is calculated as: CN = 1 - ∑ I (s_I×(p_I) 2 ); where p_I is the proportion of category I in the cell, and s_I is the average similarity score of data instances in category I (the number of combinations of summing the similarity scores of all data instances and dividing by the number of data instances); the smaller the rule quality score, the purer the data in the cell, and the more suitable it is as a rule.

[0078] Calculate the rule quality score for each cell, and select the cell with the smallest rule quality score as the initial rule. The feature condition combination corresponding to this cell is the initial rule. For example, if the selected cell corresponds to "Product Category = Electronic Products AND Transaction Amount ≥ 1000 AND Place of Shipment = Domestic", then the initial rule is this condition combination.

[0079] By following the steps above, the optimal cell can be selected from the cell space as the initial rule based on the data category distribution. This initial rule can effectively distinguish between different categories of data and is a good foundation for constructing classification rules.

[0080] From the adjacent cells of the cell corresponding to the initial rule, select the cell with the lowest rule quality score, merge the selected cell into the initial rule to form a new rule, and repeat the selection and merging until the rule quality score no longer improves, thus obtaining the preliminary classification rule.

[0081] The initial classification rule consists of several conditions. For example, for cross-border e-commerce data, we might generate a rule like this: "Product Category = Electronics AND Transaction Amount ≥ 1000 AND Shipping Location = Domestic". In this rule, "Product Category = Electronics" is one condition, "Transaction Amount ≥ 1000" is another condition, and "Shipping Location = Domestic" is the third condition. That is to say, if a data instance meets all three conditions, then according to this rule, this data instance can be classified as a high-value order.

[0082] For the initial classification rules, calculate the change in rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to their condition contributions, with the smaller the condition contribution, the further back it is, to obtain a condition list; traverse the condition list from back to front, and remove conditions whose condition contributions are less than the preset condition threshold to obtain the data classification rules.

[0083] Cross-border e-commerce trade data is classified according to the generated data classification rules, and each data classification rule corresponds to a data subset.

[0084] The methods for constructing a heterogeneous coding graph for each subset of data include:

[0085] For each data subset, iterate through each data instance and initialize a node graph; represent each data instance as a node in the node graph and use the corresponding unique identifier (such as ID) as the label of the node; calculate the fuzzy distance dist(x1,x2) between two data instances x1 and x2.

[0086] The calculation methods for the fuzzy distance dist(x1,x2) include:

[0087] Each data instance is considered as a fuzzy set, where different features correspond to different membership degrees; data instances x1 and x2 correspond to fuzzy sets x1' and x2', respectively; the membership degrees of x1' and x2' on the q-th feature are defined as x1'(q) and x2'(q), respectively.

[0088] Then fuzzy distance It should be explained that each data instance contains multiple features such as product category, transaction amount, shipping location, delivery location, payment method, and customer credit. For membership degree, numerical features, such as transaction amount, are assigned several membership degree levels. For example, [0-500] yuan corresponds to low membership degree of 0.2, (500-1000] yuan corresponds to medium membership degree of 0.5, and (1000-+∞) corresponds to high membership degree of 0.8. Therefore, a transaction of 1200 yuan has membership degrees of 0.2, 0.5, and 0.8 at the low, medium, and high membership degree levels, respectively.

[0089] For categorical features, such as product categories, membership values ​​are assigned to each category at different membership levels. For example, the membership of electronic products is 0.9 at the high-value level and 0.1 at the medium-value level. For geographical features, such as shipping location and receiving location, different membership values ​​are assigned based on the distance between the two locations, shipping costs, etc. For example, the membership of domestic shipments to domestic destinations is 0.9, and the membership of domestic shipments to overseas destinations is 0.3. Each data instance can obtain multiple membership values ​​for different features, forming a fuzzy set.

[0090] For any two data instances x1 and x2 in the data subset, if the distance dist(x1,x2) between them is greater than 0, then add an edge connecting x1 and x2 in the node graph, indicating that x1 and x2 differ in at least one feature; the weight of the edge is set to the fuzzy distance between x1 and x2; the larger the weight, the greater the difference between the two records, thus completing the construction of the heterogeneous coding graph.

[0091] The construction of heterogeneous coding graphs can effectively preserve the inherent differences in the data, laying the foundation for subsequent graph coloring and heterogeneous coding.

[0092] Methods for coloring heterogeneous coding graphs to generate corresponding heterogeneous coding data subsets include:

[0093] Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node initially. Each membrane object contains the node's label information (such as the unique identifier of the data instance) and the node's color state. All membrane objects constitute a membrane structure to simulate the graph coloring process.

[0094] Based on the topology of the heterogeneous coding graph, the neighbor set of each membrane object (node) is determined. That is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes. For the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor nodes. A color update rule for the membrane object is defined, and iterative color updates are performed based on this rule. Specifically, if two neighbor nodes are colored with the same color state, they are considered to have a conflict. For any membrane object, a conflict counter is defined. The neighbor list is traversed, and the current color state of any membrane object is compared with the color state of its neighbor nodes. If a conflict exists, the value of the corresponding conflict counter is incremented; this is recorded as the number of conflicts for the corresponding membrane object.

[0095] Based on the number of conflicts within a membrane object, the activity level of that object is calculated. The formula for calculating the activity level is as follows:

[0096] GH_z=(1-exp(-k1×DH_z))×(1-exp(-k2×DS_z))×

[0097] (1+tanh(k3×(SQ_z-x0)))+(1+k4×sin(β2×π×LB_z); where k1, k2, k3, k4, x0, and β2 are all adjustable parameters, DH_z is the number of conflicts in membrane object z, GH_z is the activity level of membrane object z, DS_z is the degree (normalized) of the node corresponding to membrane object z, SQ_z is the attribute score of membrane object z, and LB_z is the average activity level of the neighboring nodes of the node corresponding to membrane object z. In this process, when the number of conflicts and the degree are large, the value will decrease, thereby reducing the activity level. This is equivalent to non-linear scaling of the influence of the number of conflicts and the degree. When the average activity level of the neighbors is large, the value will fluctuate between positive and negative, causing a certain disturbance to the activity level.

[0098] A preset activity threshold is set. Membrane objects with an activity level greater than the threshold are marked as objects to be updated, and their current color state is updated. Specifically, the set of currently available color states is determined and denoted as C2. In general, C2 contains all possible color choices.

[0099] For each color state c' belonging to set C2, iterate through all neighboring nodes of the object to be updated, and count the number of conflicts with neighboring nodes when updating its color state to c', i.e., the number of conflicts; find the color state in C2 that minimizes the number of conflicts, denoted as c_min, and update the color state of the object to be updated to c_min; update other state information such as the conflict counter and activity level accordingly.

[0100] In each iteration, the color state of the membrane object is updated. The color states of the membrane objects are continuously updated. The iteration continues until the color states of all membrane objects are stable, that is, there are no more conflicts. Each membrane object is assigned a color state. The nodes (several data instances) corresponding to the membrane objects with the same color state are grouped together. Each group of nodes corresponds to a heterogeneous encoded data subset. Data instances in different groups are assigned different codes, thereby realizing heterogeneous encoding.

[0101] By simulating conflicts and coordination between nodes, the nodes (data instances) in the heterogeneous coding graph are finally colored and grouped to generate a subset of heterogeneous coded data, laying the foundation for subsequent distributed storage.

[0102] Methods for distributing heterogeneous coded data subsets across several heterogeneous storage nodes according to preset distribution rules include:

[0103] The distribution rule is as follows: calculate the hash value based on any attribute (such as ID) of the heterogeneous encoded data subset, and map it to the corresponding heterogeneous storage node. The number N of heterogeneous storage nodes is determined according to the data scale and available resources. A heterogeneous storage node can be a single machine or a small cluster.

[0104] According to the distribution rules, calculate the corresponding hash value for each heterogeneous encoded data subset, and record it as the storage node number; store the mapping relationship of "data subset encoding → storage node number" in the routing table, which can be distributed or centralized.

[0105] Traverse all heterogeneous encoded data subsets and, according to the routing table, transmit each heterogeneous encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes the data subset encoding, storage node information, and timestamp; the metadata information is used for data query, fault recovery, and other operations.

[0106] The methods for grouping heterogeneous storage nodes include:

[0107] Define a network coordinate structure (such as Vivaldi coordinates) and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the latency with the reference nodes. Determine the coordinate position (network coordinates) of each heterogeneous storage node.

[0108] Define the number of levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell. Calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell (represented by the mean or weighted mean of the coordinate positions), and the variance of the coordinate positions of the heterogeneous storage nodes in the cell relative to the center point.

[0109] The division index corresponding to the cell is calculated based on comprehensive statistical information (the total number of heterogeneous storage nodes contained in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell, and the weighted sum of the variances of the coordinate positions of the heterogeneous storage nodes in the cell and the center point).

[0110] If the cell's division criterion is greater than the preset division threshold, then divide it into 2. d2 Divide the cells into sub-cells (d2 is the spatial dimension), calculate the division index for each sub-cell, and repeat the division until the layer L1 is reached.

[0111] Define a level as the clustering level. Higher levels can obtain coarser-grained clusters with fewer clusters, while lower levels can obtain finer-grained clusters with more clusters. Choose an appropriate level based on actual needs (such as the number of clusters, memory usage, etc.). At the defined clustering level, traverse all non-empty cells and mark each non-empty cell as an independent cluster, resulting in an initial n3 clusters, each cluster corresponding to one cell; n3 is an integer greater than 1.

[0112] For each pair of adjacent clusters, calculate their partition index and the difference between them. If the difference is less than the preset merging threshold, merge the pair of adjacent clusters into one cluster. Repeat this process until the difference calculated for all adjacent clusters is greater than or equal to the merging threshold, and finally obtain n heterogeneous storage node groups.

[0113] This method efficiently divides heterogeneous storage nodes into n groups, where nodes in each group are geographically close and have similar statistical characteristics. This clustering based on location and statistical information can effectively group similar nodes together, laying the foundation for subsequent data distribution and query optimization.

[0114] This embodiment significantly improves the processing efficiency and storage quality of massive heterogeneous data through refined data classification and coding strategies, effectively capturing subtle relationships and differences between data, laying a solid foundation for subsequent analysis. The construction of heterogeneous coding graphs preserves the inherent structure of the data, significantly improving the speed and accuracy of data retrieval. Based on storage node clustering methods, a more intelligent data distribution is achieved, effectively balancing system load and improving overall performance. Multi-layered data processing and storage strategies enhance the system's security and privacy protection capabilities. The distributed storage architecture has excellent scalability, flexibly responding to dynamic changes in data scale and meeting ever-growing storage demands. It effectively improves the utilization rate of storage resources, reduces the risk of system downtime, ensures the continuous and stable operation of business, and provides strong protection for the secure management of cross-border e-commerce data. In addition, it can flexibly handle cross-border e-commerce data of different scales and types. It not only optimizes the data management process but also provides a more reliable data foundation for deep data mining and business intelligence analysis, helping to formulate more accurate cross-border e-commerce strategies.

[0115] Example 2

[0116] Please see Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description in Embodiment 1. A distributed storage management method for cross-border e-commerce trade data is provided, including:

[0117] S1. Collect cross-border e-commerce trade data;

[0118] S2. Develop data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0119] S3. Construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate the corresponding heterogeneous coded data subset;

[0120] S4. Distribute the heterogeneous encoded data subset into several heterogeneous storage nodes according to the preset distribution rules, and group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; allocate storage space to each heterogeneous storage node group to complete distributed storage.

[0121] Example 3

[0122] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operation mode of the distributed storage management method for cross-border e-commerce trade data described above.

[0123] Since the electronic device described in this embodiment is the electronic device used to implement the distributed storage management method for cross-border e-commerce trade data in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the distributed storage management method for cross-border e-commerce trade data described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the distributed storage management method for cross-border e-commerce trade data in this application embodiment falls within the scope of protection of this application.

[0124] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0125] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A distributed storage and management system for cross-border e-commerce trade data, characterized in that, include: The system comprises a data acquisition module, a preliminary segmentation module, a comprehensive encoding module, and a distributed storage module; these modules are connected via wired and / or wireless means. The data acquisition module is used to collect cross-border e-commerce trade data; The modules are initially divided, and data classification rules are formulated based on cross-border e-commerce trade data. The cross-border e-commerce trade data is then divided into several data subsets according to the data classification rules. The integrated coding module is used to construct a heterogeneous coding graph for each data subset, color the heterogeneous coding graph, and generate the corresponding heterogeneous coded data subset. The method for constructing a heterogeneous coding graph for each data subset includes: For each subset of data, iterate through each data instance and initialize a node graph; represent each data instance as a node in the node graph, using its corresponding unique identifier as the node's label; calculate the values ​​of two data instances. and fuzzy distance between ; Fuzzy distance The calculation methods include: Each data instance is viewed as a fuzzy set, where different features correspond to different membership degrees; data instance and Corresponding to fuzzy sets and ;definition and In the The membership degrees on each feature are respectively and ; Then fuzzy distance ; For any two data instances in the data subset and If the distance between them If the value is greater than 0, then add a connection to the node graph. and The edges; the edge weights are set to... and The fuzzy distance between them completes the construction of the heterogeneous coding graph; The method of coloring the heterogeneous coding graph to generate a corresponding heterogeneous coding data subset includes: Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node initially; each membrane object contains the node's label information and the node's color state; all membrane objects constitute a membrane structure. Based on the topology of the heterogeneous coding graph, the neighbor set of each membrane object is determined. That is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes. In the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor nodes. The color update rule of the membrane object is defined, and the color of the membrane object is iteratively updated based on the color update rule. In each iteration, the color state of the membrane object is updated. The iteration continues until the color state of all membrane objects is stable, that is, there are no more conflicts. Each membrane object is assigned a color state. The nodes corresponding to the membrane objects with the same color state are grouped together, and each group of nodes corresponds to a subset of heterogeneous coding data. The distributed storage module is used to distribute and store a subset of heterogeneous encoded data across several heterogeneous storage nodes according to a preset distribution rule, and to group the heterogeneous storage nodes into n heterogeneous storage node groups; and to allocate storage space to each heterogeneous storage node group to complete the distributed storage.

2. The distributed storage and management system for cross-border e-commerce trade data according to claim 1, characterized in that, The methods for formulating data classification rules include: Extract data features from cross-border e-commerce trade data. These features include product category, transaction amount, shipping location, delivery location, payment method, customer credit, product attributes, logistics method, order compliance, and time attribute. Standardize numerical data features and encode categorized data features to obtain standard data features. All standard data features are used to construct a feature dataset. For the i-th standard data feature in the feature dataset, the feature dataset is divided into several subsets according to the value range of the standard data feature; the calculation of each subset... Comprehensive entropy Calculate the weighted average entropy based on the size of the subset; calculate the original entropy of the feature dataset; and calculate the gain of the corresponding subset based on the weighted average entropy of the subset and the original entropy of the feature dataset. ;in, For subset Index of categories of standard data features and Assign parameters to the weights. To incorrectly classify a data instance into a category The price paid; For category with subset Average similarity of data instances in the dataset; In a subset In, it belongs to the category The probability of the data instance; Based on the calculated gain, the corresponding standard data features are sorted in descending order. From the sorted standard data features, the dimensions of the first L standard data features are selected as the initial dimensions. A grid space is constructed based on the initial dimensions. For each grid cell in the grid space, the number of data instances falling into it is counted. For each category, the number of data instances of the category in the corresponding cell is counted, and then the proportion of each category in the cell is calculated. Based on the calculated proportions, a data classification rule is fitted.

3. The distributed storage and management system for cross-border e-commerce trade data according to claim 2, characterized in that, The method of fitting data classification rules based on the calculated proportions includes: Calculate the rule quality score for the corresponding grid. ;in, For category The proportion it occupies in the grid, For the first The average similarity score of data instances in each category; Calculate the rule quality score for each cell and select the cell with the smallest rule quality score as the initial rule. From the adjacent cells of the cell corresponding to the initial rule, select the cell with the smallest rule quality score and merge the selected cell into the initial rule to form a new rule. Repeat the selection and merging until the rule quality score no longer improves, and obtain the preliminary classification rules. For the initial classification rules, calculate the change in rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to their condition contributions, with the smaller the condition contribution, the further back it is, to obtain a condition list; traverse the condition list from back to front, and remove conditions whose condition contributions are less than the preset condition threshold to obtain the data classification rules.

4. The distributed storage and management system for cross-border e-commerce trade data according to claim 3, characterized in that, The methods for constructing the lattice space based on the initial dimension include: For each dimension of the standard data feature, it is divided into a grids at equal intervals or with equal frequencies, and the grids are encoded. The grids of all dimensions of the standard data feature are combined by Cartesian product to generate all the grids in the grid space. Each grid is represented by m1 integers, which correspond to the grid numbers of the m1 dimensions of the standard data feature. That is, each dimension is divided into several intervals or grids, and each data record corresponds to one grid in the grid space. Methods for dividing a feature dataset into several subsets include: For each standard data feature, a Gaussian kernel is used to calculate its probability density function. A probability density function curve is constructed based on the probability density function. At the local peak points of the probability density function curve, the detected local peak points are used as dividing points to divide each standard data feature into several intervals. The intervals of all standard data features are combined using the Cartesian product method, and each combination corresponds to a subset.

5. A distributed storage management system for cross-border e-commerce trade data according to claim 4, characterized in that, The defined color update rules for membrane objects, and the methods for iteratively updating the color of membrane objects based on the color update rules, include: Define a conflict if two neighboring nodes are colored with the same color state. Define a conflict counter for any membrane object. Traverse the neighbor list and compare the current color state of any membrane object with the color state of its neighboring nodes. If a conflict exists, increment the value of the corresponding conflict counter. This value is recorded as the number of conflicts for the corresponding membrane object. Based on the number of conflicts within a membrane object, the activity level of that object is calculated. The formula for calculating the activity level is as follows: ;in, , , , , and All parameters are adjustable. For membrane objects Activity level; For membrane objects The number of conflicts For membrane objects The degree of the corresponding node, For membrane objects Attribute scores, For membrane objects The average activity level of the neighboring nodes of the corresponding node; A preset activity threshold is set, and membrane objects with an activity level greater than the activity threshold are recorded as objects to be updated. The set of currently available color states is determined and denoted as C2. For each color state c' belonging to set C2, all neighboring nodes of the object to be updated are traversed, and the number of conflicts with neighboring nodes when updating its color state to c' is counted, i.e., the number of conflicts. Find the color state that minimizes the number of conflicts in C2, denoted as c_min, and update the color state of the object to be updated to c_min; update the conflict counter and activity level accordingly.

6. A distributed storage and management system for cross-border e-commerce trade data according to claim 5, characterized in that, The method of distributing and storing heterogeneous coded data subsets across several heterogeneous storage nodes according to a preset distribution rule includes: The distribution rule is as follows: calculate the hash value based on any attribute of the heterogeneous encoded data subset, and map it to the corresponding heterogeneous storage node to determine the number N of heterogeneous storage nodes; According to the distribution rules, calculate the corresponding hash value for each heterogeneous encoded data subset, and record it as the storage node number; encode the data subset. The mapping relationship of storage node numbers is stored in the routing table; Traverse all heterogeneous encoded data subsets and, according to the routing table, transmit each heterogeneous encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes the data subset encoding, storage node information, and timestamp.

7. A distributed storage management system for cross-border e-commerce trade data according to claim 6, characterized in that, The methods for grouping heterogeneous storage nodes include: Define a network coordinate structure and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the latency with the reference nodes; determine the coordinate position of each heterogeneous storage node. Define the number of levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell. Calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes in the cell, the center point of the coordinate position of all heterogeneous storage nodes in the cell, and the variance of the coordinate position of the heterogeneous storage nodes in the cell relative to the center point. The cell's segmentation index is calculated based on comprehensive statistical information; if the cell's segmentation index is greater than a preset segmentation threshold, it is segmented as... Each sub-cell is divided into several sub-cells, and the division is repeated until the level L1 is reached. Define a hierarchy as the clustering hierarchy. At the defined clustering hierarchy, traverse all non-empty cells and mark each non-empty cell as an independent cluster, resulting in an initial n3 clusters, each cluster corresponding to one cell. For each pair of adjacent clusters, calculate their splitting index and find the difference between them. If the difference is less than a preset merging threshold, merge the pair of adjacent clusters into one cluster. Repeat this process until the difference calculated for all adjacent clusters is greater than or equal to the merging threshold, finally obtaining n heterogeneous storage node groups.

8. A method for distributed storage management of cross-border e-commerce trade data, implemented based on a distributed storage management system for cross-border e-commerce trade data as described in any one of claims 1 to 7, characterized in that, include: S1. Collect cross-border e-commerce trade data; S2. Develop data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules; S3. Construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate the corresponding heterogeneous coded data subset; S4. Distribute the heterogeneous encoded data subset into several heterogeneous storage nodes according to the preset distribution rules, and group the heterogeneous storage nodes to obtain n heterogeneous storage node groups. Distribute storage space to each heterogeneous storage node group to complete distributed storage.

Images