Cross-border e-commerce trade data distributed storage management system and method

Abstract

The invention belongs to the technical field of data storage management, and discloses a distributed storage management system and method for cross-border e-commerce trade data. Comprising the steps of collecting cross-border e-commerce trade data; formulating a data classification rule based on the cross-border e-commerce trade data, and dividing the cross-border e-commerce trade data into a plurality of data subsets according to the data classification rule; constructing a heterogeneous coding graph for each data subset, and coloring the heterogeneous coding graphs to generate a corresponding heterogeneous coding data subset; distributing and storing the heterogeneous coded data subsets in a plurality of heterogeneous storage nodes according to a preset distribution rule, and grouping the heterogeneous storage nodes to obtain n heterogeneous storage node groups; a storage space is allocated for each heterogeneous storage node group, and distributed storage is completed; the method not only optimizes the data management process, but also provides a more reliable data basis for deep data mining and business intelligence analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and more specifically, to a cross-border e-commerce trade data distributed storage management system and method. Background Art

[0002] With the acceleration of globalization and the development of Internet technology, the cross-border e-commerce industry has ushered in opportunities for vigorous development; according to statistics, it is expected to maintain an annual growth rate of more than 20% in the next five years; with the expansion of transaction scale, the amount of data generated by cross-border e-commerce has shown explosive growth, and the data of cross-border e-commerce has significant heterogeneity and complexity.

[0003] At present, the cross-border e-commerce industry faces many challenges in data storage management. Due to the rapid development of cross-border e-commerce business, the amount of data has exploded, and the traditional centralized storage solution can no longer meet the needs of large-scale data processing, resulting in slow system response and serious impact on business operation efficiency. At the same time, cross-border e-commerce data has significant heterogeneous characteristics, including commodity information, transaction records, logistics data and other types, and there are complex correlations between these data. The traditional storage system lacks effective data classification and organization mechanisms, resulting in data redundancy and management chaos. In terms of data security, the existing system does not provide sufficient protection for high-value data and sensitive information, which can easily lead to data leakage and security accidents, causing significant economic losses to enterprises. In addition, cross-border e-commerce data has obvious geographical distribution characteristics, and the data access needs in different regions vary greatly. The existing storage system lacks intelligent data distribution strategies and cannot optimize the data storage location in a targeted manner, resulting in high latency and high cost of cross-regional data access. Existing solutions are difficult to flexibly respond to rapid changes in business scale.

[0004] In view of this, the present invention proposes a cross-border e-commerce trade data distributed storage management system and method to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: A cross-border e-commerce trade data distributed storage management system, comprising: a data acquisition module, a preliminary division module, a comprehensive coding module and a distributed storage module; each module is connected by wire and / or wireless means;

[0006] Data collection module, used to collect cross-border e-commerce trade data;

[0007] A preliminary division module formulates data classification rules based on cross-border e-commerce trade data, and divides the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0008] The comprehensive coding module is used to construct a heterogeneous coding graph for each data subset, and to color the heterogeneous coding graph to generate the corresponding heterogeneous coding data subset;

[0009] The distributed storage module is used to store heterogeneous encoded data subsets in a number of heterogeneous storage nodes according to preset distribution rules, and to group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; storage space is allocated to each heterogeneous storage node group to complete distributed storage.

[0010] Furthermore, the method of formulating data classification rules includes:

[0011] Extract data features from cross-border e-commerce trade data, including commodity category, transaction amount, place of shipment, place of receipt, payment method, customer credit, commodity attributes, logistics method, order compliance, and time attributes; standardize numerical data features, encode categorical data features, and obtain standard data features;

[0012] All standard data features are constructed into a feature data set. For the i-th standard data feature in the feature data set, the feature data set is divided into several subsets according to the value range of the standard data feature; the comprehensive entropy H(D) of each subset D is calculated, and the weighted average entropy is calculated according to the size of the subset; and the original entropy of the feature data set is calculated; based on the weighted average entropy of the subset and the original entropy of the feature data set, the gain of the corresponding subset is calculated;

[0013] H(D)=-Σ j (w1×cost(j)+w2×sim(j,D))×p(j|D)×log2(p(j|D)); where,

[0014] j is the index of the category of the standard data feature in subset D, w1 and w2 are weight assignment parameters, cost(j) is the cost of misclassifying a data instance as category j; sim(j,D) is the average similarity between category j and data instances in subset D; p(j|D) is the probability of a data instance belonging to category j in subset D;

[0015] According to the calculated gains, the corresponding standard data features are sorted in descending order, and the dimensions of the first L standard data features are selected as the initial dimensions from the standard data features after descending order; a grid space is constructed based on the initial dimensions, and for each grid in the grid space, the number of data instances falling into it is counted, and for each category, the number of data instances of the category in the corresponding grid is counted, and then the proportion of each category in the grid is calculated; based on the calculated proportion, the data classification rules are fitted.

[0016] Furthermore, the method of fitting the data classification rule based on the calculated ratio includes:

[0017] Calculate the rule quality score of the corresponding grid CN = 1-∑ I (s_I×(p_I) 2 ; Where p_I is the proportion of category I in the grid, and s_I is the average similarity score of data instances in category I;

[0018] Calculate the rule quality score of each grid, and select the grid with the smallest rule quality score as the initial rule; select the grid with the smallest rule quality score from the adjacent grids of the grid corresponding to the initial rule, merge the selected grid into the initial rule to form a new rule, repeat the selection and merging until the rule quality score no longer increases, and obtain the preliminary classification rule;

[0019] For the preliminary classification rules, calculate the change in the rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to the condition contribution, the smaller the condition contribution, the later it is ranked, and get a condition list; traverse the condition list from back to front, remove the conditions whose condition contribution is less than the preset condition threshold, and get the data classification rules.

[0020] Furthermore, the method of constructing the grid space based on the initial dimension includes:

[0021] For each dimension of the standard data feature, divide it into a grids with equal distance or frequency, and encode the divided grids; Cartesian product is performed on the grids of all the dimensions of the standard data features to generate all the grids in the grid space; each grid is represented by m1 integer codes, which correspond to the numbers of the grids on the dimensions of the m1 standard data features; that is, each dimension is equally divided into several intervals or grids, and each data record corresponds to a grid in the grid space;

[0022] Ways to divide a feature dataset into several subsets include:

[0023] For each standard data feature, use the Gaussian kernel to calculate its probability density function, construct a probability density function curve based on its probability density function, and divide each standard data feature into several intervals at the local peak point of the probability density function curve with the detected local peak point as the segmentation point. Use the Cartesian product method to combine the intervals of all standard data features, and each combination corresponds to a subset.

[0024] Furthermore, the method of constructing a heterogeneous coding graph for each data subset includes:

[0025] For each data subset, traverse each data instance in it and initialize a node graph; represent each data instance as a node in the node graph, and use the corresponding unique identifier as the node label; calculate the fuzzy distance dist(x1,x2) between the two data instances x1 and x2;

[0026] The calculation method of the fuzzy distance dist(x1,x2) includes:

[0027] Each data instance is considered as a fuzzy set, where different features correspond to different memberships; data instances x1 and x2 correspond to fuzzy sets x1' and x2' respectively; the memberships of x1' and x2' on the qth feature are defined as x1'(q) and x2'(q) respectively;

[0028] The fuzzy distance

[0029] For any two data instances x1 and x2 in the data subset, if the distance between them dist(x1, x2) is greater than 0, an edge connecting x1 and x2 is added to the node graph; the weight of the edge is set to the fuzzy distance between x1 and x2, and the construction of the heterogeneous coding graph is completed.

[0030] Furthermore, the method of coloring the heterogeneous coding graph to generate a corresponding heterogeneous coding data subset includes:

[0031] Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node at the beginning; each membrane object contains the label information of the node and the color state of the node; all membrane objects constitute a membrane structure;

[0032] According to the topological structure of the heterogeneous coding graph, the neighbor set of each membrane object is determined, that is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes; in the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor nodes; the color update rule of the membrane object is defined, and the color of the membrane object is iteratively updated based on the color update rule. In each iteration, the color state of the membrane object is updated, and it is iterated until the color state of all membrane objects is stable, that is, there is no conflict. Each membrane object is assigned a color state, and the nodes corresponding to the membrane objects with the same color state are grouped together, and each group of nodes corresponds to a subset of heterogeneous coding data.

[0033] Furthermore, the method of defining a color update rule for the film object and iteratively updating the color of the film object based on the color update rule includes:

[0034] It is defined that if two neighboring nodes are colored with the same color state, there is a conflict between them. For any membrane object, a conflict counter is defined. The neighbor list is traversed, and the current color state of any membrane object is compared with the color state of the neighboring node. If there is a conflict, the value of the corresponding conflict counter is increased; recorded as the number of conflicts of the corresponding membrane object;

[0035] Based on the number of conflicts of the membrane object, the activity corresponding to the membrane object is calculated. The calculation formula of the activity is:

[0036] GH_z=(1-exp(-k1×DH_z))×(1-exp(-k2×DS_z))×(1+

[0037] tanh(k3×(SQ_z-x0)))+(1+k4×sin(β2×π×LB_z); where k1, k2, k3, k4, x0 and β2 are all adjustable parameters, GH_z is the activity of membrane object z; DH_z is the number of conflicts of membrane object z, DS_z is the degree of the node corresponding to membrane object z, SQ_z is the attribute score of membrane object z, and LB_z is the average activity of the neighboring nodes of the node corresponding to membrane object z;

[0038] Preset an activity threshold, record the membrane objects whose activity is greater than the activity threshold as the objects to be updated, determine the set of currently available color states, record it as C2; for each color state c' in set C2, traverse all neighbor nodes of the object to be updated, and count the number of conflicts with neighbor nodes when updating its color state to c', i.e., the number of conflicts;

[0039] Find the color state that minimizes the number of conflicts in C2, record it as c_min, and update the color state of the object to be updated to c_min; update the conflict counter and activity accordingly.

[0040] Furthermore, the method of distributing and storing the heterogeneous encoded data subsets in a plurality of heterogeneous storage nodes according to a preset distribution rule includes:

[0041] The distribution rule is: calculate the hash value according to any attribute of the heterogeneous encoded data subset, and map it to the corresponding heterogeneous storage node to determine the number N of heterogeneous storage nodes;

[0042] According to the distribution rule, the corresponding hash value is calculated for each heterogeneous encoded data subset and recorded as the storage node number; the mapping relationship of data subset encoding → storage node number is stored in the routing table;

[0043] Traverse all heterogeneously encoded data subsets, and according to the routing table, transmit each heterogeneously encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes data subset encoding, storage node information and timestamp.

[0044] Furthermore, the method of grouping heterogeneous storage nodes includes:

[0045] Define a network coordinate structure and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the delay with the reference nodes. Determine the coordinate position of each heterogeneous storage node.

[0046] Define the number of divided levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell, and calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes contained in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell, and the variance between the coordinate positions of the heterogeneous storage nodes in the cell and the center point;

[0047] Calculate the division index corresponding to the cell based on the comprehensive statistical information; if the division index of the cell is greater than the preset division threshold, divide it into 2 d2 sub-cells, calculate the division index of each sub-cell, and repeat the division until the number of layers L1 is reached;

[0048] Define a level as the clustering level. On the defined clustering level, traverse all non-empty cells and mark each non-empty cell as an independent cluster to obtain the initial n3 clusters, each cluster corresponding to a cell. For each pair of adjacent clusters, calculate their partitioning index and find the difference between the two. If the difference is less than the preset merge threshold, merge the pair of adjacent clusters into one cluster. Repeat until the calculated differences of all adjacent clusters are greater than or equal to the merge threshold, and finally obtain n heterogeneous storage node groups.

[0049] A cross-border e-commerce trade data distributed storage management method is implemented based on the cross-border e-commerce trade data distributed storage management system, comprising: S1, collecting cross-border e-commerce trade data;

[0050] S2. Formulate data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0051] S3, construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate a corresponding heterogeneous coding data subset;

[0052] S4. Distribute and store the heterogeneous encoded data subsets in a number of heterogeneous storage nodes according to a preset distribution rule, and group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; allocate storage space to each heterogeneous storage node group to complete distributed storage.

[0053] The technical effects and advantages of a cross-border e-commerce trade data distributed storage management system and method of the present invention are as follows:

[0054] The present invention greatly improves the processing efficiency and storage quality of massive heterogeneous data through refined data classification and coding strategies, effectively captures the subtle connections and differences between data, and lays a solid foundation for subsequent analysis; the construction of heterogeneous coding graphs preserves the intrinsic structure of the data, significantly improving the speed and accuracy of data retrieval; based on the storage node clustering method, a more intelligent data distribution is achieved, the system load is effectively balanced, and the overall performance is improved; the multi-level data processing and storage strategies enhance the security and privacy protection capabilities of the system, and the distributed storage architecture has excellent scalability, which can flexibly respond to dynamic changes in data scale and meet the growing storage needs; it effectively improves the utilization rate of storage resources, reduces the risk of system downtime, ensures the continuous and stable operation of the business, and provides a strong guarantee for the safe management of cross-border e-commerce data; in addition, it can flexibly respond to cross-border e-commerce data of different scales and types; it not only optimizes the data management process, but also provides a more reliable data foundation for deep data mining and business intelligence analysis, which is helpful to formulate more accurate cross-border e-commerce strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a cross-border e-commerce trade data distributed storage management system of the present invention;

[0056] Figure 2 A schematic diagram of a distributed storage management method for cross-border e-commerce trade data according to the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Example 1

[0059] See also Figure 1As shown, a cross-border e-commerce trade data distributed storage management system described in this embodiment includes: a data acquisition module, a preliminary division module, a comprehensive coding module and a distributed storage module; each module is connected by wired and / or wireless means to realize data transmission between modules;

[0060] Data collection module, used to collect cross-border e-commerce trade data;

[0061] A preliminary division module formulates data classification rules based on cross-border e-commerce trade data, and divides the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0062] The comprehensive coding module is used to construct a heterogeneous coding graph for each data subset, and to color the heterogeneous coding graph to generate the corresponding heterogeneous coding data subset;

[0063] A distributed storage module is used to store heterogeneous encoded data subsets in a number of heterogeneous storage nodes according to preset distribution rules, and to group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; n is an integer greater than 1; storage space is allocated to each heterogeneous storage node group to complete distributed storage; when a data subset needs to be accessed, the required data subset is retrieved and decoded from the corresponding heterogeneous storage node group according to the encoding characteristics of the required data subset.

[0064] The sources of cross-border e-commerce trade data include e-commerce platforms and third-party data providers. According to the characteristics of the data sources, establish corresponding data collection interfaces, such as API interfaces, web crawlers, database connections, etc.; determine the frequency of collecting cross-border e-commerce trade data, such as real-time collection, collection once an hour, collection once a day, etc., to meet business needs; and determine the scope of collected data, such as collecting data in a specific area, specific commodity categories, and specific time periods; according to the set frequency and data range, perform data collection, obtain original cross-border e-commerce trade data from various data sources, and perform necessary preprocessing on the collected original cross-border e-commerce trade data, such as deduplication, format conversion, missing value processing, etc.

[0065] Ways to develop data classification rules include:

[0066] Extract data features from cross-border e-commerce trade data, including commodity category, transaction amount, place of shipment, place of receipt, payment method, customer credit (buyer's past transaction completion rate, payment timeliness), commodity attributes (such as commodity value, size, weight, etc., which affect the difficulty of transportation and customs clearance), logistics method, order compliance and time attributes; time attributes include order creation time, payment time, shipment time, customs clearance time, delivery time and delivery time: order creation time reflects the starting point of the order life cycle; payment time reflects transaction timeliness; shipment time shows logistics processing speed; customs clearance time reflects customs clearance efficiency; delivery time reflects logistics timeliness; delivery time shows order completion status; and standardize numerical data features, encode categorized data features to obtain standard data features.

[0067] All standard data features are constructed into a feature data set. For the i-th standard data feature in the feature data set, the feature data set is divided into several subsets according to the value range of the standard data feature; the comprehensive entropy H(D) of each subset D is calculated, and the weighted average entropy is calculated according to the size of the subset (the amount of data); and the original entropy of the feature data set is calculated; based on the weighted average entropy of the subset and the original entropy of the feature data set, the gain of the corresponding subset is calculated (the original entropy minus the weighted average entropy);

[0068] H(D)=-∑ j (w1×cost(j)+w2×sim(j,D))×p(j|D)×log2(p(j|D)); where,

[0069] j is the index of the category of the standard data feature in subset D, w1 and w2 are weight allocation parameters used to control the weight between cost and similarity, cost(j) is the cost of misclassifying a data instance as category j; sim(j,D) is the average similarity between category j and data instances in subset D; p(j|D) is the probability of a data instance belonging to category j in subset D.

[0070] It should be noted that a data instance usually contains multiple data features. Each cross-border e-commerce order data is regarded as a data instance, which contains various feature information of the order; for example, a data instance may be, {product category: electronic products, transaction amount: US$1,500, shipping place: China, receiving place: the United States, payment method: credit card, customer credit: good, logistics method: air transport, order time: 2023-05-01}.

[0071] Specifically, for any data instance x, a cost matrix M is defined, where the element M(I, j) represents the cost of misclassifying a data instance x belonging to category I (different from j) as category j; then cost(j) = ∑ I M(I,j)×p(I); where p(I) is the prior probability of category I in the entire feature data set.

[0072] The definition process of the cost matrix M is as follows:

[0073] Set the cost of misclassifying high-value goods (such as electronics, jewelry, etc.) as low-value goods to a higher value, and set the cost of misclassifying low-value goods as high-value goods to a lower value; this can reduce the misclassification of high-value goods and avoid potential economic losses; set the cost of misclassifying high-value transactions as low-value transactions to a higher value, and set the cost of misclassifying low-value transactions as high-value transactions to a lower value, so as to better identify and protect high-value transactions and prevent fraud, set the cost of misclassifying domestic orders as foreign orders to a higher value, and set the cost of misclassifying long-distance transportation orders (such as across continents) as short-distance transportation to a higher value.

[0074] The cost of misclassifying orders using high-risk payment methods such as credit cards as low-risk payment methods is set to a higher value; the cost of misclassifying orders from customers with good credit as orders from customers with poor credit is set to a higher value; the cost of misclassifying orders that may violate relevant regulations (such as prohibited items) as compliant orders is set to a higher value; both higher and lower values ​​fluctuate between the interval (0, 1) and are set according to actual conditions; based on the above aspects, a cost matrix is ​​constructed.

[0075] According to the calculated gains, the corresponding standard data features are sorted in descending order, and the dimensions of the first L standard data features are selected as the initial dimensions from the standard data features after descending order; a grid space is constructed based on the initial dimensions, and for each dimension of a standard data feature, it is equally spaced or equally frequent divided into a grids, and the divided grids are encoded, such as represented by integer serial numbers; the grids of the dimensions of all standard data features are combined by Cartesian products to generate all grids in the grid space; each grid is represented by m1 integer codes, which correspond to the numbers of the grids on the dimensions of the m1 standard data features; that is, each dimension is equally divided into a number of intervals or grids; that is, the construction of the grid space is completed.

[0076] Ways to divide a feature dataset into several subsets include:

[0077] For each standard data feature, use the Gaussian kernel to calculate its probability density function, construct a probability density function curve based on its probability density function, and divide each standard data feature (corresponding data instance) into several intervals at the local peak point of the probability density function curve, using the detected local peak point as the segmentation point. Use the Cartesian product method to combine the intervals of all standard data features, and each combination corresponds to a subset.

[0078] For each grid in the grid space, count the number of data instances falling into it. For each category, count the number of data instances of the category in the corresponding grid, and then calculate the proportion of each category in the grid; based on the calculated proportion, fit the data classification rules.

[0079] Specifically, calculate the rule quality score CN of the corresponding grid = 1-∑ I (s_I×(p_I) 2 ; Where p_I is the proportion of category I in the grid, s_I is the average similarity score of data instances in the I-th category (the similarity of all data instances is calculated pairwise, and the sum is divided by the number of combinations of the number of data instances); the smaller the rule quality score, the purer the data in the grid, and the more suitable it is as a rule.

[0080] Calculate the rule quality score of each grid, and select the grid with the smallest rule quality score as the initial rule. The characteristic condition combination corresponding to the grid is the initial rule. For example, if the selected grid corresponds to "product category = electronic products AND transaction amount ≥ 1000 AND place of shipment = domestic", the initial rule is this condition combination.

[0081] Through the above steps, the optimal grid can be selected as the initial rule from the grid space according to the category distribution of the data; this initial rule can well distinguish different categories of data and is a good basis for building classification rules.

[0082] From the adjacent grids of the grid corresponding to the initial rule, select the grid with the smallest rule quality score, merge the selected grid into the initial rule to form a new rule, repeat the selection and merging until the rule quality score no longer improves, and obtain the preliminary classification rule.

[0083] The preliminary classification rules are composed of several conditions. For example, for cross-border e-commerce data, we may generate a rule like this: "Product Category = Electronic Products AND Transaction Amount ≥ 1000 AND Place of Shipping = Domestic" In this rule, "Product Category = Electronic Products" is a condition, "Transaction Amount ≥ 1000" is another condition, and "Place of Shipping = Domestic" is the third condition; that is, if a data instance meets all three conditions at the same time, then according to this rule, this data instance can be classified as a high-value order.

[0084] For the preliminary classification rules, calculate the change in the rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to the condition contribution, the smaller the condition contribution, the later it is ranked, and get a condition list; traverse the condition list from back to front, remove the conditions whose condition contribution is less than the preset condition threshold, and get the data classification rules.

[0085] The cross-border e-commerce trade data is classified according to the generated data classification rules, and each corresponding data classification rule corresponds to a data subset.

[0086] The methods for constructing heterogeneous encoding graphs for each data subset include:

[0087] For each data subset, traverse each data instance in it and initialize a node graph; represent each data instance as a node in the node graph, and use the corresponding unique identifier (such as ID) as the node label; calculate the fuzzy distance dist(x1,x2) between the two data instances x1 and x2;

[0088] The calculation method of the fuzzy distance dist(x1,x2) includes:

[0089] Each data instance is considered as a fuzzy set, where different features correspond to different memberships; data instances x1 and x2 correspond to fuzzy sets x1' and x2' respectively; the memberships of x1' and x2' on the qth feature are defined as x1'(q) and x2'(q) respectively;

[0090] The fuzzy distance It should be explained that each data instance contains multiple features such as product category, transaction amount, shipping place, receiving place, payment method, customer credit, etc. For membership, numerical features such as transaction amount are set to several membership levels, such as [0-500] yuan corresponds to a low membership of 0.2, (500-1000] yuan corresponds to a medium membership of 0.5, and (1000-+∞) corresponds to a high membership of 0.8. For a transaction of 1,200 yuan, the memberships at the low, medium, and high levels are 0.2, 0.5, and 0.8 respectively.

[0091] For categorical features, such as commodity categories, the membership values ​​of each category at different membership levels are specified, such as the membership of electronic products at high value levels is 0.9, and the membership at medium value levels is 0.1. For geographical location features, such as the place of shipment and the place of receipt, different membership values ​​are assigned according to the distance between the two places, freight costs, etc.; for example, the membership of domestic shipments to domestic shipments is 0.9, and the membership of domestic shipments to overseas shipments is 0.3; each data instance can obtain multiple membership values ​​on different features, forming a fuzzy set.

[0092] For any two data instances x1 and x2 in the data subset, if the distance between them dist(x1, x2) is greater than 0, an edge connecting x1 and x2 is added to the node graph, indicating that x1 and x2 differ in at least one feature; the weight of the edge is set to the fuzzy distance between x1 and x2; the larger the weight, the greater the difference between the two records, and the construction of the heterogeneous coding graph is completed.

[0093] The construction of heterogeneous coding graphs can well preserve the intrinsic difference information of the data, laying the foundation for subsequent graph coloring and heterogeneous coding.

[0094] The methods of coloring the heterogeneous coding graph and generating the corresponding heterogeneous coding data subsets include:

[0095] Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node initially; each membrane object contains the node's label information (such as the unique identifier of the data instance) and the node's color state; all membrane objects constitute a membrane structure, which is used to simulate the graph coloring process.

[0096] According to the topological structure of the heterogeneous coding graph, the neighbor set of each membrane object (node) is determined, that is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes; in the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor node; the color update rule of the membrane object is defined, and the color update is iterated based on the color update rule. Specifically, it is defined that if two neighbor nodes are colored with the same color state, there is a conflict between them. For any membrane object, a conflict counter is defined, and the neighbor list is traversed to compare the current color state of any membrane object with the color state of the neighbor node. If there is a conflict, the value of the corresponding conflict counter is increased; it is recorded as the number of conflicts of the corresponding membrane object.

[0097] Based on the number of conflicts of the membrane object, the activity corresponding to the membrane object is calculated. The calculation formula of the activity is:

[0098] GH_z=(1-exp(-k1×DH_z))×(1-exp(-k2×DS_z))×(1+

[0099] tanh(k3×(SQ_z-x0)))+(1+k4×sin(β2×π×LB_z); where k1, k2, k3, k4, x0 and β2 are all adjustable parameters, DH_z is the number of conflicts of membrane object z, GH_z is the activity of membrane object z, DS_z is the degree of the node corresponding to membrane object z (normalized), SQ_z is the attribute score of membrane object z, and LB_z is the mean activity of the neighboring nodes of the node corresponding to membrane object z. In this process, when the number and degree of conflicts are large, the value will become smaller, thereby reducing the activity, which is equivalent to a nonlinear scaling of the impact of the number and degree of conflicts. When the mean activity of the neighbors is large, the value will fluctuate between positive and negative, causing a certain disturbance to the activity.

[0100] An activity threshold is preset, and the membrane objects whose activity is greater than the activity threshold are recorded as objects to be updated, and their current color states are updated. Specifically, a set of currently available color states is determined, which is recorded as C2; generally, C2 includes all possible color choices.

[0101] For each color state c' in set C2, traverse all neighbor nodes of the object to be updated, and count the number of conflicts with neighbor nodes when its color state is updated to c', that is, the number of conflicts; find the color state with the smallest number of conflicts in C2, recorded as c_min, and update the color state of the object to be updated to c_min; update other state information such as conflict counter and activity accordingly.

[0102] In each iteration, the color state of the membrane object is updated, and the color state of the membrane objects is continuously updated until the color state of all membrane objects is stable, that is, there is no conflict. Each membrane object is assigned a color state, and the nodes (several data instances) corresponding to the membrane objects with the same color state are grouped together. Each group of nodes corresponds to a heterogeneous coded data subset, and data instances between different groups are assigned different codes, thereby realizing heterogeneous coding.

[0103] By simulating the conflicts and coordination between nodes, the nodes (data instances) in the heterogeneous coding graph are eventually colored and grouped to generate heterogeneous coding data subsets, laying the foundation for subsequent distributed storage.

[0104] The method of distributing and storing heterogeneous encoded data subsets in a plurality of heterogeneous storage nodes according to a preset distribution rule includes:

[0105] The distribution rule is: calculate the hash value according to any attribute (such as ID) of the heterogeneous encoded data subset and map it to the corresponding heterogeneous storage node. Determine the number N of heterogeneous storage nodes based on the data scale and available resources. The heterogeneous storage node can be a single machine or a small cluster.

[0106] According to the distribution rules, the corresponding hash value is calculated for each heterogeneous encoded data subset and recorded as the storage node number; the mapping relationship of "data subset encoding→storage node number" is stored in the routing table, which can be distributed or centralized.

[0107] Traverse all heterogeneously encoded data subsets, and according to the routing table, transfer each heterogeneously encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes the data subset encoding, storage node information and timestamp; the metadata information is used for data query, fault recovery and other operations.

[0108] Ways to group heterogeneous storage nodes include:

[0109] Define a network coordinate structure (such as Viva ldi coordinates) and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the delay with the reference nodes; determine the coordinate position (network coordinates) of each heterogeneous storage node.

[0110] Define the number of divided levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell, and calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes contained in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell (expressed by the mean or weighted mean of the coordinate positions), and the variance between the coordinate positions of the heterogeneous storage nodes in the cell and the center point;

[0111] The partitioning index corresponding to the cell (the total number of heterogeneous storage nodes contained in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell, and the weighted sum of the variances between the coordinate positions of the heterogeneous storage nodes in the cell and the center point) is calculated based on the comprehensive statistical information.

[0112] If the cell's partition index is greater than the preset partition threshold, it will be divided into 2 d2 sub-cells (d2 is the spatial dimension), calculate the division index of each sub-cell, and repeat the division until the number of layers L1 is reached.

[0113] Define a level as the clustering level. A higher level can obtain coarser-grained clustering with a smaller number of clusters, while a lower level can obtain finer-grained clustering with a larger number of clusters. Select the appropriate level based on actual needs (such as the number of clusters, memory usage, etc.); on the defined clustering level, traverse all non-empty cells, mark each non-empty cell as an independent cluster, and obtain the initial n3 clusters, each cluster corresponding to a cell; n3 is an integer greater than 1.

[0114] For each pair of adjacent clusters, their partitioning indices are calculated and the difference between the two is obtained. If the difference is less than the preset merge threshold, the pair of adjacent clusters are merged into one cluster. Repeat this process until the differences calculated for all adjacent clusters are greater than or equal to the merge threshold, and finally n heterogeneous storage node groups are obtained.

[0115] Efficiently divide heterogeneous storage nodes into n groups, with nodes in each group having similar locations and statistical characteristics. This clustering based on location and statistical information can effectively group similar nodes together, laying the foundation for subsequent data distribution and query optimization.

[0116] This embodiment, through refined data classification and coding strategies, greatly improves the processing efficiency and storage quality of massive heterogeneous data, effectively captures the subtle connections and differences between data, and lays a solid foundation for subsequent analysis; the construction of heterogeneous coding graphs preserves the intrinsic structure of the data, significantly improving the speed and accuracy of data retrieval; based on the storage node clustering method, a more intelligent data distribution is achieved, the system load is effectively balanced, and the overall performance is improved; the multi-level data processing and storage strategies enhance the security and privacy protection capabilities of the system, and the distributed storage architecture has excellent scalability, which can flexibly respond to dynamic changes in data scale and meet the growing storage needs; it effectively improves the utilization rate of storage resources, reduces the risk of system downtime, ensures the continuous and stable operation of the business, and provides a strong guarantee for the security management of cross-border e-commerce data; in addition, it can flexibly respond to cross-border e-commerce data of different scales and types; it not only optimizes the data management process, but also provides a more reliable data foundation for deep data mining and business intelligence analysis, which helps to formulate more accurate cross-border e-commerce strategies.

[0117] Example 2

[0118] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a distributed storage management method for cross-border e-commerce trade data, including:

[0119] S1. Collect cross-border e-commerce trade data;

[0120] S2. Formulate data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules;

[0121] S3, construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate a corresponding heterogeneous coding data subset;

[0122] S4. Distribute and store the heterogeneous encoded data subsets in a number of heterogeneous storage nodes according to a preset distribution rule, and group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; allocate storage space to each heterogeneous storage node group to complete distributed storage.

[0123] Example 3

[0124] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the above-mentioned distributed storage management method for cross-border e-commerce trade data is implemented.

[0125] Since the electronic device introduced in this embodiment is an electronic device used to implement a cross-border e-commerce trade data distributed storage management method in the embodiment of this application, based on the cross-border e-commerce trade data distributed storage management method introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as the technical personnel of this field implement the electronic device used in the cross-border e-commerce trade data distributed storage management method in the embodiment of this application, it belongs to the scope of protection of this application.

[0126] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0127] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A distributed storage management system for cross-border e-commerce trade data, characterized in that: include: Data acquisition module, preliminary division module, comprehensive coding module and distributed storage module; each module is connected by wired and / or wireless means; Data collection module, used to collect cross-border e-commerce trade data; A preliminary division module formulates data classification rules based on cross-border e-commerce trade data, and divides the cross-border e-commerce trade data into several data subsets according to the data classification rules; The comprehensive coding module is used to construct a heterogeneous coding graph for each data subset, and to color the heterogeneous coding graph to generate the corresponding heterogeneous coding data subset; The distributed storage module is used to store heterogeneous encoded data subsets in a number of heterogeneous storage nodes according to preset distribution rules, and to group the heterogeneous storage nodes to obtain n heterogeneous storage node groups; storage space is allocated to each heterogeneous storage node group to complete distributed storage.

2. A cross-border e-commerce trade data distributed storage management system according to claim 1, characterized in that: The method of formulating data classification rules includes: Extract data features from cross-border e-commerce trade data, including commodity category, transaction amount, place of shipment, place of receipt, payment method, customer credit, commodity attributes, logistics method, order compliance, and time attributes; standardize numerical data features, encode categorical data features, and obtain standard data features; All standard data features are constructed into a feature data set. For the i-th standard data feature in the feature data set, the feature data set is divided into several subsets according to the value range of the standard data feature; the comprehensive entropy H(D) of each subset D is calculated, and the weighted average entropy is calculated according to the size of the subset; and the original entropy of the feature data set is calculated; based on the weighted average entropy of the subset and the original entropy of the feature data set, the gain of the corresponding subset is calculated; H(D)=-Σ j (w1×cost(j)+w2×sim(j,D))×p(j|D)×log2(p(j|D)); among them, j is the index of the category of the standard data feature in subset D, w1 and w2 are weight assignment parameters, cost(j) is the cost of misclassifying a data instance as category j; sim(j,D) is the average similarity between category j and data instances in subset D; p(j|D) is the probability of a data instance belonging to category j in subset D; According to the calculated gains, the corresponding standard data features are sorted in descending order, and the dimensions of the first L standard data features are selected as the initial dimensions from the standard data features after descending order; a grid space is constructed based on the initial dimensions, and for each grid in the grid space, the number of data instances falling into it is counted, and for each category, the number of data instances of the category in the corresponding grid is counted, and then the proportion of each category in the grid is calculated; based on the calculated proportion, the data classification rules are fitted.

3. A cross-border e-commerce trade data distributed storage management system according to claim 2, characterized in that: The method of fitting the data classification rule based on the calculated ratio includes: Calculate the rule quality score of the corresponding grid CN = 1-∑ I (s_I×(p_I) 2 ; Where p_I is the proportion of category I in the grid, and s_I is the average similarity score of data instances in category I; Calculate the rule quality score of each grid, and select the grid with the smallest rule quality score as the initial rule; select the grid with the smallest rule quality score from the adjacent grids of the grid corresponding to the initial rule, merge the selected grid into the initial rule to form a new rule, repeat the selection and merging until the rule quality score no longer increases, and obtain the preliminary classification rule; For the preliminary classification rules, calculate the change in the rule quality score after removing each condition, and record it as the condition contribution of the corresponding condition; sort the conditions according to the condition contribution, the smaller the condition contribution, the later it is ranked, and get a condition list; traverse the condition list from back to front, remove the conditions whose condition contribution is less than the preset condition threshold, and get the data classification rules.

4. A cross-border e-commerce trade data distributed storage management system according to claim 3, characterized in that: The method of constructing the grid space based on the initial dimension includes: For each dimension of the standard data feature, divide it into a grids with equal distance or frequency, and encode the divided grids; Cartesian product is performed on the grids of all the dimensions of the standard data features to generate all the grids in the grid space; each grid is represented by m1 integer codes, which correspond to the numbers of the grids on the dimensions of the m1 standard data features; that is, each dimension is equally divided into several intervals or grids, and each data record corresponds to a grid in the grid space; Ways to divide a feature dataset into several subsets include: For each standard data feature, use the Gaussian kernel to calculate its probability density function, construct a probability density function curve based on its probability density function, and divide each standard data feature into several intervals at the local peak point of the probability density function curve with the detected local peak point as the segmentation point. Use the Cartesian product method to combine the intervals of all standard data features, and each combination corresponds to a subset.

5. A cross-border e-commerce trade data distributed storage management system according to claim 4, characterized in that: The method of constructing a heterogeneous coding graph for each data subset includes: For each data subset, traverse each data instance in it and initialize a node graph; represent each data instance as a node in the node graph, and use the corresponding unique identifier as the node label; calculate the fuzzy distance dist(x1,x2) between the two data instances x1 and x2; The calculation method of the fuzzy distance dist(x1,x2) includes: Each data instance is considered as a fuzzy set, where different features correspond to different memberships; data instances x1 and x2 correspond to fuzzy sets x1' and x2' respectively; the memberships of x1' and x2' on the qth feature are defined as x1'(q) and x2'(q) respectively; The fuzzy distance For any two data instances x1 and x2 in the data subset, if the distance between them dist(x1, x2) is greater than 0, an edge connecting x1 and x2 is added to the node graph; the weight of the edge is set to the fuzzy distance between x1 and x2, and the construction of the heterogeneous coding graph is completed.

6. A cross-border e-commerce trade data distributed storage management system according to claim 5, characterized in that: The method of coloring the heterogeneous coding graph to generate the corresponding heterogeneous coding data subset includes: Each node in the heterogeneous coding graph is represented as a membrane object, and a color state is randomly assigned to each node at the beginning; each membrane object contains the label information of the node and the color state of the node; all membrane objects constitute a membrane structure; According to the topological structure of the heterogeneous coding graph, the neighbor set of each membrane object is determined, that is, if two nodes are connected by an edge in the heterogeneous coding graph, they are each other's neighbor nodes; in the color state of the membrane object, a neighbor list is maintained to store the label information and color state of the neighbor nodes; the color update rule of the membrane object is defined, and the color of the membrane object is iteratively updated based on the color update rule. In each iteration, the color state of the membrane object is updated, and it is iterated until the color state of all membrane objects is stable, that is, there is no conflict. Each membrane object is assigned a color state, and the nodes corresponding to the membrane objects with the same color state are grouped together, and each group of nodes corresponds to a subset of heterogeneous coding data.

7. A cross-border e-commerce trade data distributed storage management system according to claim 6, characterized in that: The method of defining the color update rule of the film object and iteratively updating the color of the film object based on the color update rule includes: It is defined that if two neighboring nodes are colored with the same color state, there is a conflict between them. For any membrane object, a conflict counter is defined. The neighbor list is traversed, and the current color state of any membrane object is compared with the color state of the neighboring node. If there is a conflict, the value of the corresponding conflict counter is increased; recorded as the number of conflicts of the corresponding membrane object; Based on the number of conflicts of the membrane object, the activity of the membrane object is calculated. The activity calculation formula is: GH_z=(1-exp(-k1×DH_z))×(1-exp(-k2×DS_z))×(1+ tanh(k3×(SQ_z-x0)))+(1+k4×sin(β2×π×LB_z); where k1, k2, k3, k4, x0 and β2 are all adjustable parameters, GH_z is the activity of membrane object z; DH_z is the number of conflicts of membrane object z, DS_z is the degree of the node corresponding to membrane object z, SQ_z is the attribute score of membrane object z, and LB_z is the average activity of the neighboring nodes of the node corresponding to membrane object z; Preset an activity threshold, record the membrane objects whose activity is greater than the activity threshold as the objects to be updated, determine the set of currently available color states, record it as C2; for each color state c' in set C2, traverse all neighbor nodes of the object to be updated, and count the number of conflicts with neighbor nodes when updating its color state to c', i.e., the number of conflicts; Find the color state that minimizes the number of conflicts in C2, record it as c_min, and update the color state of the object to be updated to c_min; update the conflict counter and activity accordingly.

8. A cross-border e-commerce trade data distributed storage management system according to claim 7, characterized in that: The method of distributing and storing the heterogeneous encoded data subsets in a plurality of heterogeneous storage nodes according to a preset distribution rule includes: The distribution rule is: calculate the hash value according to any attribute of the heterogeneous encoded data subset, and map it to the corresponding heterogeneous storage node to determine the number N of heterogeneous storage nodes; According to the distribution rule, the corresponding hash value is calculated for each heterogeneous encoded data subset and recorded as the storage node number; the mapping relationship of data subset encoding → storage node number is stored in the routing table; Traverse all heterogeneously encoded data subsets, and according to the routing table, transmit each heterogeneously encoded data subset to the corresponding heterogeneous storage node; during the transmission process, update the corresponding metadata information; the metadata information includes data subset encoding, storage node information and timestamp.

9. A cross-border e-commerce trade data distributed storage management system according to claim 8, characterized in that: The method of grouping heterogeneous storage nodes includes: Define a network coordinate structure and deploy a set of fixed reference nodes. The coordinate positions of the reference nodes are known. Other heterogeneous storage nodes calculate their own coordinate positions by measuring the delay with the reference nodes. Determine the coordinate position of each heterogeneous storage node. Define the number of divided levels L1 and the number of cells in each level, and divide the entire data space where the heterogeneous storage nodes are located into a top-level cell, and calculate the comprehensive statistical information of all heterogeneous storage nodes in the cell. The comprehensive statistical information includes the total number of heterogeneous storage nodes contained in the cell, the center point of the coordinate positions of all heterogeneous storage nodes in the cell, and the variance between the coordinate positions of the heterogeneous storage nodes in the cell and the center point; Calculate the division index corresponding to the cell based on the comprehensive statistical information; if the division index of the cell is greater than the preset division threshold, divide it into 2 d2 sub-cells, calculate the division index of each sub-cell, and repeat the division until the number of layers L1 is reached; Define a level as the clustering level. On the defined clustering level, traverse all non-empty cells and mark each non-empty cell as an independent cluster to obtain the initial n3 clusters, each cluster corresponding to a cell. For each pair of adjacent clusters, calculate their partitioning index and find the difference between the two. If the difference is less than the preset merge threshold, merge the pair of adjacent clusters into one cluster. Repeat until the calculated differences of all adjacent clusters are greater than or equal to the merge threshold, and finally obtain n heterogeneous storage node groups.

10. A cross-border e-commerce trade data distributed storage management method, which is implemented based on a cross-border e-commerce trade data distributed storage management system according to any one of claims 1 to 9, characterized in that: include: S1. Collect cross-border e-commerce trade data; S2. Formulate data classification rules based on cross-border e-commerce trade data, and divide the cross-border e-commerce trade data into several data subsets according to the data classification rules; S3, construct a heterogeneous coding graph for each data subset, and color the heterogeneous coding graph to generate a corresponding heterogeneous coding data subset; S4, distributing and storing the heterogeneous encoded data subsets in a plurality of heterogeneous storage nodes according to a preset distribution rule, and grouping the heterogeneous storage nodes to obtain n heterogeneous storage node groups; Allocate storage space to each heterogeneous storage node group to complete distributed storage.

Images