Adaptive data management method for polymorphic addressing

Abstract

An adaptive data management method for polymorphic addressing comprises the following steps: acquiring multi-scene data from the Internet of Things, preprocessing the multi-scene data, and constructing a standardized data set; and performing multi-dimensional classification on the multi-scene data based on the similarity among the key indexes in the standardized data set to obtain a classification table. And obtaining a polymorphic addressing scheme corresponding to each type of data in the classification table, and generating a polymorphic addressing mapping table in combination with the property parameters of each type of data. And storing the polymorphic addressing mapping table in a distributed storage end, carrying out addressing position distribution on each storage node in the distributed storage end through a network topology structure, and outputting a distributed storage strategy. And constructing a virtual node associated with each storage node, and when the distributed storage strategy is executed, splitting the polymorphic addressing mapping table so as to store the polymorphic addressing data with the first scene label to the virtual node. Thread blockage during reading and writing of the multi-scene data in the Internet of Things is avoided, and the data management efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things and big data processing, and more specifically, relates to an adaptive data management method with polymorphic addressing. Background Art

[0002] Internet of Things data has characteristics such as massive volume, dynamics, polymorphism, and correlation. For example, sensor data may generate tens of thousands of records per second, device addresses may be dynamically updated according to location changes, data sources include different forms such as text, coordinates, and codes, with polymorphism, and need to be associated with information such as geographical location and device status. And polymorphic addressed data, as one of many data types in the Internet of Things, refers to the diverse address forms of devices in the Internet of Things at different levels (physical, network, logical) and scenarios (static / dynamic). For example, physical identifiers: unique hardware identifiers such as MAC addresses, serial numbers, IMEI, etc.; network addresses: protocol layer addressing such as IPv4 / IPv6, Zigbee short addresses, MQTT topics, etc.; logical addresses: geofence codes, hierarchical structures (such as factory - production line - device), and the address formats generated by different devices and protocols vary significantly.

[0003] Currently, when dealing with complex and diverse data types in the Internet of Things, due to the limitations of traditional single - addressing performance, the data management efficiency is low when processing complex and diverse data types in the Internet of Things, and in the process of a large amount of different data pouring in, it may also cause the situation of individual thread jams. Summary of the Invention

[0004] To solve the deficiencies in the prior art, the purpose of the present invention is to solve the above - mentioned defects, and further propose an adaptive data management method with polymorphic addressing.

[0005] The present invention adopts the following technical solutions.

[0006] The first aspect of the present invention discloses an adaptive data management method with polymorphic addressing, and the method includes: Obtain multi - scenario data from the Internet of Things, and pre - process the multi - scenario data to construct a standardized data set; Perform multi - dimensional classification on the multi - scenario data based on the similarity between key indicators in the standardized data set to obtain a classification table; Obtain the polymorphic addressing scheme corresponding to each type of data in the classification table, and generate a polymorphic addressing mapping table in combination with the property parameters of each type of data; Store the polymorphic addressing mapping table in a distributed storage end, and allocate addressing positions to each storage node in the distributed storage end through a network topology structure, and output a distributed storage strategy; Construct virtual nodes associated with each storage node, and when the distributed storage policy is executed, split the polymorphic addressing mapping table to store polymorphic addressing data with a first scenario label in the virtual nodes; Among them, the property parameters include the prefix, attribute, and scenario label of each type of data. The polymorphic addressing mapping table has an address segment addressing policy for retrieving the address segment of the polymorphic addressing mapping table. The virtual node is used for distributed storage of the polymorphic addressing mapping table that meets the preset conditions. The first scenario label is the scenario label defined in the preset conditions.

[0007] Further, the preprocessing of multi-scenario data from different sources to construct a standardized data set includes: Obtain multi-scenario data from different sources, and perform field mapping and format normalization processing on the multi-scenario data to obtain standardized data; the standardized data is data that meets the processing requirements of the same parser; Based on the standardized data, construct the standardized data set.

[0008] Further, the multi-dimensional classification of the multi-scenario data based on the similarity between key indicators in the standardized data set to obtain a classification table includes: Extract key indicators from the standardized data set, and construct key indicator vectors based on the key indicators, so as to calculate the similarity between key indicators based on the standard deviation of the key indicator vectors, and classify the key indicators according to the similarity; Divide the standard data in the standardized data set into multiple scenario labels according to the types of the key indicators, and label the corresponding priorities for each scenario label according to requirements, so as to output a classification table including multi-scenario labels and multi-dimensional classification information; Among them, the key indicators include the concurrency, write rate, read density, and geographical span of the multi-scenario data.

[0009] Further, the obtaining of the polymorphic addressing scheme corresponding to each type of data in the classification table and the generation of a polymorphic addressing mapping table in combination with the property parameters of each type of data include: Obtain multiple address segments defined based on each type of data in the classification table, and map each type of data in the classification table to the corresponding address segment, so as to set an adjustable access priority for the address segment; Construct an addressing policy unit with a combined function based on each type of data in the classification table, the polymorphic addressing scheme corresponding to each type of data, and the property parameters; Call the addressing policy unit to parse the address segment according to the access priority to obtain the call path and redundancy policy when reading and writing the address segment; Determine different addressing prefix allocation results according to the key metrics corresponding to each scenario label in the classification table, and establish an initial mapping record including an address segment - scenario label mapping table to determine the writing strategy and prefix consistency format in the initial mapping record; Obtain the defined addressing attribute matrix according to the addressing prefix allocation result and the initial mapping record; Calculate the attribute priority score based on the addressing prefix and the corresponding attributes and attribute weights in the addressing attribute matrix; Integrate the attribute priority score, the call path, and the redundancy strategy to generate the polymorphic addressing mapping table and the address segment addressing strategy.

[0010] Further, storing the polymorphic addressing mapping table in a distributed storage end, and performing addressing location allocation for each storage node in the distributed storage end through a network topology structure, and outputting a distributed storage strategy, including: Analyze the hardware characteristics and network connection relationships of all current storage nodes to construct a storage node loading table, and store the polymorphic addressing mapping table in the distributed storage end according to the storage node loading table. The storage node loading table includes the resource quota and read - write performance parameters of each storage node; Couple and allocate the addressing prefixes in the polymorphic addressing mapping table in the distributed storage end with the storage node loading table, and calculate the network cost of deploying each address segment on different storage nodes, so as to select the storage node or storage node combination with the minimum network cost for each addressing prefix, and output the coupled addressing location allocation strategy; Determine the judgment result for characterizing whether any two storage nodes are directly connected based on the network topology structure, and fine - tune the storage strategy of each address segment between storage nodes to obtain the fine - tuned storage strategy; Integrate the fine - tuned storage strategy with the addressing location allocation strategy to obtain the distributed storage strategy for multi - storage - node integration.

[0011] Further, constructing virtual nodes associated with each storage node, and when the distributed storage strategy is executed, splitting the polymorphic addressing mapping table to store the polymorphic addressing data with the first scenario label into the virtual nodes, including: Construct virtual nodes associated with each storage node; Read the scenario label information from the polymorphic addressing mapping table, and when executing the distributed storage strategy corresponding to the scenario label information, split the polymorphic addressing data with the first scenario label from the polymorphic addressing mapping table; The addressing positions of the virtual nodes are allocated through a ring or tree topology structure, so as to store polymorphic addressing data with a first scenario label in each virtual node.

[0012] The present invention also discloses an adaptive data management system for polymorphic addressing, which is used to implement the steps of the adaptive data management method for polymorphic addressing described in the first aspect. The system includes: A data preprocessing module, which is used to obtain multi-scenario data from the Internet of Things and preprocess the multi-scenario data to construct a standardized data set; A data classification module, which is used to perform multi-dimensional classification on the multi-scenario data based on the similarity between key indicators in the standardized data set to obtain a classification table; A mapping table generation module, which is used to obtain the polymorphic addressing scheme corresponding to each type of data in the classification table, and generate a polymorphic addressing mapping table and an address segment addressing strategy in combination with the property parameters of each type of data; A distributed storage module, which is used to store the polymorphic addressing mapping table in a distributed storage end, and allocate addressing positions to each storage node in the distributed storage end through a network topology structure, and output a distributed storage strategy; An addressing data transfer module, which is used to construct virtual nodes associated with each storage node, and when the distributed storage strategy is executed, split the polymorphic addressing mapping table to store the polymorphic addressing data with a first scenario label in the virtual nodes; Wherein, the property parameters include the prefix, attribute, and scenario label of each type of data. The polymorphic addressing mapping table has an address segment addressing strategy for retrieving the address segment of the polymorphic addressing mapping table. The virtual node is used to perform distributed storage on the polymorphic addressing mapping table that meets the preset conditions, and the first scenario label is the scenario label defined in the preset conditions.

[0013] The third aspect of the present invention discloses a terminal, which includes a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0014] The fourth aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored, characterized in that the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] The beneficial effects of the present invention are that, compared with the prior art, the present invention has the following advantages: (1) The present invention analyzes the features and classifies multi-scenario data from different sources in multiple dimensions, and generates corresponding polymorphic addressing schemes, polymorphic addressing mapping tables, and corresponding address segment addressing strategies for each type of data in the classification table. The polymorphic addressing mapping table is stored in the distributed storage end, and the addressing positions of each storage node in the distributed storage end are allocated through the network topology structure, effectively avoiding the problem that single addressing is difficult to handle complex multi-scenario data. In addition, the present invention performs orderly distributed storage on complex multi-scenario data, and multiple storage nodes share the complex multi-scenario data, which not only avoids the situation of thread blockage during massive data reading and writing, but also improves the efficiency of data management.

[0016] (2) The present invention constructs virtual nodes associated with each storage node, and when the distributed storage strategy is executed, the polymorphic addressing mapping table is split to distribute the polymorphic addressing data with special set scenario tags to the virtual nodes, so that the polymorphic addressing data that may cause thread blockage or abnormal situations (such as addressing data with cross-node sharing tags, addressing data with scenario tags of "high-throughput computing" or "ultra-low latency") can be stored in the virtual nodes in advance to prevent it from affecting each storage node and other threads, further reducing the abnormal situation during the reading and writing of complex multi-scenario data and improving the data management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flowchart of an adaptive data management method with polymorphic addressing. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following further describes the present application with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present application.

[0019] As Figure 1 shown, in one embodiment, an adaptive data management method with polymorphic addressing includes the following steps: Step S110, obtain multi-scenario data from the Internet of Things and preprocess the multi-scenario data to construct a standardized data set.

[0020] In some embodiments, for the adaptive data management method with polymorphic addressing provided by the present invention, step S110 specifically includes the following steps: Step S111, obtain multi-scenario data from different sources, perform field mapping and format normalization processing on the multi-scenario data to obtain standardized data, and the standardized data is data that meets the processing requirements of the same parser.

[0021] Step S112, construct a standardized data set based on the standardized data.

[0022] Step S120: Classify the multi-scenario data in multiple dimensions based on the similarity between key indicators in the standardized dataset to obtain a classification table.

[0023] Each type of data in the classification table has a corresponding scenario label.

[0024] In some embodiments, for the adaptive data management method with polymorphic addressing provided by the present invention, step S120 specifically includes the following steps: Step S121: Extract key indicators from the standardized dataset, construct key indicator vectors based on the key indicators, calculate the similarity between key indicators based on the standard deviation of the key indicator vectors, and classify the key indicators according to the similarity.

[0025] Step S122: Divide the standard data in the standardized dataset into multiple scenario labels according to the types of key indicators, and label corresponding priorities for each scenario label as required, so as to output a classification table containing multi-scenario labels and multi-dimensional classification information.

[0026] The key indicators include the concurrency, write rate, read density, and geographical span of the multi-scenario data.

[0027] Step S130: Obtain the polymorphic addressing schemes corresponding to each type of data in the classification table, and generate a polymorphic addressing mapping table in combination with the property parameters of each type of data.

[0028] The property parameters include the prefix, attribute, and scenario label of each type of data, and the polymorphic addressing mapping table has an address segment addressing strategy for address segment retrieval of the polymorphic addressing mapping table.

[0029] In some embodiments, for the adaptive data management method with polymorphic addressing provided by the present invention, step S130 specifically includes the following steps: Step S131: Obtain multiple address segments defined based on each type of data in the classification table, map each type of data in the classification table to the corresponding address segment, and set adjustable access priorities for the address segments.

[0030] Step S132: Construct an addressing strategy unit with combined functions based on each type of data in the classification table, the polymorphic addressing scheme corresponding to each type of data, and the property parameters.

[0031] Step S133: Call the addressing strategy unit to parse the address segments according to the access priorities to obtain the call path and redundancy strategy during address segment reading and writing.

[0032] Step S134: Determine different addressing prefix allocation results according to the key metrics corresponding to each type of scenario label in the classification table, and establish an initial mapping record including an address segment - scenario label mapping table to determine the writing policy and prefix consistency format in the initial mapping record.

[0033] Step S135: Obtain the defined addressing attribute matrix according to the addressing prefix allocation result and the initial mapping record.

[0034] Step S136: Calculate the attribute priority score based on the addressing prefix and the corresponding attributes and attribute weights in the addressing attribute matrix.

[0035] Step S137: Integrate the attribute priority score, the call path, and the redundancy policy to generate a polymorphic addressing mapping table and an address segment addressing policy.

[0036] Step S140: Store the polymorphic addressing mapping table in the distributed storage end, and allocate addressing positions to each storage node in the distributed storage end through the network topology structure, and output a distributed storage policy.

[0037] In some embodiments, for the adaptive data management method of polymorphic addressing provided by the present invention, step S140 specifically includes the following steps: Step S141: Analyze the hardware characteristics and network connection relationships of all current storage nodes to construct a storage node loading table, and store the polymorphic addressing mapping table in the distributed storage end according to the storage node loading table. The storage node loading table includes the resource quota and read - write performance parameters of each storage node.

[0038] Step S142: Couple and allocate the addressing prefixes in the polymorphic addressing mapping table in the distributed storage end with the storage node loading table, and calculate the network cost of deploying each address segment on different storage nodes, so as to select the storage node or storage node combination with the minimum network cost for each addressing prefix, and output a coupled addressing position allocation policy.

[0039] Step S143: Determine the judgment result for characterizing whether any two nodes are directly connected based on the network topology structure, and fine - tune the storage policy of each address segment between each storage node to obtain a fine - tuned storage policy.

[0040] Step S144: Integrate the fine - tuned storage policy with the addressing position allocation policy to obtain a distributed storage policy for multi - storage - node fusion.

[0041] Step S150: Construct virtual nodes associated with each storage node, and when the distributed storage policy is executed, split the polymorphic addressing mapping table to store the polymorphic addressing data with the first scenario label in the virtual nodes.

[0042] Among them, the virtual node is a virtual distributed storage end, which is used for distributed storage of a polymorphic addressing mapping table that meets preset conditions. The first scenario tag is the scenario tag defined in the preset conditions and can be set in advance. For example, for addressing data with a cross-node sharing tag, a ring or tree topology structure can be used to allocate the addressing positions of virtual nodes to avoid data replicas concentrating on storage nodes and causing hotspots. Another example is to perform targeted scheduling for scenario tags such as "high-throughput computing" or "ultra-low latency", and schedule them to the constructed virtual nodes, that is, replicate core data in virtual nodes with higher bandwidth or lower latency to prevent the loss of core data and ensure the security of data management.

[0043] In some embodiments, for the adaptive data management method with polymorphic addressing provided by the present invention, step S150 specifically includes the following steps: Step S151, construct virtual nodes associated with each storage node.

[0044] Step S152, read the scenario tag information from the polymorphic addressing mapping table, and when executing the distributed storage policy corresponding to the scenario tag information, split the polymorphic addressing data with the first scenario tag from the polymorphic addressing mapping table.

[0045] Step S153, allocate the addressing positions of virtual nodes through a ring or tree topology structure to store the polymorphic addressing data with the first scenario tag into each virtual node.

[0046] In a specific embodiment, the adaptive data management method with polymorphic addressing provided by the present invention includes steps 1 to 4: Step 1, multi-scenario data analysis and multi-dimensional classification.

[0047] To solve the inefficiency problem of traditional single addressing in processing complex data types in the Internet of Things, first analyze various application scenarios and data access characteristics that may occur in the system to form a multi-dimensional classification result.

[0048] It should be noted that polymorphic addressing data refers to the diverse address forms of devices in the Internet of Things at different levels (physical, network, logical) and scenarios (static / dynamic). For example, physical identifiers: hardware unique identifiers such as MAC addresses, serial numbers, IMEI, etc.; network addresses: protocol layer addressing such as IPv4 / IPv6, Zigbee short addresses, MQTT topics; logical addresses: geofence codes, hierarchical structures (such as factory-production line-equipment). Therefore, for polymorphic addressing data in the Internet of Things, it has the characteristics of different scenario representations, different scenarios, and diverse address forms, and the address formats generated by different devices and protocols vary significantly.

[0049] Specifically, it includes steps 1.1 to 1.3: Step 1.1, read the classification table and establish a processing mapping.

[0050] Collect logs or data from different sources in the Internet of Things, that is, multi-scenario data, such as: "Video surveillance high-frequency write logs at night" (timestamp, write rate), "E-commerce flash sale activity order records" (number of orders, peak QPS), and "IoT (Internet of Things) sensor batch reporting data" (sensor type, geographical location, reporting period). Perform field mapping and format normalization on them to ensure that all data can be processed by the same parser, and finally output a standardized data set for subsequent analysis.

[0051] Step 1.2, based on the standardized log set, extract core metrics to calculate the similarity matrix.

[0052] Key metrics: concurrency C, write rate W, read density R, geographical span G, etc. The specific similarity formula can be calculated based on the vector composed of key metrics using the standard deviation.

[0053] Step 1.3, scenario label division and priority annotation.

[0054] Divide the above multi-scenario data records into several scenario labels according to key features such as concurrency, writing, and reading, such as: "High concurrency + high-speed writing + medium geographical distribution", "Medium concurrency + extremely high-speed writing + high geographical distribution", and "Low concurrency + batch reading + negligible geographical span", and label the corresponding priority for each scenario label (such as assigning the highest security requirements to the financial transaction scenario and strengthening the geographical dimension for the IoT scenario, etc.). Finally, output a classification table containing multi-scenario labels and multi-dimensional classification information.

[0055] Step 2, construct a polymorphic addressing mapping table.

[0056] Based on the classification table in Step 1, generate the corresponding polymorphic addressing scheme for each type of data and generate a polymorphic addressing mapping table within the system. First, define multiple address segments or page attributes, such as "cache page", "streaming mapping segment", "cross-node shared label", etc. Then map each type of data in the classification table to the corresponding address segment and set an adjustable access priority for it (such as CPU / GPU parallel access, remote cold storage, etc.). After that, introduce a combinable addressing strategy unit, which determines the call path and redundancy strategy during data reading and writing by parsing the address segment label (for example, enabling proximal cache direct access for the "cache page", adopting prefetching and pipelined writing strategies for the "streaming mapping segment", and linking the consistency protocol for multi-node synchronization for the "shared label"). By allocating addressing forms for different data according to local conditions, the problems of high concurrency conflicts, latency jitter, and space waste in the single addressing mode are solved, making the system more scalable.

[0057] Specifically, it includes steps 2.1 to 2.3: Step 2.1, read the classification table and allocate addressing prefixes.

[0058] According to the performance, security, or geographical dimensions corresponding to each scenario label in the table (such as "high concurrency + strong consistency", "large batch read-only + cross-region", etc.), allocate different addressing prefixes (such as high-speed segment, cold storage segment, distributed shared segment). Establish an initial mapping record to form a list of address segment - scenario label, and at the same time indicate the required write policy or consistency mode.

[0059] For example, if the label "high concurrency + strong consistency" corresponds to the high-speed page prefix "HSP_", then write it as "HSP_ → high concurrency + strong consistency" in the mapping record.

[0060] Step 2.2, derive the addressing attribute matrix and addressing strategy.

[0061] Specifically, first, define the addressing attribute matrix, which is composed of the values of the addressing prefix and its attributes (write amplification factor, access latency, and fault tolerance level), as well as the performance cost or benefit of each addressing prefix on any attribute. Secondly, calculate the priority score of the corresponding addressed data according to the attribute weight corresponding to each addressing prefix and the values in the addressing attribute matrix. Among them, the addressing prefix with a high priority score is adapted to high-priority scenarios, and the one with a low priority score is adapted to secondary-priority scenarios. Based on this, an addressing strategy that maps "addressing prefix - priority score - scenario label" can be formed.

[0062] Step 2.3, integrate and output the policy mapping.

[0063] Specifically, unify and integrate the addressing prefix, attribute score, and scenario label to obtain a complete polymorphic addressing mapping table and address segment addressing strategy. In this process, it is first necessary to determine the write policy (such as sequential write, block write, parallel replication) and read policy (cache priority, distributed consistency reading, etc.) corresponding to each addressing prefix, and then the polymorphic addressing mapping table and address segment addressing strategy For example: "HSP_ prefix → high concurrency + strong consistency scenario; write policy = multi-way parallel, read policy = local cache"; "CS_ prefix → large batch read-only + cross-region scenario; write policy = periodic batch synchronization, read policy = cross-node load balancing"; "BK_ prefix → low-priority archiving scenario; write policy = compressed storage, read policy = delayed access".

[0064] Step 3, adapt to the distributed topology and integrate multiple storage nodes.

[0065] After the polymorphic addressing mapping is generated, it needs to be integrated into the distributed environment to adapt to the resource differences of different storage nodes and utilize the network topology characteristics to accelerate data access. First, deploy the "polymorphic addressing proxy unit" on each storage node. This proxy unit can read the scenario label information from the mapping table in Step 2 and execute the corresponding addressing strategy. For the addressed data with cross-node shared labels, the ring or tree topology can be used to allocate the addressing positions of each storage node to avoid data replicas being concentrated on a few nodes, causing hotspots. Then, in combination with the node hardware characteristics (such as GPU computing acceleration cards, FPGA dedicated processing, NVM storage media, etc.), targeted scheduling is performed on the scenario labels of "high-throughput computing" or "ultra-low latency", that is, the core data is placed on the nodes with higher bandwidth or lower latency. Finally, a distributed index and routing table are established. Once a storage node receives an access request, it will determine whether it needs to be processed locally or forwarded across nodes according to the address segment scenario label in the polymorphic addressing mapping table, improving the access efficiency and reducing unnecessary network jumps.

[0066] The above multi-storage node integration effectively connects the polymorphic addressing to the large-scale distributed system, which can further solve the performance bottleneck and resource waste of the traditional "single mapping + centralized storage" in heterogeneous clusters.

[0067] Specifically, it includes Steps 3.1 to 3.3: Step 3.1, node loading and network topology scanning.

[0068] Specifically, analyze the hardware characteristics (such as the number of CPU cores, bandwidth, storage media) and network connection relationships (such as ring or tree topology) of all storage nodes in the current cluster, construct a storage node loading table, which consists of multiple storage nodes and the quotas or performance values (such as available bandwidth, maximum concurrent connection number) that each storage node can provide for any one resource. The values can be obtained through system monitoring or node self-check, and output the node loading information for subsequent distributed adaptation algorithms.

[0069] For example, if storage node X1 has a gigabit network interface and SSD storage; X2 has a 10Gb network interface and NVMe storage, then record information such as X1(bandwidth = 1, storage media = SSD), X2(bandwidth = 10, storage media = NVMe) in the storage node loading table.

[0070] Step 3.2, addressing and node coupling allocation.

[0071] Specifically, according to the addressing prefixes (such as HSP_, CS_, BK_) in the polymorphic addressing mapping table, it is coupled and allocated with the storage node loading table, and the overall network cost of each address segment deployed on different storage nodes is calculated. This network cost represents the comprehensive cost when any addressing prefix is deployed on any storage node, and is jointly determined by the weight of the storage node in the corresponding resource dimension, the demand for this resource dimension, and the load penalty value corresponding to the storage node. This load penalty value is used to prevent a single storage node from being overloaded. Finally, for each addressing prefix, select the storage node or node combination with the lowest network cost, map the address segment to the corresponding location, and form a preliminary distributed layout, that is, output the addressing location allocation decision after coupling.

[0072] For example, if the HSP_ prefix has a high demand for network bandwidth (i.e., resource dimension demand = 0.8), and the storage node X2 has 10 Gb bandwidth (resource dimension = 0.6, load penalty value = 0.1), it can be calculated that the network cost of node X2 is lower than that of node X1, so node X2 is preferentially selected.

[0073] Step 3.3, Policy fine-tuning and multi-node fusion.

[0074] Specifically, an adjacency matrix is constructed based on the network topology structure. This adjacency matrix is used to represent whether node a and node b are directly connected. For example, if the adjacency matrix value of node a and node b is 1, it means that node a and node b are directly connected, and if the adjacency matrix value is 0, it means that the two are not directly connected. Then, the replication policy of each address segment between storage nodes is fine-tuned to obtain the fine-tuned storage policy. If the communication cost between adjacent nodes is relatively low, local mirroring or replica synchronization is preferentially performed to reduce the traffic across remote links. Finally, the fine-tuned storage policy is integrated with the addressing location allocation policy to output the final multi-node fusion distributed topology adaptation policy.

[0075] For example, if the adjacency matrix value of node X2 and node X3 in the local area network is 1 and the bandwidth is large, a mirror replica of the HSP_ prefix can be set on node X3 to achieve fast synchronization and fault tolerance.

[0076] Step 4, Construct virtual nodes associated with each storage node and split the polymorphic addressing data with the first scenario label.

[0077] Specifically, when implementing the final distributed storage policy, the scenario label information is read from the polymorphic addressing mapping table, and the address segment addressing policy corresponding to the scenario label information is executed. Furthermore, polymorphic addressing data with special scenario markings is selected from the polymorphic addressing mapping table. For example, addressing data with a cross-node sharing label (i.e., the above-mentioned first scenario label) can use a ring or tree topology to allocate addressing positions for virtual nodes, avoiding data replicas concentrating on storage nodes and causing hotspots. Another example is to perform targeted scheduling on scenario labels such as "high-throughput computing" or "ultra-low latency" (i.e., the above-mentioned first scenario label), and schedule them to the constructed virtual nodes, that is, replicate core data in virtual nodes with higher bandwidth or lower latency to prevent the loss of core data and ensure the security of data management.

[0078] The adaptive data management system with polymorphic addressing provided by the present invention will be described below. The adaptive data management system with polymorphic addressing described below can be mutually referred to and corresponding to the adaptive data management method with polymorphic addressing described above.

[0079] In one embodiment, an adaptive data management system with polymorphic addressing includes a data preprocessing module, a data classification module, a mapping table generation module, a distributed storage module, and an addressing data transfer module.

[0080] The data preprocessing module is used to obtain multi-scenario data from the Internet of Things and preprocess the multi-scenario data to construct a standardized data set.

[0081] The data classification module is used to perform multi-dimensional classification on the multi-scenario data based on the similarity between key metrics in the standardized data set to obtain a classification table.

[0082] The mapping table generation module is used to obtain the polymorphic addressing scheme corresponding to each type of data in the classification table and generate a polymorphic addressing mapping table in combination with the property parameters of each type of data.

[0083] The distributed storage module is used to store the polymorphic addressing mapping table in the distributed storage end, and allocate addressing positions for each storage node in the distributed storage end through a network topology structure, and output a distributed storage policy.

[0084] The addressing data transfer module is used to construct virtual nodes associated with each storage node, and when the distributed storage policy is executed, split the polymorphic addressing mapping table to store the polymorphic addressing data with the first scenario label in the virtual nodes.

[0085] Among them, each type of data in the classification table has a corresponding scenario label. The property parameters include the prefix, attribute, and scenario label of each type of data. The polymorphic addressing mapping table has an address segment addressing strategy for retrieving the address segment of the polymorphic addressing mapping table. The virtual node is a virtual distributed storage end for distributed storage of the polymorphic addressing mapping table that meets the preset conditions. The first scenario label is the scenario label defined in the preset conditions.

[0086] In this embodiment, for the adaptive data management system with polymorphic addressing provided by the present invention, the data preprocessing module is specifically used for: Obtain multi-scenario data from different sources, perform field mapping and format normalization processing on the multi-scenario data to obtain standardized data, and the standardized data is data that meets the processing requirements of the same parser.

[0087] Based on the standardized data, construct a standardized data set.

[0088] In this embodiment, for the adaptive data management system with polymorphic addressing provided by the present invention, the data classification module is specifically used for: Extract key indicators from the standardized data set, construct a key indicator vector based on the key indicators, calculate the similarity between the key indicators based on the standard deviation of the key indicator vector, and classify the key indicators according to the similarity.

[0089] Divide the standard data in the standardized data set into multiple scenario labels according to the types of key indicators, and label the corresponding priority for each scenario label according to the requirements, so as to output a classification table containing multi-scenario labels and multi-dimensional classification information.

[0090] Among them, the key indicators include the concurrency, write rate, read density, and geographical span of the multi-scenario data.

[0091] In this embodiment, for the adaptive data management system with polymorphic addressing provided by the present invention, the mapping table generation module is specifically used for: Obtain multiple address segments defined based on each type of data in the classification table, map each type of data in the classification table to the corresponding address segment, and set an adjustable access priority for the address segment.

[0092] Based on each type of data in the classification table, the polymorphic addressing scheme corresponding to each type of data, and the property parameters, construct an addressing strategy unit with combined functions.

[0093] Call the addressing strategy unit to parse the address segment according to the access priority to obtain the call path and redundancy strategy during address segment reading and writing.

[0094] Determine different addressing prefix allocation results according to the key metrics corresponding to each type of scenario label in the classification table, and establish an initial mapping record including an address segment - scenario label mapping table to determine the writing strategy and prefix consistency format in the initial mapping record.

[0095] Obtain the defined addressing attribute matrix according to the addressing prefix allocation result and the initial mapping record.

[0096] Calculate the attribute priority score based on the addressing prefix and the corresponding attributes and attribute weights in the addressing attribute matrix.

[0097] Integrate the attribute priority score, the call path, and the redundancy strategy to generate a polymorphic addressing mapping table and an address segment addressing strategy.

[0098] In this embodiment, for the adaptive data management system with polymorphic addressing provided by the present invention, the distributed storage module is specifically used for: Analyze the hardware characteristics and network connection relationships of all current storage nodes, and store the polymorphic addressing mapping table in the distributed storage end according to the storage node loading table to construct a storage node loading table, where the storage node loading table includes the resource quota and read - write performance parameters of each storage node.

[0099] Couple and allocate the addressing prefixes in the polymorphic addressing mapping table in the distributed storage end with the storage node loading table, and calculate the network cost of deploying each address segment on different storage nodes, so as to select the storage node or storage node combination with the minimum network cost for each addressing prefix, and output the coupled addressing location allocation strategy.

[0100] Determine the judgment result for characterizing whether any two storage nodes are directly connected based on the network topology structure, and fine - tune the storage strategy of each address segment between storage nodes to obtain the fine - tuned storage strategy.

[0101] Integrate the fine - tuned storage strategy with the addressing location allocation strategy to obtain a distributed storage strategy for multi - storage node integration.

[0102] In this embodiment, for the adaptive data management system with polymorphic addressing provided by the present invention, the addressing data transfer module is specifically used for: Construct virtual nodes associated with each storage node.

[0103] Read the scenario label information from the polymorphic addressing mapping table, and when executing the distributed storage strategy corresponding to the scenario label information, split the polymorphic addressing data with the first scenario label from the polymorphic addressing mapping table.

[0104] Allocate addressing locations for the virtual nodes through a ring - shaped or tree - shaped topology structure to store the polymorphic addressing data with the first scenario label in each virtual node.

[0105] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.

[0106] The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0107] The computer-readable program instructions described herein may be downloaded to respective computing / processing devices from a computer-readable storage medium or may be downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0108] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via an Internet service provider through the Internet). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0109] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer - readable program instructions.

[0110] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner, so that the computer - readable medium storing the instructions comprises a manufacture, which includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent substitutions can still be made to the specific embodiments of the present invention, and any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. An adaptive data management method with polymorphic addressing, characterized in that The method includes: Obtaining multi-scenario data from the Internet of Things, and preprocessing the multi-scenario data to construct a standardized data set; Performing multi-dimensional classification on the multi-scenario data based on the similarity between key metrics in the standardized data set to obtain a classification table; Obtaining a polymorphic addressing scheme corresponding to each type of data in the classification table, and generating a polymorphic addressing mapping table by combining the property parameters of each type of data; Storing the polymorphic addressing mapping table in a distributed storage end, and allocating addressing positions to each storage node in the distributed storage end through a network topology structure to output a distributed storage strategy; Constructing virtual nodes associated with each storage node, and when the distributed storage strategy is executed, splitting the polymorphic addressing mapping table to store the polymorphic addressing data with a first scenario label in the virtual node; The property parameters include the prefix, attribute, and scenario label of each type of data. The polymorphic addressing mapping table has an address segment addressing strategy for retrieving the address segment of the polymorphic addressing mapping table. The virtual node is used for distributed storage of the polymorphic addressing mapping table that meets the preset conditions. The first scenario label is the scenario label defined in the preset conditions.

2. The polymorphic addressing-based adaptive data management method according to claim 1, wherein The obtaining multi-scenario data from the Internet of Things and preprocessing the multi-scenario data to construct a standardized data set includes: Obtaining multi-scenario data from the Internet of Things, and performing field mapping and format normalization processing on the multi-scenario data to obtain standardized data; the standardized data is data that meets the processing requirements of the same parser; Based on the standardized data, constructing the standardized data set.

3. The polymorphic addressing-based adaptive data management method according to claim 2, wherein The performing multi-dimensional classification on the multi-scenario data based on the similarity between key metrics in the standardized data set to obtain a classification table includes: Extracting key metrics from the standardized data set, and constructing a key metric vector based on the key metrics to calculate the similarity between key metrics based on the standard deviation of the key metric vector, and classifying the key metrics according to the similarity; Dividing the standard data in the standardized data set into multiple scenario labels according to the types of the key metrics, and marking corresponding priorities for each scenario label according to requirements to output a classification table including multi-scenario labels and multi-dimensional classification information; Among them, the key metrics include the concurrency, write rate, read density, and geographical span of the multi-scenario data.

4. The polymorphic addressing-based adaptive data management method according to claim 1, wherein The obtaining a polymorphic addressing scheme corresponding to each type of data in the classification table and generating a polymorphic addressing mapping table by combining the property parameters of each type of data includes: Obtaining multiple address segments defined based on each type of data in the classification table, and mapping each type of data in the classification table to the corresponding address segment to set an adjustable access priority for the address segment; Constructing an addressing strategy unit with a combined function based on each type of data in the classification table, the polymorphic addressing scheme corresponding to each type of data, and the property parameters; Invoking the addressing strategy unit to parse the address segment according to the access priority to obtain the call path and redundancy strategy when the address segment is read and written; Determine different addressing prefix allocation results according to the key metrics corresponding to each type of scenario label in the classification table, and establish an initial mapping record including an address segment - scenario label mapping table to determine the writing strategy and prefix consistency format in the initial mapping record; Obtain the defined addressing attribute matrix according to the addressing prefix allocation result and the initial mapping record; Calculate the attribute priority score based on the addressing prefix and the corresponding attributes and attribute weights in the addressing attribute matrix; Integrate the attribute priority score, the call path, and the redundancy strategy to generate the polymorphic addressing mapping table and the address segment addressing strategy.

5. The polymorphic addressing-based adaptive data management method according to claim 4, characterized in that, Store the polymorphic addressing mapping table in the distributed storage end, and allocate addressing positions to each storage node in the distributed storage end through the network topology structure, and output the distributed storage strategy, including: Analyze the hardware characteristics and network connection relationships of all current storage nodes to construct a storage node loading table, and store the polymorphic addressing mapping table in the distributed storage end according to the storage node loading table. The storage node loading table includes the resource quota and read - write performance parameters of each storage node; Couple and allocate the addressing prefixes in the polymorphic addressing mapping table in the distributed storage end with the storage node loading table, and calculate the network cost of each address segment deployed on different storage nodes to select the storage node or storage node combination with the minimum network cost for each addressing prefix, and output the coupled addressing position allocation strategy; Determine the judgment result used to characterize whether any two storage nodes are directly connected based on the network topology structure, and fine - tune the storage strategy of each address segment between each storage node to obtain the fine - tuned storage strategy; Integrate the fine - tuned storage strategy with the addressing position allocation strategy to obtain the distributed storage strategy for multi - storage node integration.

6. The polymorphic addressing-based adaptive data management method according to claim 5, characterized in that Construct virtual nodes associated with each storage node, and when the distributed storage strategy is executed, split the polymorphic addressing mapping table to store the polymorphic addressing data with the first scenario label in the virtual nodes, including: Construct virtual nodes associated with each storage node; Read the scenario label information from the polymorphic addressing mapping table, and when the distributed storage strategy corresponding to the scenario label information is executed, split the polymorphic addressing data with the first scenario label from the polymorphic addressing mapping table; Allocate addressing positions to the virtual nodes through a ring - shaped or tree - shaped topology structure to store the polymorphic addressing data with the first scenario label in each virtual node.

7. An adaptive data management system with polymorphic addressing, characterized in that, Steps for implementing the adaptive data management method for polymorphic addressing according to any one of claims 1 to 6, the system includes: A data pre - processing module for obtaining multi - scenario data from the Internet of Things and pre - processing the multi - scenario data to construct a standardized data set; A data classification module for performing multi - dimensional classification on the multi - scenario data based on the similarity between key metrics in the standardized data set to obtain a classification table; A mapping table generation module, configured to obtain the polymorphic addressing scheme corresponding to each type of data in the classification table, and generate a polymorphic addressing mapping table and an address segment addressing strategy by combining the property parameters of each type of data; A distributed storage module, configured to store the polymorphic addressing mapping table in a distributed storage end, and allocate addressing positions to each storage node in the distributed storage end through a network topology structure, and output a distributed storage strategy; An addressing data transfer module, configured to construct virtual nodes associated with each storage node, and when the distributed storage strategy is executed, split the polymorphic addressing mapping table to store the polymorphic addressing data with a first scenario label into the virtual nodes; Wherein, the property parameters include the prefix, attribute, and scenario label of each type of data, the polymorphic addressing mapping table has an address segment addressing strategy for performing address segment retrieval on the polymorphic addressing mapping table, the virtual nodes are used for distributed storage of the polymorphic addressing mapping table that meets preset conditions, and the first scenario label is the scenario label defined in the preset conditions.

8. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used for storing instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

Images