A multi-engine intelligent data warehouse storage model construction method and system

Abstract

The application relates to the technical field of data storage, in particular to a multi-engine intelligent data warehouse storage model construction method and system, which solves the technical problem that, in the prior art, a sharding random dispersion storage strategy is adopted for unstructured data, resulting in a large input / output overhead and high access cost in a data access calling process. The method comprises the following steps: constructing a distributed storage architecture comprising a plurality of storage nodes; performing sharding processing on unstructured data to be stored to obtain a plurality of data shards and acquire data features of each data shard; comparing the similarity of the data features between each data shard and historical data to evaluate the access granularity of each data shard; establishing an intelligent matching rule between the data shards and the storage nodes according to the access granularity of each data shard and the node resources of the storage nodes; and storing each data shard to a corresponding storage node according to the intelligent matching rule to form a multi-engine intelligent data warehouse storage model.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a method and system for constructing a multi-engine intelligent data warehouse storage model. Background Technology

[0002] With the development of information technology, unstructured data such as long videos, audio, medical images, and satellite remote sensing videos have become an important part of data storage. The core characteristic of this type of data is that it does not have a fixed length, type, or format, which is quite different from the key-value pair format of traditional structured data.

[0003] To address the storage needs of unstructured data, a distributed storage solution consisting of multiple data warehouses is typically adopted. This solution divides the massive amount of unstructured data into multiple data shards, which are then randomly allocated to different storage nodes for distributed storage, in order to leverage the scalability of the distributed system to handle the massive amount of data.

[0004] However, when accessing and retrieving this data, since the storage nodes are only responsible for data storage and do not participate in computation, all data needs to be fragmented and transferred to a centralized computing server for processing. This generates a large amount of input / output (IO) overhead, resulting in high data access costs. Summary of the Invention

[0005] To address the technical problem of excessive input / output overhead and high access costs during data access caused by the existing strategy of fragmented and randomly distributed storage for unstructured data, the present invention aims to provide a method and system for constructing a multi-engine intelligent data warehouse storage model. The specific technical solution adopted is as follows: Firstly, a method for constructing a multi-engine intelligent data warehouse storage model is provided, comprising: constructing a distributed storage architecture containing multiple storage nodes; configuring at least one data processing engine in each storage node and identifying node resources for data processing in the storage node; performing sharding processing on the unstructured data to be stored to obtain multiple data shards and obtaining the data characteristics of each data shard; evaluating the access granularity of each data shard by comparing the similarity of data characteristics between each data shard and historical data; the access granularity is used to characterize the input and output overhead of the data shard during access; establishing intelligent matching rules between data shards and storage nodes based on the access granularity of each data shard and the node resources of the storage nodes; and storing each data shard to the corresponding storage node according to the intelligent matching rules to form a multi-engine intelligent data warehouse storage model.

[0006] Based on the above technical solution, in the multi-engine intelligent data warehouse storage model construction method provided by this invention, a distributed storage architecture with multiple data processing engines configured and node resources identified is constructed. The access granularity is evaluated by combining the characteristics of data sharding and the similarity of historical data. Then, intelligent matching rules are established based on the access granularity and node resources to complete sharding storage, forming a multi-engine intelligent data warehouse storage model. This effectively avoids the drawbacks of random storage of data shards in the prior art, achieves accurate matching between data shards and storage node resources, reduces invalid data transfer during data access, significantly reduces input / output overhead and access costs, and improves the adaptability and efficiency of unstructured data storage and access by relying on multi-engine configuration and intelligent matching logic.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the method for establishing intelligent matching rules between data shards and storage nodes based on the access granularity of each data shard and the node resources of the storage nodes specifically includes: clustering and classifying multiple storage nodes according to the node resources of the storage nodes; determining the matching degree between the data shard and each level according to the access granularity of the data shard and the node resources of the storage nodes within each level, and taking the level with the highest matching degree as the matching level of the data shard; constructing a hash ring structure for the storage nodes within each level, and allocating the data shard to the storage nodes within the matching level through a hash mapping algorithm.

[0008] In conjunction with the first aspect above, in one possible implementation, the node resources include local computing power and network bandwidth; the method for determining the matching degree between data shards and each tier based on the access granularity of data shards and the node resources of storage nodes within each tier specifically includes: performing a first association between the access granularity of data shards and the average local computing power of storage nodes within each tier; the first association is configured such that the degree of influence of the average local computing power on the matching degree increases when the access granularity increases; performing a second association between the access granularity of data shards and the average network bandwidth of storage nodes within each tier; the second association is configured such that the degree of influence of the average network bandwidth on the matching degree increases when the access granularity decreases; and determining the matching degree between data shards and each tier based on the results of the first and second associations.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: formulating an access processing mechanism for a multi-engine intelligent data warehouse storage model based on the access granularity of data shards and the node resources of storage nodes.

[0010] In conjunction with the first aspect above, in one possible implementation, the aforementioned access processing mechanism specifically includes: if the access granularity of the data shard is greater than a preset granularity threshold, and the remaining local computing power of the corresponding storage node is greater than the preset computing power, the computing task is scheduled to the storage node to perform local computing through the data processing engine and return the result; if the access granularity of the data shard is less than the preset granularity threshold, and the remaining network bandwidth of the corresponding storage node is greater than the preset bandwidth, the data shard is transmitted to the cloud computing node through the bandwidth to complete the processing and return the result.

[0011] In conjunction with the first aspect above, in one possible implementation, the method further includes: receiving an access request message for requesting access to target data; locating the storage node of the data fragment of the target data; and performing data access on the data fragment of the target data according to the access granularity of the data fragment of the target data and the remaining node resources of the storage node, in accordance with the access processing mechanism.

[0012] In conjunction with the first aspect above, in one possible implementation, the method of constructing a hash ring structure for each storage node within a hierarchy and allocating data shards to matching storage nodes within a hierarchy using a hash mapping algorithm specifically includes: generating at least one replica shard for each data shard; when allocating replica shards using the hash mapping algorithm, if the allocated storage node has already stored the data shard or other replica shards, then the replica shard is allocated to the next adjacent storage node on the hash ring.

[0013] In conjunction with the first aspect above, in one possible implementation, the aforementioned data features include user features and content features; the method for evaluating the access granularity of each data shard by comparing the similarity of data features between each data shard and historical data specifically includes: comparing the similarity of user features between the data shard and historical data, and determining historical data whose similarity of user features is greater than a preset similarity threshold as reference data; comparing the similarity of content features between the data shard and the reference data, and combining this with the historical access volume of the reference data, to determine the expected access volume of the data shard; and determining the access granularity of the data shard based on the expected access volume of the data shard.

[0014] In conjunction with the first aspect above, in one possible implementation, the method for comparing the similarity of content features between data shards and reference data specifically includes: extracting content feature vectors from data shards and reference data, generating a sequence of content feature vectors according to the same sorting rules; and determining the sequence similarity between the content feature vector sequences of data shards and reference data using a dynamic sequence matching algorithm, which is used as the similarity of content features.

[0015] Secondly, a multi-engine intelligent data warehouse storage model construction system is provided, comprising: an architecture management module for constructing a distributed storage architecture containing multiple storage nodes; each storage node is configured with at least one data processing engine and is identified with node resources for data processing; a sharding management module for sharding the unstructured data to be stored to obtain multiple data shards and acquiring the data characteristics of each data shard; a granularity evaluation module for evaluating the access granularity of each data shard by comparing the similarity of data characteristics between each data shard and historical data; the access granularity is used to characterize the input and output overhead of the data shard during access; an intelligent matching module for establishing intelligent matching rules between data shards and storage nodes based on the access granularity of each data shard and the node resources of the storage nodes; and a model deployment module for storing each data shard to the corresponding storage node according to the intelligent matching rules, forming a multi-engine intelligent data warehouse storage model.

[0016] Thirdly, a multi-engine intelligent data warehouse storage model construction apparatus is provided, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is used to execute the instructions to perform the actions described in the first aspect and any possible implementation thereof. This multi-engine intelligent data warehouse storage model construction apparatus can be an electronic device or a chip within an electronic device.

[0017] Fourthly, a computer-readable storage medium is provided, which stores instructions that, when executed on a multi-engine intelligent data warehouse storage model building apparatus, cause the multi-engine intelligent data warehouse storage model building apparatus to perform the actions described in the first aspect and any possible implementation thereof.

[0018] Fifthly, a computer program product containing instructions is provided, which, when run on a multi-engine intelligent data warehouse storage model building device, causes the multi-engine intelligent data warehouse storage model building device to perform the actions described in the first aspect and any possible implementation thereof.

[0019] The present invention has the following beneficial effects: By constructing a distributed storage architecture that configures multiple data processing engines and identifies node resources, and by combining the characteristics of data sharding with the similarity of historical data to evaluate access granularity, and then establishing intelligent matching rules based on access granularity and node resources to complete sharded storage, a multi-engine intelligent data warehouse storage model is formed. This effectively avoids the drawbacks of random storage of data shards in existing technologies, achieves accurate adaptation between data shards and storage node resources, reduces invalid data transfer during data access, and significantly reduces input / output overhead and access costs. At the same time, relying on multi-engine configuration and intelligent matching logic, the adaptability and efficiency of unstructured data storage and access are improved. Attached Figure Description

[0020] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A system architecture diagram of a multi-engine intelligent data warehouse storage model construction system provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of a distributed storage architecture provided in one embodiment of the present invention; Figure 3 A flowchart illustrating a method for constructing a multi-engine intelligent data warehouse storage model according to an embodiment of the present invention; Figure 4 A flowchart illustrating a method for constructing a multi-engine intelligent data warehouse storage model according to an embodiment of the present invention; Figure 5 A flowchart illustrating a method for constructing a multi-engine intelligent data warehouse storage model according to an embodiment of the present invention; Figure 6 A flowchart illustrating a method for constructing a multi-engine intelligent data warehouse storage model according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the hardware structure of a multi-engine intelligent data warehouse storage model construction device provided in one embodiment of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multi-engine intelligent data warehouse storage model construction method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] The following description, in conjunction with the accompanying drawings, details the specific scheme of the multi-engine intelligent data warehouse storage model construction method and system provided by the present invention.

[0025] Please see Figure 1 The diagram illustrates a system architecture of a multi-engine intelligent data warehouse storage model construction system according to an embodiment of the present invention. The multi-engine intelligent data warehouse storage model construction system includes: an architecture management module 1, a sharding management module 2, a granularity evaluation module 3, an intelligent matching module 4, and a model deployment module 5.

[0026] Among them, the architecture management module 1 is the basic support module of the system. It is responsible for building the distributed storage architecture and completing the core configuration of the storage nodes. Its output architecture framework, node resource data and engine configuration information provide core basic support for the node hierarchical structure of the intelligent matching module 4 and the sharded storage of the model deployment module 5.

[0027] In some implementations, the architecture management module 1 includes an architecture setup submodule 11 and a node configuration submodule 12: The architecture building submodule 11 uses a distributed file system (such as Hadoop HDFS) as its technical foundation, such as... Figure 2 As shown, a distributed storage architecture is constructed, comprising one driver node (central coordinator) and multiple distributed storage nodes. The central coordinator is responsible for global task scheduling, storage node management, and subsequent access request distribution, ensuring the coordinated operation of the entire architecture. Multiple distributed storage nodes, serving as the core carriers of data storage and local computation, are configured according to preset deployment rules to guarantee the distributed nature of data storage.

[0028] The node configuration submodule 12 configures at least one data processing engine for each distributed storage node, covering a row-column hybrid storage engine, a massively parallel computing engine, an index database query engine, a memory caching engine, and an unstructured data extension processing engine (including dedicated engines such as a full-text indexing engine and an audio / video data processing engine), adapting to the processing needs of different types of unstructured data. Simultaneously, through real-time collection and statistical analysis, it identifies the node resources of each storage node, clarifies the local computing power (representing parallel computing and analysis processing capabilities) and network bandwidth (representing data transmission rate capabilities) of each node, and synchronizes the node resource information to the intelligent matching module 4.

[0029] Sharding Management Module 2 inherits the storage architecture built by Architecture Management Module 1. It performs preprocessing and sharding on the unstructured data to be stored. The extracted data features and generated data shards are the core inputs for access granularity evaluation by Granularity Evaluation Module 3, which directly determines the accuracy of subsequent granularity evaluation.

[0030] In some implementations, the sharding management module 2 includes a data preprocessing submodule 21, a sharding processing submodule 22, and a feature extraction submodule 23. The data preprocessing submodule 21 performs data cleaning, redundant information removal, and format regularization on the unstructured data to be stored (such as long videos, audio, medical images, etc.), removes invalid data interference, ensures that the data meets the requirements of subsequent fragmentation processing, and provides a guarantee for the integrity and consistency of fragmentation.

[0031] The fragmentation submodule 22 uses a fixed-length segmentation method to divide the preprocessed unstructured data into multiple consecutive data fragments, ensuring that adjacent data fragments maintain sequential continuity in the original unstructured data, facilitating data splicing and integrity verification during subsequent access. Simultaneously, a unique identifier is assigned to each data fragment for easy tracking and management.

[0032] The feature extraction submodule 23 extracts data features from each data segment, including user features and content features. User features include the uploader's age, gender, device information, and geographical location, obtained by parsing the information accompanying the data upload. Content features are extracted differently for different data types; text data is directly extracted as continuous data segments of a preset proportion, while audio and video data are sampled and analyzed using preset recognition technology and converted into keyword sequences. All data features are converted into feature vectors in a unified format to ensure the feasibility of subsequent similarity comparisons.

[0033] Granularity evaluation module 3, based on the data shards and data characteristics provided by shard management module 2, completes the access granularity evaluation by comparing the characteristics with historical data. The evaluation results (access granularity of each data shard) are directly transmitted to intelligent matching module 4, serving as the key basis for matching data shards with storage nodes and determining the accuracy of the matching rules.

[0034] In some implementations, the granularity evaluation module 3 includes a historical data matching submodule 31, a feature similarity calculation submodule 32, and an access granularity determination submodule 33. The historical data matching submodule 31 retrieves recent similar historical data stored in the system, compares the user characteristics of the current data segment extracted by the segment management module 2 with the user characteristics of the historical data, and selects historical data with user characteristic similarity greater than the preset similarity threshold as reference data to ensure the relevance and accuracy of subsequent access volume prediction.

[0035] The feature similarity calculation submodule 32 extracts the content feature vectors of the reference data and generates a sequence of content feature vectors according to the same sorting rules. It then calculates the sequence similarity between the current data shard and the content feature vector sequence of the reference data using a dynamic sequence matching algorithm. Simultaneously, it combines the historical access volume of the reference data (including access frequency and access range) to comprehensively predict the expected access volume of the current data shard.

[0036] The access granularity determination submodule 33 sets an access volume threshold, determines data fragments with expected access volume greater than the threshold as fine-grained fragments (higher input / output overhead during access), and determines data fragments with expected access volume less than or equal to the threshold as coarse-grained fragments (lower input / output overhead during access), and synchronizes the access granularity identifier of each data fragment to the intelligent matching module 4.

[0037] The intelligent matching module 4 is the core decision-making module of the system. It integrates the node resource information provided by the architecture management module 1 and the access granularity data provided by the granularity evaluation module 3 to establish intelligent matching rules. The rule results directly guide the sharding storage operation of the model deployment module 5, which is the core link to achieve accurate matching between data sharding and storage nodes.

[0038] In some implementations, the intelligent matching module 4 includes a node hierarchical submodule 41, a matching degree calculation submodule 42, and a sharding allocation rule submodule 43. The node grading submodule 41 uses the local computing power and network bandwidth of the storage nodes identified by the architecture management module 1 as the core grading indicators. It calculates the difference in indicators between any two storage nodes. Based on the storage node with the best local computing power, it sorts all storage nodes according to the difference in indicators and divides them evenly into multiple levels. Among them, the nodes with superior local computing power are classified as coarse-grained sharding adaptation level, and the nodes with superior network bandwidth are classified as fine-grained sharding adaptation level.

[0039] The matching degree calculation submodule 42 establishes a correlation based on the access granularity of the data shards and the average local computing power and average network bandwidth of the storage nodes within each level. As the access granularity increases, the matching degree is more influenced by the average local computing power; conversely, as the access granularity decreases, the matching degree is more influenced by the average network bandwidth. By determining the matching degree between each data shard and each node level, the level with the highest matching degree is taken as the matching level for that data shard.

[0040] The shard allocation rules submodule 43 constructs a hash ring structure for each level of storage nodes, formulates hash mapping algorithm rules, and clarifies the allocation logic of data shards. Simultaneously, it formulates rules for replica shard generation and allocation, determines the number of replicas for each data shard (e.g., 2-3), and stipulates that replica shards must not be stored on the same node as the original data shard or other replica shards to ensure storage reliability and load balancing.

[0041] The model deployment module 5, based on the intelligent matching rules established by the intelligent matching module 4, completes the landing and storage of data shards and forms the final storage model. At the same time, the generated storage index table provides support for subsequent access and is the core module for the implementation of system functions.

[0042] In some implementations, the model deployment module 5 includes a sharded storage submodule 51, a replica management submodule 52, and a storage index submodule 53. The sharded storage submodule 51 transmits each data shard to the corresponding storage node according to the matching hierarchy and hash mapping algorithm determined by the intelligent matching module 4. The data is then persisted through the storage module of the storage node. During the storage process, the node hierarchical adaptation rules are strictly followed: coarse-grained shards are stored on nodes with superior local computing power, while fine-grained shards are stored on nodes with superior network bandwidth.

[0043] The replica management submodule 52 generates a preset number of replica fragments for each data fragment according to the replica rules defined by the intelligent matching module 4, and allocates them to other storage nodes within the matching level through a hash mapping algorithm. If the node pointed to by the allocation result already stores the original data of the data fragment or other replica fragments, the replica fragment is automatically allocated to the next adjacent storage node on the hash ring to ensure the independence of replica storage.

[0044] The storage index submodule 53 records the storage location information (including the storage node and storage path) of each data shard and its replica shard, generating a data storage index table. This index table will be synchronized to the subsequent access processing module to support the rapid location of the target data shard during access. After all data shards and replicas are stored, the system automatically integrates the architecture configuration, matching rules, and storage distribution information to form a complete multi-engine intelligent data warehouse storage model.

[0045] In addition, the multi-engine intelligent data warehouse storage model construction system may also include: an access processing module 6. The access processing module 6 takes over the storage model and data storage index table generated by the model deployment module 5, and combines the node resource information of the architecture management module 1 and the access granularity data of the granularity evaluation module 3 to achieve efficient access processing of stored data, completing the entire closed loop of storage and access.

[0046] In some implementations, the access processing module 6 includes a request receiving submodule 61, a node location submodule 62, and a computation scheduling submodule 63. The request receiving submodule 61 receives the access request message initiated by the user, parses the target data identifier in the request, and clarifies the range of target data that needs to be accessed.

[0047] Based on the data storage index table generated by the model deployment module 5, the node positioning submodule 62 quickly locates the storage nodes of the data shards and their replica shards corresponding to the target data, and obtains the current remaining node resources (remaining local computing power, remaining network bandwidth) of the storage nodes.

[0048] The computation scheduling submodule 63 executes the corresponding processing mechanism based on the access granularity of the target data shards determined by the granularity evaluation module 3 and the remaining node resources of the storage nodes. If it is a coarse-grained shard and the remaining local computing power is greater than the preset computing power (e.g., half of the total local computing power), the data processing engine configured on the storage node performs local computation and returns the result; if it is a fine-grained shard and the remaining network bandwidth is greater than the preset bandwidth (e.g., half of the total network bandwidth), the key data is transmitted to the cloud computing node through the network bandwidth to complete the processing and return the result, ensuring low input / output overhead during the access process.

[0049] Please see Figure 3 The diagram illustrates a flowchart of a method for constructing a multi-engine intelligent data warehouse storage model according to an embodiment of the present invention. This method includes: S1. Construct a distributed storage architecture that includes multiple storage nodes.

[0050] Among them, distributed storage architecture such as Figure 2As shown, a mature distributed file system (such as Hadoop HDFS) is used as the core support. This technology platform has high reliability, high scalability, and massive data storage capabilities, which can meet the large-scale storage needs of unstructured data (such as long videos, audio, medical images, etc.). Based on this technology platform, an architecture framework of "driver node + multiple distributed storage nodes" is built. The driver node acts as a central coordinator, and through the deployment of a global task scheduling system, node status monitoring module, and access request distribution component, it realizes unified management of the entire distributed storage architecture. Its core role is to coordinate the running status of each storage node, allocate storage and computing tasks, and forward user access requests. The effect of this design is to ensure the coordination and orderliness of the architecture operation and avoid problems such as chaotic node scheduling or uneven task distribution. Multiple distributed storage nodes are distributed and configured according to the principles of proximity deployment and load balancing. For example, according to the distribution characteristics of data access, several storage nodes are deployed in different regions to provide hardware support for the proximity storage and access of subsequent data shards. The effect of this layout is to shorten the data transmission path and initially reduce network latency during the access process.

[0051] Each distributed storage node is configured with at least one data processing engine. Depending on the type of unstructured data and processing requirements, optional engines include a row-column hybrid storage engine, a massively parallel computing engine, an index database query engine, a memory caching engine, and an unstructured data extension processing engine. The unstructured data extension processing engine further encompasses a full-text indexing engine (suitable for text-based data), an audio / video data processing engine (suitable for audio / video data), a spatial geographic information processing engine (suitable for spatial geographic data), a graph-Rag knowledge base construction engine, a text-to-vector database engine, and an intelligent question-answering indicator analysis engine. Each storage node can flexibly combine and configure engine types according to its preset functional positioning (e.g., focusing on local computation or data transmission). For example, storage nodes focusing on local computation can prioritize configuring a massively parallel computing engine, while storage nodes focusing on data transmission can be paired with a memory caching engine to improve data read / write speeds. This configuration enables each storage node to have targeted data processing capabilities, providing engine support for subsequent local computation in coarse-grained sharding and fast read / write in fine-grained sharding, avoiding low data processing efficiency due to a single engine configuration.

[0052] Simultaneously, the node resources of each storage node are collected, statistically analyzed, and identified. Node resources specifically include local computing power and network bandwidth. Local computing power is calculated by collecting hardware parameters such as the number of CPU cores, graphics processing unit (GPU) configuration, memory capacity, and peak parallel computing performance of the storage node. The average available computing power of the node over the past week is calculated to quantify its ability to perform parallel computing and analysis of data. Network bandwidth is calculated by monitoring the bidirectional data transfer rates between the storage node and the central coordinator, other storage nodes, and cloud computing nodes in real time. The average transmission bandwidth value over the past week is calculated to quantify the data transfer capacity of the storage node. The calculated local computing power and network bandwidth values ​​are bound to the corresponding storage nodes and synchronously stored in the central coordinator's node resource database to avoid resource waste or low access efficiency caused by subsequent blind allocation.

[0053] S2. Perform sharding on the unstructured data to be stored to obtain multiple data shards, and obtain the data characteristics of each data shard.

[0054] In some implementations, preprocessing is first performed on the unstructured data (such as long videos, audio, medical images, and satellite remote sensing data) to be stored in the distributed storage architecture. Since unstructured data often suffers from inconsistent formats and contains invalid and redundant information (such as silent segments in audio and video, repeated characters in text, and garbled text generated during data transmission), data cleaning tools are used to remove this invalid information. Simultaneously, unstructured data from different sources and in different formats undergo format normalization (e.g., converting audio with different encoding formats to a unified format, and adjusting images with different resolutions to a preset standard pixel size). This eliminates data noise interference and ensures the integrity and consistency of the data to be segmented.

[0055] Secondly, a fixed-length partitioning method is used to partition the unstructured data to be processed. Taking into account the storage optimization features of distributed file systems (such as the default 128MB partition size in Hadoop HDFS), a uniform partition length is set based on the total capacity of the unstructured data to be stored and the storage efficiency requirements of the storage nodes (which can be flexibly adjusted to 64MB, 256MB, etc.). Following the order of the data in the original files, the preprocessed unstructured data is continuously partitioned into multiple data partitions, ensuring that adjacent data partitions maintain sequential continuity in the original unstructured data, and each data partition is assigned a unique identifier (such as data type + timestamp + partition sequence number). The effect of this partitioning action is to break down massive amounts of unstructured data into uniformly sized, easily manageable independent units, adapting to the storage characteristics of the distributed storage architecture.

[0056] Finally, for each data shard, data features are extracted to form a complete set of data features, which are then bound to the identity identifier of the corresponding data shard and synchronously transmitted to the granularity evaluation module. These data features may include user features and content features.

[0057] When extracting user features, the core attributes of the data uploader are obtained by parsing the metadata information recorded during the data upload process. Specifically, these include the uploader's age, gender, device information (such as device model and operating system), and geographical location (parsed based on Internet Protocol (IP) address or device location information).

[0058] When extracting content features, differentiated extraction strategies are adopted for different types of data slices: For text data slices, a predetermined proportion (e.g., 30%) of continuous data segments are randomly selected as feature carriers; for audio data slices, audio sampling technology is used to uniformly collect audio segments from multiple time points, which are then converted into text keyword sequences through a speech recognition model; for image or video data slices, key visual elements (such as object outlines, color features, and scene information) are extracted through an image recognition model and converted into keyword sequences. Subsequently, a bag-of-words model or vector coding technology is used to uniformly convert the content features of all data slices into fixed-dimensional feature vectors, also known as standardized feature vectors, to achieve a unified representation of the content features of different types of data slices.

[0059] S3. By comparing the similarity of data features between each data shard and historical data, the access granularity of each data shard is evaluated.

[0060] The access granularity is used to characterize the input and output overhead of data shards during access.

[0061] In some implementations, similar unstructured data stored in the system recently (e.g., within one week) is first retrieved as candidate historical data, and the corresponding data features of all candidate historical data are also converted into standardized feature vectors of the same dimension to ensure the feasibility and accuracy of feature comparison.

[0062] Secondly, the feature similarity between the data shards and candidate historical data is calculated. A preset feature vector similarity algorithm (such as cosine similarity) is used to compare the standardized feature vector of each data shard with the standardized feature vector of each candidate historical data to obtain the feature similarity between them. If multiple data features exist, the average similarity of the multiple data features is used as the feature similarity between the data shard and the candidate historical data. A similarity threshold (such as 0.8) is set, and candidate historical data with feature similarity values ​​greater than this threshold are selected as reference data. This ensures that the reference data has a high correlation with the current data shard, accurately identifying historical data with features similar to the current data shard and ensuring that the prediction results closely match the actual access scenario.

[0063] Next, the expected access volume of the data shards is predicted based on the similarity results. A weight coefficient is assigned to each reference data point; this weight coefficient is positively correlated with the feature similarity between the current data shard and the reference data, meaning the higher the similarity, the larger the weight coefficient. Then, the recent historical access volume of each reference data point is retrieved (the total frequency of complete access to the data is counted), and the expected access volume of the current data shard is calculated by weighted summation of the historical access volumes using the weight coefficients.

[0064] Finally, the access granularity of data shards is determined based on the expected access volume. Access granularity refers to the size and fineness of a processor's or system's access to memory or data resources in a computer system, and it can characterize the input / output overhead during access. For data with a larger access volume, finer-grained sharding (corresponding to a smaller data volume accessed per task) can adapt to distributed parallel processing and high-concurrency, low-latency requirements, but with higher input / output overhead during access; conversely, for data with a smaller access volume, coarser-grained access results in lower input / output overhead. Furthermore, an access volume threshold can be dynamically set based on historical access data distribution characteristics, such as using the median or average of historical access volumes as the threshold. If the expected access volume of the current data shard is greater than this threshold, it is determined to be a fine-grained shard with high I / O overhead; if the expected access volume is less than or equal to this threshold, it is determined to be a coarse-grained shard with low I / O overhead. The access granularity result of each data shard is bound to the corresponding data shard identity identifier, clarifying the access characteristics and input / output overhead scale of each data shard.

[0065] S4. Based on the access granularity of each data shard and the node resources of the storage node, establish intelligent matching rules between data shards and storage nodes.

[0066] In some implementations, the resource representation and matching priority logic for a single storage node are first clearly defined. The node resources (local computing power and network bandwidth) of each storage node are retrieved as the matching basis. Combined with the access granularity, resource matching priorities are set: if the data sharding is coarse-grained (high IO overhead, dependent on local computation), local computing power is used as the primary matching indicator, and network bandwidth as the secondary indicator; if the data sharding is fine-grained (high access frequency, dependent on data transmission), network bandwidth is used as the primary matching indicator, and local computing power as the secondary indicator. This ensures that the matching logic is directly linked to the sharding access requirements.

[0067] Secondly, for each data shard, based on preset weight allocation rules (e.g., coarse-grained shards: local computing power weight 0.7, network bandwidth weight 0.3; fine-grained shards: network bandwidth weight 0.7, local computing power weight 0.3), a weighted summation is used to obtain the matching degree between the data shard and each storage node. The nodes are sorted from highest to lowest matching degree, and the top three nodes are selected as candidate target nodes to avoid the risk of load concentration or failure caused by matching a single node.

[0068] Finally, a global hash ring is constructed to achieve balanced distribution. All storage nodes are incorporated into a globally consistent hash ring, and each storage node is assigned a unique hash identifier and mapped onto the ring. Simultaneously, each data shard (including replica shards) is assigned a unique identity identifier. A hash mapping algorithm is used to associate the identity identifier of the data shard with the hash identifier of the node on the hash ring, determining the final storage node from candidate target nodes. Specifically, this involves filtering candidate nodes on the ring that are closest to the shard's hash value, and verifying whether the candidate node's real-time remaining resources meet a preset threshold. If the node's resources meet the threshold, it is directly determined as the final storage node; otherwise, subsequent candidate nodes are checked sequentially according to their matching degree. If none meet the threshold, a node with a lower matching degree ranking but meeting the resource threshold is selected from the global storage nodes as a fallback, ensuring that the final storage node both meets the data shard access requirements and has sufficient capacity. Ultimately, this ensures that data shards are evenly distributed across all storage nodes, avoiding excessive load on local nodes.

[0069] S5. According to the intelligent matching rules, each data shard is stored to the corresponding storage node to form a multi-engine intelligent data warehouse storage model.

[0070] In some implementations, based on the final storage node determined by the intelligent matching rules, each data fragment, along with its associated data characteristics and access granularity identifier, is synchronously transmitted to the corresponding storage node via a distributed transmission protocol (such as Transmission Control Protocol (TCP) / IP). Upon receiving the data, the storage node completes the structured storage of the data fragments using its configured data processing engine (such as a row-column hybrid storage engine or a memory caching engine). Simultaneously, it performs integrity verification on the stored data (by verifying that the hash value of the data fragment is consistent with that before transmission), ensuring that the data fragments are not lost or tampered with during transmission and storage.

[0071] Finally, after all data shards have been stored and verified, the central coordinator aggregates all key information, including the topology of the distributed storage architecture, the engine configuration and node resource ledger of each storage node, the storage location (node ​​identifier, storage path), access granularity, data characteristics, and intelligent matching rule details of each data shard (including replicas), and constructs a unified system metadata management library and data storage index table. This information is then integrated with the distributed storage architecture, multi-engine configuration, and data shard storage system to form a complete multi-engine intelligent data warehouse storage model. This model possesses the core characteristic of precise adaptation to storage, resource, and access requirements, enabling it to directly respond to subsequent data access requests and achieve a low-IO-overhead, highly adaptable storage and access closed loop.

[0072] Based on the above technical solution, a distributed storage architecture with multiple data processing engines and identified node resources is constructed. The access granularity is evaluated by combining the characteristics of data sharding with the similarity of historical data. Then, intelligent matching rules are established based on the access granularity and node resources to complete sharded storage, forming a multi-engine intelligent data warehouse storage model. This effectively avoids the drawbacks of random storage of data shards in existing technologies, achieves accurate matching between data shards and storage node resources, reduces invalid data transfer during data access, and significantly reduces input / output overhead and access costs. At the same time, relying on multi-engine configuration and intelligent matching logic, the adaptability and efficiency of unstructured data storage and access are improved.

[0073] In one possible implementation, combining Figure 3 ,like Figure 4 As shown, the method in S4 above can be specifically implemented through the following steps S41 to S43, which are explained in detail below: S41. Based on the node resources of the storage nodes, cluster and classify multiple storage nodes.

[0074] In some implementations, the node resources of all storage nodes are first retrieved, and a standardized algorithm (such as the Min-Max normalization method) is used to uniformly transform all indicators to the [0, 1] interval, eliminating the scale difference of different resource indicators, avoiding the clustering results from being biased towards a certain indicator due to different numerical magnitudes, and ensuring the fairness and accuracy of the classification basis.

[0075] Then, considering the discreteness of storage node resource distribution and the flexibility of matching requirements, the K-means clustering algorithm was selected as the core hierarchical algorithm. The K-means clustering algorithm is computationally efficient and adaptable to numerical data clustering, quickly aggregating nodes with similar resource characteristics into clusters. Based on the system's storage scale and resource matching accuracy requirements, the number of hierarchical levels is preset (e.g., 3-7 levels). A smaller preset level emphasizes local computing power advantages, a larger preset level emphasizes network bandwidth advantages, and intermediate levels are balanced, ensuring differentiation between levels while avoiding excessive levels that would increase matching complexity.

[0076] Perform clustering operations and generate initial hierarchies. Using the standardized local computing power and network bandwidth data of all storage nodes as input, the K-means clustering algorithm is applied: first, initial cluster centers are randomly selected (each center corresponds to a resource baseline for a hierarchy); then, the Euclidean distance between each storage node and each cluster center is iteratively calculated, assigning the node to the cluster containing the nearest cluster center. The cluster centers are continuously updated until the convergence condition is met (the offset of the cluster centers between two adjacent iterations is less than a preset threshold). After iteration, independent node clusters are obtained, and each cluster represents an initial hierarchy.

[0077] Alternatively, resource differences can be quantified using the product of the absolute differences of various metrics between storage nodes, instead of Euclidean distance, as follows: ; In the formula, This represents the normalized value of the local computing power of storage node A (obtained through maximum and minimum value normalization). This represents the normalized value of the local computing power of storage node B (obtained through maximum and minimum value normalization). This represents the normalized value of the network bandwidth of storage node A (obtained by normalization using maximum and minimum values). This represents the normalized value of the network bandwidth of storage node B (obtained through maximum-minimum normalization). Multiplying the absolute values ​​of the two single-dimensional differences yields the comprehensive difference in node resources between storage node A and storage node B. The node is assigned to the cluster containing the cluster center with the smallest difference.

[0078] Finally, the average local computing power and average network bandwidth of all storage nodes within each initial tier are calculated, and the tier boundaries are optimized using the average as the core indicator. If the average computing power of a tier differs too little from that of adjacent tiers, the cluster centers are adjusted and the iteration is repeated to ensure that the resource advantages of each tier are significantly differentiated. Specifically, the average computing power of all tiers can be sorted from largest to smallest, and the difference between adjacent tiers can be calculated. The difference threshold is quantified based on the global standard deviation of the difference (e.g., 0.3 times the global standard deviation is used as the threshold). If the difference between adjacent tiers is less than the threshold, it is considered that the difference is too small. Then, the two original cluster centers of adjacent tiers are extracted, and according to a fixed adjustment range (e.g., 0.05), the cluster centers of the previous tier are moved towards the direction with greater computing power (e.g., to the node with the nearest computing power greater than the cluster center), and the cluster centers of the next tier are moved towards the direction with less computing power (e.g., to the node with the nearest computing power less than the cluster center). The clusters are then re-clustered using the adjusted cluster centers until the differences between adjacent tiers all meet the standard.

[0079] After optimization, each level is assigned a clear identifier and adaptation direction. Taking five levels as an example, levels 1-2 are defined as coarse-grained sharding adaptation level, which adapts to coarse-grained sharding that relies on local computation; levels 4-5 are defined as fine-grained sharding adaptation level, which adapts to fine-grained sharding that relies on data transmission; and level 3 is defined as balanced adaptation level, which adapts to sharding with unclear access requirements or serves as a fallback storage node.

[0080] S42. Based on the access granularity of the data shards and the node resources of the storage nodes within each level, determine the matching degree between the data shards and each level, and take the level with the highest matching degree as the matching level of the data shards.

[0081] In some implementations, a first association is made between the access granularity of data shards and the average local computing power of the storage nodes within each tier. This first association is configured such that the degree of influence of the average local computing power increases as the access granularity increases. Simultaneously, a second association is made between the access granularity of data shards and the average network bandwidth of the storage nodes within each tier. This second association is configured such that the degree of influence of the average network bandwidth increases as the access granularity decreases.

[0082] Specifically, resource weights are dynamically allocated based on the first and second association rules, ensuring that the sum of local computing power weight and network bandwidth weight is always 1, and adjusting the access granularity. Defined as an evaluation value with a range of [0, 1], it can be... As a weight for the local average computing power As a weight for the average network bandwidth, the larger the access granularity, the greater the weight of the average local computing power, and the smaller the weight of the average network bandwidth.

[0083] Then, based on the results of the first and second associations, the matching degree between the data shards and each level is determined. Specifically, this can be calculated using a weighted summation method, expressed as: ; In the formula, This represents the evaluation value of the access granularity of data shard a. The value range is [0, 1], and it is dimensionless. The larger the value, the coarser the access granularity.

[0084] This represents the mean of the normalized local computing power of all storage nodes in level E (obtained through maximum-minimum normalization). The normalized value of the network bandwidth of all storage nodes (obtained by normalization through maximum and minimum values) is the mean, which is dimensionless.

[0085] Perform a linear weighted calculation to obtain the matching degree between data partition a and level E. This allows for dynamic allocation of resource weights. The higher the matching degree, the better the resources (local computing power or network bandwidth) of level E can meet the access granularity requirements of data shard a.

[0086] S43. Construct a hash ring structure for each storage node within a hierarchy, and use a hash mapping algorithm to distribute data shards to storage nodes within the matching hierarchy.

[0087] First, a consistent hash ring is constructed for each storage node within a tier. For any node tier (e.g., coarse-grained sharding adaptation tier, fine-grained sharding adaptation tier), a preset hash algorithm (e.g., MD5, SHA-256) is used to hash the node identifier of each storage node to obtain the node hash value. The node hash value is then mapped onto the hash ring, logically organizing the storage nodes within the tier into a ring topology, resulting in the ring position indices of the storage nodes from 1 to n (where n is the number of storage nodes within the node tier).

[0088] Next, hash mapping is performed on the data shards. The same hash algorithm as the storage nodes is used to hash the identity of the data shards to obtain the shard hash value. The shard hash value is then mapped to the corresponding hash ring of the hierarchy to obtain the ring position index 1 to m of the data shard (m is the number of data shards).

[0089] Then, the numbering remainder algorithm is used, which determines the resource allocation relationship by taking the remainder (m%n) of the data shard number with respect to the total number of nodes. The data shard is then allocated to the node for storage, thus achieving a uniform distribution of data shards among the storage nodes in the hierarchy.

[0090] For example, suppose there are 3 storage nodes in a certain node hierarchy (node ​​numbers: node1, node2, node3), and a total of 5 data shards to be allocated (data shard numbers: shard1, shard2, ..., shard5). Shard1 determines the target storage node as node1 by calculating 1%3=1; shard2 determines the target storage node as node2 by calculating 2%3=2; shard3 determines the target storage node as node3 by calculating 3%3=0 (the remainder 0 corresponds to the total number of nodes); shard4 determines the target storage node as node1 by calculating 4%3=1; and shard5 determines the target storage node as node2 by calculating 5%3=2.

[0091] In some implementations, at least one replica fragment is generated for each data fragment, and each replica fragment carries the same identification information as the original data fragment. When allocating replica fragments using a hash mapping algorithm, if the allocated storage node already stores the data fragment or other replica fragments, the replica fragment is allocated to the next adjacent storage node on the hash ring (searching sequentially in a clockwise direction until a node that does not store the data fragment and its replica is found), ensuring that the replica fragment and the original data fragment are stored on different nodes, thus avoiding data loss due to the failure of a single node.

[0092] Based on the above technical solution, by clustering and classifying storage nodes according to resource characteristics, dynamically calculating the matching degree between data shard access granularity and classification and selecting the optimal classification, and constructing hash rings within the classification to achieve balanced allocation of data shards, the precise matching between storage node resources and data shard access requirements is achieved. This avoids load imbalance caused by centralized data storage and improves data reliability through distributed replica storage. It lays the foundation for resource adaptation and load balancing for efficient storage and subsequent access in multi-engine intelligent data warehouses, significantly improving storage resource utilization and data access efficiency.

[0093] In one possible implementation, combining Figure 3 ,like Figure 5 As shown, the above method may also include the following S6, which will be explained in detail below: S6. Based on the access granularity of data sharding and the node resources of storage nodes, formulate the access processing mechanism of the multi-engine intelligent data warehouse storage model.

[0094] In some implementations, the access processing mechanism includes: locating the storage node of the data shard to be accessed based on the principle of proximity between the storage node and the driver node (such as the lowest network latency or the closest physical location).

[0095] If the access granularity of a data shard is greater than a preset granularity threshold (e.g., set to 0.5, where the evaluation range of the access granularity is [0, 1]), and the remaining local computing power of the corresponding storage node is greater than the preset computing power (which can be defined as half of the total available computing power), the computing task is scheduled to the storage node for local computation through the data processing engine and the results are returned. The data locality of coarse-grained sharding can be leveraged to improve processing efficiency through parallel computing, reduce the cost of migrating data to the cloud, and decrease the number of IO accesses.

[0096] If the access granularity of a data shard is less than a preset granularity threshold, and the remaining network bandwidth of the corresponding storage node is greater than the preset bandwidth (which can be defined as half of the total available bandwidth), the data shard is transmitted to the cloud computing node via bandwidth for processing and the result is returned. This leverages the massive computing power of the cloud to offload local loads while ensuring real-time data transmission with sufficient bandwidth, adapting to the high access frequency requirements of fine-grained sharding.

[0097] The above mechanisms can clearly define the boundaries of data processing and ensure that storage nodes have sufficient capacity to process tasks by using remaining resource thresholds, thus avoiding resource overload.

[0098] Based on this, upon receiving an access request message for the target data, the driving node decomposes the request into multiple smaller tasks for data shards, thus achieving task parallelization. For each data shard, the storage node is located from the multi-engine intelligent data warehouse storage model based on the principle of proximity.

[0099] Then, after the storage node receives the access request of the small task, it performs data access on the data fragments of the target data according to the access granularity of the data fragments of the target data and the remaining node resources of the storage node, in accordance with the access processing mechanism.

[0100] Furthermore, if the remaining resources of the current storage node are temporarily insufficient (e.g., coarse-grained sharding with insufficient remaining computing power, or fine-grained sharding with insufficient remaining bandwidth) and the data shard has replicas, the task will be preferentially scheduled to other storage nodes with the same data shard. If the remaining resources of all storage nodes with the same data shard are insufficient, the task can be temporarily scheduled to other storage nodes with sufficient resources (prioritizing those within the same tier) or cloud nodes to ensure that the request is eventually completed. This fallback strategy ensures the reliability of access and enables elastic processing of data access.

[0101] Based on the above technical solution, by formulating an access mechanism of coarse-grained local parallel computing and fine-grained cloud transmission processing, combined with task decomposition, proximity positioning and dynamic resource verification, the precise adaptation of local and cloud resources is achieved. This not only reduces data migration and IO access times by leveraging local parallel computing, but also distributes local load through cloud elasticity, ultimately improving the data access efficiency and resource utilization of the multi-engine intelligent data warehouse.

[0102] In one possible implementation, combining Figure 3 ,like Figure 6 As shown, the method in S3 above can be specifically implemented through the following steps S31 to S33, which are explained in detail below: S31. Compare the similarity of user features between the data shards and historical data, and determine the historical data whose user feature similarity is greater than the preset similarity threshold as reference data.

[0103] In some implementations, user features are first standardized. Since the extracted user features contain different types of data, such as categorical features like access preference tags and data demand types, and numerical features like historical access frequency, direct calculation would be affected by differences in units. Categorical features are converted to binary values ​​(0 or 1) using one-hot encoding, while numerical features like historical access frequency are mapped to the [0, 1] interval using Min-Max normalization. Then, the differences between standardized feature values ​​are calculated to measure similarity, expressed as: ; In the formula, The standardized feature value represents the i-th user feature of the k-th historical data. This represents the standardized feature value of the i-th user feature in the data shard a to be stored; n represents the number of user features.

[0104] First, for the i-th user feature, calculate The absolute difference between data shard a and historical data in a single label dimension is quantified; the greater the difference, the larger the value. Then, the absolute differences of all n user features are summed to obtain the total absolute difference sum. The larger the total difference sum, the greater the overall difference between data shard a and the user features of the k-th historical data.

[0105] Finally, the total absolute difference is negativeized and then exponentially calculated. The similarity of user features between data shard a and the k-th historical data point is obtained. The range is (0, 1). Since the exponential function is monotonically increasing, the larger the total difference, the smaller the similarity.

[0106] Simultaneously, an empirical threshold (e.g., 0.7) is set based on the similarity verification results of historical data to filter valid reference data. If no historical data with a similarity greater than the preset similarity threshold exists, the top preset number of historical data with the highest similarity can be determined as reference data.

[0107] The filtered reference data (which can be a complete dataset or a data shard) has a high degree of matching with the data shard to be stored in terms of user characteristics, and contains complete key related information such as storage node information, access processing mechanism, and resource usage. This can provide direct and reliable historical basis for the formulation of current unstructured data storage strategies and the adaptation of access mechanisms.

[0108] S32. Compare the similarity of content features between the data shards and the reference data, and combine the historical access volume of the reference data to determine the expected access volume of the data shards.

[0109] In some implementations, content feature vectors are first extracted from data fragments and reference data, and then a sequence of content feature vectors is generated according to the same sorting rules. Specifically, considering the large volume of unstructured data (such as audio, images, and long text), a portion of the content (such as keyframes) can be selected as feature extraction samples. The extraction method should also be tailored to the data type: text data is directly extracted as segments; audio data is converted into keyword descriptions using speech recognition technology; and image data has visual features (such as object outlines and color distribution) extracted using image recognition models and converted into text keywords. Then, a bag-of-words model is used to transform these text descriptions into content feature vectors with consistent dimensions. Finally, a sequence of content feature vectors is generated according to the same sorting rules (such as the data generation order) to establish a consistent comparison benchmark, avoiding sequence matching bias caused by different dimensional orders, and ensuring that similarity calculations focus on the differences in the content features themselves.

[0110] Then, a dynamic sequence matching algorithm is used to determine the sequence similarity between the content feature vector sequences of the data shards and the reference data, which is then used as the content feature similarity. Specifically, the dynamic time warping (DTW) algorithm, adapted for high-dimensional sequence comparison, can be selected. This algorithm can accurately calculate the dynamic warping distance between the content feature vector sequences of the data shards and the reference data by elastically aligning corresponding feature points in the sequences. To quantify the similarity, the dynamic warping distance is substituted into the mapping formula: ; In the formula, This represents the sequence of content feature vectors for the b-th reference data. This represents the sequence of feature vectors representing the content of the data fragment 'a' to be stored.

[0111] This indicates that the dynamic warping distance between two high-dimensional sequences is calculated using the dynamic time warping function; the larger the distance, the greater the difference. Furthermore, in the DTW algorithm, vector Euclidean distance replaces other distance metrics (such as Manhattan distance) in DTW as the distance metric between points, amplifying the differences between vectors through square and root operations.

[0112] Finally, the exponential operation is performed after negating the dynamic regularization distance. The sequence similarity between the content feature vector sequences of data segment a and the b-th reference data is obtained, i.e., the content feature similarity. The range is (0, 1). Since the exponential function is monotonically increasing, the larger the dynamic regularization distance, the smaller the similarity.

[0113] Following this logic, the similarity of content features between each data shard and each reference data is calculated one by one.

[0114] Finally, the expected access volume of the data shards is calculated by combining the historical access volume of the reference data, and is expressed as follows: ; In the formula, This represents the content feature similarity between data shard a and the b-th reference data. This represents the sum of content feature similarities between data shard 'a' and all reference data. This represents the historical access count of the b-th reference data, and m is the number of reference data.

[0115] Divide the content feature similarity of the b-th reference data by the sum of the content feature similarities of all reference data, and use this ratio as the weight of the b-th reference data, along with the historical access volume of the b-th reference data. Multiply by the data to obtain the contribution value of the b-th reference data to the access volume of data shard a. .

[0116] Then, sum the contribution values ​​of all reference data to obtain the expected access volume of data shard a. This approach not only reflects the relevance of the content but also takes into account the actual access patterns of historical data.

[0117] In other implementations, the expected access volume of a data shard can be calculated by combining the similarity of user characteristics and content characteristics between the data shard and historical data with the historical access volume of the reference data, as follows: ; In the formula, This represents the similarity of user features between data shard a and the b-th reference data. This represents the similarity of user features between data shard 'a' and all reference data. This represents the content feature similarity between the content feature vector sequences of data slice a and the b-th reference data. This represents the sum of content feature similarities between the content feature vector sequences of data slice 'a' and all reference data. This represents the historical access count of the b-th reference data, and m is the number of reference data.

[0118] Based on the previous formula, the expected access volume of data segment a is obtained by weighting the data twice based on user feature similarity. By evaluating expected traffic through two dimensions—user characteristics and content characteristics—the results more closely reflect the actual traffic scenarios following data sharding.

[0119] S33. Determine the access granularity of the data shards based on the expected access volume of the data shards.

[0120] In some implementation methods, the negative correlation between access volume and access granularity is first clarified: the higher the access volume, the more likely the data is to be the core of real-time interaction, and it needs to be split into fine-grained parts (small data volume per access) to adapt to the requirements of distributed parallel processing and high concurrency and low latency, so as to avoid system overload caused by large-scale data read and write; the lower the access volume, the more likely it is to be aggregated and stored in a coarse-grained manner to reduce data management and data migration overhead and improve resource utilization.

[0121] The expected access volume for each data shard is normalized (e.g., min-max normalization), and then the access granularity of the data shard is determined based on the normalized result of the expected access volume, expressed as: ; In the formula, This indicates the granularity of accessing data shard 'a'. This indicates the expected number of visits to data shard a. This represents the normalized result of the expected access volume for data shard a, with a value range of [0, 1]. A higher access volume indicates a finer granularity of access calls.

[0122] Based on the above technical solution, by first screening recent reference data that meets the similarity criteria with the current data shard features, then weighting the historical access volume of the reference data based on the content feature sequence similarity to obtain the accurate expected access volume, and finally determining the access granularity of the data shard based on the negative correlation between the expected access volume and the access granularity, the objectivity and accuracy of the access granularity determination are achieved. This provides a reliable quantitative basis for the subsequent matching of data shards and storage nodes and the formulation of access processing mechanisms, and helps the multi-engine intelligent data warehouse to achieve resource adaptation and access efficiency optimization.

[0123] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0125] In this embodiment of the invention, the multi-engine intelligent data warehouse storage model construction device can be divided into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0126] This invention also provides a hardware structure diagram of a multi-engine intelligent data warehouse storage model construction device, see [link / reference]. Figure 7 The multi-engine intelligent data warehouse storage model building device 700 includes a processor 701, and optionally, a memory 702 connected to the processor 701.

[0127] In the first possible implementation, see Figure 7 The multi-engine intelligent data warehouse storage model construction device 700 also includes a transceiver 703. The processor 701, memory 702, and transceiver 703 are connected via a bus. The transceiver 703 is used to communicate with other devices or communication networks. Optionally, the transceiver 703 may include a transmitter and a receiver. The device in the transceiver 703 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of the present invention. The device in the transceiver 703 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of the present invention.

[0128] Based on the first possible implementation method Figure 7 The structural diagram shown can be used to illustrate the structure of the multi-engine intelligent data warehouse storage model construction device involved in the above embodiments.

[0129] in, Figure 7The system chip in the multi-engine intelligent data warehouse storage model building device can also be illustrated. In this case, the actions performed by the aforementioned multi-engine intelligent data warehouse storage model building device can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0130] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in this embodiment can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0131] The processor in this invention may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a System-on-a-Chip (SoC), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.

[0132] The memory in the embodiments of the present invention may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0133] This invention also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0134] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0135] This invention also provides a chip, which includes a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0136] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0137] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In this invention, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several of the functions listed in this invention.

[0138] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for constructing a multi-engine intelligent data warehouse storage model, characterized in that, include: Construct a distributed storage architecture that includes multiple storage nodes; Each storage node is configured with at least one data processing engine and is identified as a node resource for data processing. The unstructured data to be stored is sharded to obtain multiple data shards, and the data characteristics of each data shard are obtained. The access granularity of each data shard is evaluated by comparing the similarity of data features between each data shard and historical data; the access granularity is used to characterize the input and output overhead of the data shard during access. Based on the access granularity of each data shard and the node resources of the storage node, intelligent matching rules are established between data shards and storage nodes, including: Based on the node resources of the storage nodes, multiple storage nodes are clustered and classified. Based on the access granularity of the data shards and the node resources of the storage nodes within each tier, the matching degree between the data shards and each tier is determined, including: establishing a first association between the access granularity of the data shards and the average local computing power of the storage nodes within each tier; the first association is configured such that: as the access granularity increases, the degree to which the matching degree is affected by the average local computing power increases. A second correlation is established between the access granularity of the data shards and the average network bandwidth of the storage nodes within each tier; the second correlation is configured such that the degree to which the matching degree is affected by the average network bandwidth increases as the access granularity decreases. Based on the results of the first and second associations, the matching degree between the data shards and each level is determined, and the level with the highest matching degree is taken as the matching level of the data shard. A hash ring structure is constructed for each storage node within a tier, and data shards are allocated to the matching storage nodes within the tier using a hash mapping algorithm; Each data shard is stored to the corresponding storage node according to the intelligent matching rules, forming a multi-engine intelligent data warehouse storage model; Based on the access granularity of data sharding and the node resources of storage nodes, an access processing mechanism for the multi-engine intelligent data warehouse storage model is formulated, including: If the access granularity of the data shard is greater than the preset granularity threshold, and the remaining local computing power of the corresponding storage node is greater than the preset computing power, the computing task will be scheduled to the storage node so that the data processing engine can perform local computing and return the result. If the access granularity of the data shard is less than the preset granularity threshold, and the remaining network bandwidth of the corresponding storage node is greater than the preset bandwidth, the data shard will be transmitted to the cloud computing node through the bandwidth to complete the processing and return the result.

2. The method for constructing a multi-engine intelligent data warehouse storage model according to claim 1, characterized in that, Also includes: Receive an access request message that requests access to the target data; Locate the storage node for the data fragments of the target data; Based on the access granularity of the data fragments of the target data and the remaining node resources of the storage nodes, data access is performed on the data fragments of the target data according to the access processing mechanism.

3. The method for constructing a multi-engine intelligent data warehouse storage model according to claim 1, characterized in that, For each tier of storage nodes, a hash ring structure is constructed. Data shards are then allocated to matching storage nodes within the tier using a hash mapping algorithm, including: For each data shard, generate at least one replica shard; When allocating replica shards using a hash mapping algorithm, if the allocated storage node already stores data shards or other replica shards, the replica shard will be allocated to the next adjacent storage node on the hash ring.

4. The method for constructing a multi-engine intelligent data warehouse storage model according to claim 1, characterized in that, The data features include user features and content features; The access granularity of each data shard is evaluated by comparing the similarity of data features between each data shard and historical data, including: Compare the similarity of user features between the data shards and historical data, and determine the historical data whose user feature similarity is greater than the preset similarity threshold as reference data; By comparing the similarity of content features between the data shards and the reference data, and combining the historical access volume of the reference data, the expected access volume of the data shards is determined. Determine the access granularity of the data shards based on the expected access volume of the data shards.

5. The method for constructing a multi-engine intelligent data warehouse storage model according to claim 4, characterized in that, Compare the similarity of content features between the data fragments and the reference data, including: Extract the content feature vectors from the data slices and reference data, and generate a sequence of content feature vectors according to the same sorting rules; The sequence similarity between the content feature vector sequences of the data fragments and the reference data is determined by a dynamic sequence matching algorithm, and is used as the similarity of the content features.

6. A multi-engine intelligent data warehouse storage model construction system, characterized in that, include: The architecture management module is used to build a distributed storage architecture that includes multiple storage nodes; Each storage node is configured with at least one data processing engine and is identified as a node resource for data processing. The sharding management module is used to shard the unstructured data to be stored, obtain multiple data shards, and acquire the data characteristics of each data shard. The granularity evaluation module is used to evaluate the access granularity of each data shard by comparing the similarity of data features between each data shard and historical data; the access granularity is used to characterize the input and output overhead of the data shard during access. The intelligent matching module is used to establish intelligent matching rules between data shards and storage nodes based on the access granularity of each data shard and the node resources of the storage node. The intelligent matching module is specifically used to cluster and classify multiple storage nodes based on their node resources. Based on the access granularity of the data shards and the node resources of the storage nodes within each level, the matching degree between the data shards and each level is determined, and the level with the highest matching degree is taken as the matching level of the data shard; the node resources include local computing power and network bandwidth. A hash ring structure is constructed for each storage node within a tier, and data shards are allocated to the matching storage nodes within the tier using a hash mapping algorithm; The intelligent matching module is specifically used to establish a first association between the access granularity of data shards and the average local computing power of each storage node within the hierarchy; the first association is configured such that when the access granularity increases, the degree to which the matching degree is affected by the average local computing power increases. A second correlation is established between the access granularity of the data shards and the average network bandwidth of the storage nodes within each tier; the second correlation is configured such that the degree to which the matching degree is affected by the average network bandwidth increases as the access granularity decreases. Based on the results of the first and second associations, determine the matching degree between the data shards and each level; Based on the access granularity of data sharding and the node resources of storage nodes, an access processing mechanism for the multi-engine intelligent data warehouse storage model is formulated. If the access granularity of the data shard is greater than the preset granularity threshold, and the remaining local computing power of the corresponding storage node is greater than the preset computing power, the computing task will be scheduled to the storage node so that the data processing engine can perform local computing and return the result. If the access granularity of the data shard is less than the preset granularity threshold, and the remaining network bandwidth of the corresponding storage node is greater than the preset bandwidth, the data shard will be transmitted to the cloud computing node through the bandwidth to complete the processing and return the result. The model deployment module is used to store each data shard to the corresponding storage node according to the intelligent matching rules, forming a multi-engine intelligent data warehouse storage model.

Images