A data storage retrieval method and system based on big data

Abstract

The application discloses a data storage and retrieval method and system based on big data, and particularly relates to the technical field of big data storage and retrieval. The method comprises the following steps: semantic extraction and domain storage, log analysis, hotness and correlation graph, cold and hot classification, double index construction and cross-domain pointer, multi-factor routing after query analysis, target node index retrieval and hierarchical merging, and feedback updating. The system comprises semantic domain storage, adaptive index construction, intelligent retrieval routing and hierarchical merging feedback modules. The application improves storage efficiency and retrieval performance, and realizes semantic perception, adaptive index and intelligent routing.

Description

Technical Field

[0001] This invention relates to the field of big data storage and retrieval technology, specifically to a data storage and retrieval method and system based on big data. Background Technology

[0002] With the rapid development of the Internet, the Internet of Things and mobile communication technologies, the amount of data is growing exponentially and the types of data are becoming increasingly diversified. The coexistence of structured data, semi-structured data and unstructured data in multi-source heterogeneous data streams poses a severe challenge to traditional data storage and retrieval systems. Existing big data storage and retrieval schemes usually use fixed partitioning, range partitioning or hash partitioning to divide and store data, and build global index or distributed index structure on top of it to support retrieval operations. Existing technologies still have shortcomings: At the data storage level, traditional partitioning strategies are usually based on time range, data volume or simple hash to distribute data, without fully considering the semantic relationship of the data itself. This results in semantically similar data being scattered across different nodes in physical storage, requiring queries across multiple nodes during retrieval, which increases network overhead and retrieval latency. At the index building level, existing solutions mostly use a unified index structure to manage all data, which cannot distinguish the hot and cold access characteristics of data. Hot data and cold data share the same index resources, which not only creates a bottleneck of index competition when hot data is accessed, but also maintains redundant index overhead for cold data, thus limiting the overall retrieval efficiency. At the same time, there is a lack of effective cross-domain association mechanisms between different data domains. For joint queries involving multiple domains, it is often necessary to traverse each domain, and there is a lack of guidance for optimizing the retrieval path. At the level of retrieval routing and result merging, existing systems typically make routing decisions based solely on the similarity matching between queries and data or keyword hits, failing to comprehensively consider the real-time load status of each storage node and network transmission costs, resulting in low overall system resource utilization. After obtaining multiple retrieval results, a globally unified sorting method is often used for processing, which incurs excessive computational overhead when the data scale is large. Furthermore, there is a lack of effective user feedback utilization mechanisms, making it impossible to dynamically optimize system parameters based on user retrieval behavior, resulting in insufficient adaptive capabilities. In summary, existing big data storage and retrieval methods have shortcomings in areas such as semantic awareness of data layout, hot and cold adaptation of indexes and cross-domain association, multi-objective optimization of routing, and feedback loop. There is an urgent need for a solution that can comprehensively improve storage efficiency and retrieval performance in the big data environment. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a data storage and retrieval method and system based on big data.

[0004] The objective of this invention can be achieved through the following technical solutions: This application provides a data storage and retrieval method based on big data, including the following steps: S1: Acquire multi-source heterogeneous data streams and extract semantic feature vectors and attribute tag sets. After dynamically dividing several data domains based on vector similarity and tag sets, the data is distributed and stored. At the same time, a local metadata mapping table corresponding to semantic abbreviation values ​​and physical addresses is established for each storage node. S2: Collect historical access logs to generate a heat time series and cross-domain correlation graph. Based on this, the data is classified as hot or cold. Hot data is used to build a high-frequency index tree and dynamically adjusted, while cold data is used to build a low-frequency compressed index. At the same time, cross-domain jump pointers are established based on the correlation graph. S3: Parse the user query and expand the semantics to obtain the query vector and intent label. Match the domain label index to filter candidate domains and calculate the vector distance score. Combine the candidate domain load and network cost to optimize the selection of the routing path and direct the query to the target domain and storage node. S4: Select the corresponding index for retrieval based on the hot / cold classification status of the target node, obtain the hit result set, and combine it with the additional result set obtained through cross-domain jump pointer retrieval. After accurately sorting the hit results, insert them into a merged set according to the relevance threshold. After verification and deduplication, output the result set and update the heat sequence and association weights based on user feedback.

[0005] This application provides a data storage and retrieval system based on big data, including the following modules: Semantic domain storage module; used to acquire multi-source heterogeneous data streams and extract semantic feature vectors and attribute tag sets, dynamically divide multiple data domains for distributed storage based on vector similarity and tag sets, and build a local metadata mapping table for each storage node that corresponds one-to-one with semantic abbreviation values ​​and physical addresses; The adaptive index building module, connected to the semantic domain storage module, is used to collect historical access logs to generate a hot time series and a cross-domain association graph. Based on the hot time series, it completes the hot and cold data classification, builds a dynamically adjustable high-frequency index tree for hot data, builds a low-frequency compressed index for cold data, and builds a cross-domain jump pointer based on the cross-domain association graph. The intelligent retrieval routing module, connected to the adaptive index building module, is used to parse user query statements and perform semantic expansion to generate query vectors and intent labels, match domain label indexes to filter candidate data domains and calculate weighted vector distance scores, and select the optimal routing path by comprehensively considering node load cost and network transmission cost, and then distribute the query task to the target storage node. The hierarchical merging feedback module, connected to the intelligent retrieval routing module and the adaptive index building module, retrieves the corresponding primary or secondary indexes at the target storage node based on the hot / cold classification of the target data domain to obtain the hit result set. It then combines this with additional hit results retrieved from related data domains via cross-domain jump pointers. After precise sorting of the hit results, it selects the best performers based on dynamic relevance thresholds and merges them into a merged retrieval set. After verification and deduplication, it outputs the retrieval content to the user. It also collects user click feedback data and sends it back to the adaptive index building module to update the popularity time series and cross-domain association graph weights.

[0006] Beneficial effects This invention extracts semantic feature vectors and attribute tag sets from multi-source heterogeneous data streams, dynamically divides data domains based on vector similarity and tag sets, and stores them in a distributed manner. This allows semantically similar data to be clustered in physical storage, effectively avoiding the problem of semantically related data being scattered across different nodes due to traditional fixed partitioning strategies, and significantly reducing network overhead and retrieval latency for cross-node queries. This invention generates a time series of data popularity and a cross-domain association graph by collecting historical access logs. Based on this, the data is classified into hot and cold categories, and high-frequency index trees and low-frequency compressed indexes are built respectively. This allows hot data to enjoy efficient index resources, while cold data only requires low-cost index maintenance, avoiding resource competition and redundant overhead caused by hot and cold data sharing the same index structure. At the same time, the cross-domain association graph establishes jump pointers between data domains, enabling joint queries involving multiple domains to directly jump to the associated data domains, avoiding the inefficiency of traversing domain by domain. This invention optimizes routing by comprehensively considering the relevance scores of candidate data domains, real-time load status, and network transmission costs during the query routing phase. This avoids the uneven node load caused by relying solely on similarity matching, thus improving the overall resource utilization of the system. In the result merging phase, a hierarchical merging strategy is adopted, prioritizing accurate sorting of primary results before selectively inserting secondary results based on thresholds, avoiding the high computational overhead of globally unified sorting. Simultaneously, the invention dynamically updates the popularity time series and cross-domain association graph weights based on user click feedback, forming an adaptive closed loop of retrieval-feedback-optimization. This allows the system's index structure and routing strategy to continuously evolve with user behavior. Attached Figure Description

[0007] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings.

[0008] Figure 1 A flowchart illustrating a data storage and retrieval method based on big data, provided in Embodiment 1 of this application; Figure 2 This is a schematic diagram of the structure of a data storage and retrieval system based on big data, provided in Embodiment 1 of this application. Detailed Implementation

[0009] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0010] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0011] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0012] Example 1 Please see Figure 1 This embodiment provides a data storage and retrieval method based on big data, including the following steps: S1: The system receives multi-source heterogeneous data streams from different data sources. These data streams simultaneously contain three types of data: structured data, semi-structured data, and unstructured data. Structured data includes relational database records and Excel spreadsheets; semi-structured data includes JSON documents, XML files, and log data; and unstructured data includes text reports, web page content, and email bodies.

[0013] First, the data type of the input data stream is identified based on the data format identifier or file header information. The multi-source heterogeneous data stream is then split into corresponding parsing channels according to the data type. Each channel uses a different parsing strategy to achieve semantic extraction of the data content.

[0014] For structured data, semantic extraction is performed by directly mapping field semantics. Specifically, for each field in each structured data record, the semantic vector of the field name is obtained by looking up the pre-trained embedding matrix. The field values ​​are processed according to their data types: numeric field values ​​are normalized by max-min values ​​and then expanded into vectors by a fully connected layer; categorical field values ​​are obtained by looking up the pre-trained categorical embedding matrix to obtain categorical vectors; and short text field values ​​are obtained by averaging word vectors to obtain text vectors. The field name vector and the field value vector are concatenated end to end and then weighted and summed according to the diversity weights of each field to obtain the initial feature vector of the structured data record. This method uses field names and field values ​​as directly readable sources of field semantics, and transforms symbolic field semantics into numerical vectors that can be used for subsequent calculations through encoding and weighted aggregation.

[0015] For semi-structured data, the hierarchical structure of JSON or XML is first parsed to identify key-value pairs or tag content combinations. Then, pattern recognition methods are used to extract frequently occurring key names or tag names as key tags, and the corresponding values ​​are used as tag content, thus completing the conversion of semi-structured data into structured semantic tags.

[0016] For unstructured data, plain text data is input into a pre-trained natural language processing model. This model, based on the Transformer architecture, encodes the input text into a high-dimensional semantic vector. Specifically, a pre-trained BERT semantic representation model is loaded, and the text content is segmented into several token sequences based on words. After multi-layer self-attention computation by the model, the output vector of the CLS token bits is taken as the high-dimensional semantic vector of the text data. Its dimension is 768 or 1024, depending on the specifications of the selected pre-trained model.

[0017] Since the semantics of fields directly mapped from structured data, the key labels extracted from semi-structured data through pattern recognition, and the high-dimensional semantic vectors output by NLP models from unstructured data all have significant differences in original feature dimensions, numerical units, and semantic expression methods, the initial features output from each channel are uniformly mapped to a shared semantic space, enabling data from different sources to be compared and calculated in the same semantic coordinate system.

[0018] A shared semantic mapping matrix is ​​maintained, projecting the initial feature vectors output from each channel onto a unified high-dimensional semantic space through a linear transformation. This mapping matrix is ​​a trainable parameter matrix, obtained through pre-training on a large amount of labeled data during the initialization phase. The training process of the shared semantic mapping matrix is ​​as follows: A training dataset is constructed, containing several triplet samples. Each triplet consists of one structured data record, one semi-structured data record, and one unstructured data record. The three data records within the same triplet are semantically related, such as describing the same business event or the same entity object. The structured data records are encoded in two stages to obtain the initial feature vector. The semi-structured data records are processed through pattern recognition to obtain the initial feature vector. The unstructured data records are processed through the BERT model... The output yields initial feature vectors. These three initial feature vectors are then input into a shared semantic mapping matrix for linear transformation, resulting in three corresponding transformed vectors. The training objective is to minimize the pairwise distances between the three transformed vectors within the same triplet, ensuring that semantically related modal data are close to each other in the transformed space, while maximizing the distance between transformed vectors from different triplets. Training uses a triplet loss function as the optimization objective, iteratively updating the parameters in the mapping matrix using a gradient descent algorithm until the loss function converges. After training, the parameters of the shared semantic mapping matrix are fixed, and a unified linear projection transformation is performed on the initial feature vectors of all data channels. All channels share the same mapping matrix, ensuring that data from different sources are mapped to the same semantic space and are comparable.

[0019] The high-dimensional semantic vector after mapping is reduced in dimensionality. Principal component analysis is used to project the original high-dimensional vector to a low-dimensional semantic space. The top d principal components with a cumulative variance contribution rate of a preset percentage are retained to generate a standard-dimensional semantic feature vector. In this embodiment, d is 128 dimensions and the cumulative variance contribution rate threshold is 95%.

[0020] While extracting semantic feature vectors, an attribute tag set is constructed for each data record based on data source, timestamp, business type, and entity relationship information. This attribute tag set, together with the semantic feature vector, participates in the subsequent data domain attribution determination.

[0021] The system identifies the source system of the recorded data and encodes it as a source tag. Source system identifiers include CRM system identifiers, log collection system identifiers, and user feedback platform identifiers. The generation or collection time of the data is extracted and formatted as a standard timestamp tag. Further time granularity tags such as year, month, date, and hour are extracted from this data. These time granularity tags are used for time dimension aggregation in subsequent popularity analysis. The data is categorized and labeled according to its business domain or application scenario. Business types include sales data, operation and maintenance logs, customer feedback, and product information. Business type tags can be based on keyword rule matching or automatically inferred by a classification model. The system identifies the entity objects involved in the data and their relationships, extracting them as entity relationship tags. Entity objects include user identifiers, product identifiers, order numbers, and project numbers. Relationships include relationships between users purchasing products and relationships between employees belonging to departments. Finally, each data record is represented by a binary tuple, where the first element is a semantic feature vector and the second element is a set of attribute tags.

[0022] Each data domain maintains a domain feature centroid (mean vector), a label index table (inverted frequency index), and a record list, which stores the globally unique identifier, actual data content, and corresponding semantic feature vector of all data records in the domain. After generating a universally unique identifier (version 6) for each data record, the following domain entry determination is performed.

[0023] Calculate the semantic vector distance between the semantic feature vector of the new data record and the centroid of the domain features of each existing data domain. In the process of vector distance calculation, cosine similarity is used to measure the degree of semantic similarity between the two. The cosine similarity is calculated by dividing the dot product of the semantic feature vector of the new data record and the centroid vector of the domain features by the product of their magnitudes. The cosine similarity value ranges from negative one to positive one. The closer the value is to positive one, the more similar the semantics are.

[0024] The tag overlap rate is compared between the attribute tag set of the new data record and the tag index table of each domain. The tag overlap rate is calculated by dividing the number of tags in the attribute tag set of the new data record that appear in the domain tag index table by the total number of tags in the attribute tag set of the data record. The value ranges from zero to one. The higher the value, the higher the degree of attribute tag matching.

[0025] The semantic vector distance and label overlap rate are weighted and fused to generate a comprehensive attribution score for each existing data domain for the new data record. The formula for calculating the comprehensive attribution score is as follows:

[0026] in The cosine similarity between the two is represented by a value in the range [-1, 1]. After mapping, it falls into the [0,1] interval, which is consistent with the range of the label overlap rate; The preset fusion weight coefficient is α, which is 0.65 in this embodiment; This indicates that the new data record is related to the first... The overall attribution score of each data domain. The semantic feature vector representing the new data record. Indicates the first The centroid of the domain features of a data domain Indicates the cosine similarity between the two. The attribute label set representing the new data record. Indicates the first A domain label index table for each data field. Indicates the label overlap rate. The preset fusion weighting coefficients; fusion weighting coefficients The value ranges from zero to one, meaning that semantic similarity has a high weight in the attribution determination, reflecting that semantic association is a more fundamental attribute feature than label matching; the comprehensive attribution score also ranges from zero to one, and the closer it is to one, the higher the attribution fit between the data record and the corresponding data domain.

[0027] A preset entry threshold is set. In this embodiment, the entry threshold is set to 0.6, and the entry determination is based on the highest comprehensive affiliation score.

[0028] When the highest comprehensive attribution score is greater than or equal to the preset entry threshold, the new data record is assigned to the data domain corresponding to the highest comprehensive attribution score, and the domain feature centroid and label index table of that data domain are updated. The domain feature centroid is updated by multiplying the original centroid vector by the number of data records in the original domain plus the semantic feature vector of the new data record, and then dividing by the number of data records in the original domain plus one. The label index table is updated by adding each label from the attribute label set of the new data record to the index table and incrementing the corresponding frequency count by one.

[0029] When the overall affiliation score of all existing data domains is lower than the preset domain entry threshold, it indicates that the data record does not match any existing data domain in terms of semantics and label features. A new data domain is created, and the data record is used as the first data in the domain. The domain feature centroid is set to the semantic feature vector of the data record, and the label index table is initialized to the labels and frequencies in the attribute label set of the data record.

[0030] To prevent excessive data volume in a single data field from causing a decrease in retrieval performance, a capacity limit is preset for each data field. The value of the capacity limit is determined comprehensively based on the disk capacity of the storage node, the memory cache space, and the index building efficiency. When the number of stored records in a data field reaches the capacity limit, a split operation within the field is triggered.

[0031] Obtain the semantic feature vector set of all data records in the domain, and perform binary clustering on this set using the K-Means clustering algorithm. Set the number of clusters to two, and after iterative convergence, obtain two sub-clusters and their corresponding sub-cluster centroids. Split the original data domain into two sub-data domains. The first sub-data domain contains all data records belonging to the first sub-cluster, and the domain feature centroid is set as the centroid of the first sub-cluster. The label index table is reconstructed from the attribute labels of all data records in this sub-data domain. The second sub-data domain is processed in the same way. Delete the original data domain and add the two sub-data domains to the data domain management list to participate in subsequent domain entry determination and retrieval routing. Preferably, during domain splitting, the multi-table cascaded LSH index of the original data domain, including all random projection matrices and hash buckets, is inherited by both the first and second subdomains as the initial semantic recall index for the two subdomains. After inheritance, the system independently collects new data for each of the two subdomains and updates their respective hash buckets. The system presets an LSH retraining threshold, which can be set to 150% of the original data volume at the time of the split. This value can be flexibly adjusted according to the actual system load and retrieval accuracy requirements. When the data volume of any subdomain exceeds this threshold, the system asynchronously triggers global retraining of the LSH projection matrix of that subdomain in the background. During retraining, the old matrix is ​​still used to provide retrieval services. After retraining is completed, the old matrix is ​​replaced by atomic pointer switching, and the old hash bucket is gradually migrated to the new hash bucket. After the migration is completed, the memory of the old matrix is ​​released. The asynchronous retraining does not block front-end read and write requests.

[0032] In the above intra-domain splitting operation, when the original data domain is deleted and split into the first subdomain and the second subdomain, in order to ensure the continuity and effectiveness of the access popularity time series and cross-domain association map constructed in the subsequent S2 step, the system synchronously performs the following metadata inheritance and migration operations: The access popularity time series of the original data domain is completely copied to the first and second subdomains as the initial access popularity time series of the two subdomains. Thereafter, the system independently collects new access logs for the first and second subdomains and updates their respective popularity values. The popularity values ​​of the two subdomains gradually differentiate as subsequent access differences occur. In the cross-domain association graph, the node corresponding to the original data domain is deleted, and two new nodes corresponding to the first subdomain and the second subdomain are created respectively. For each edge between the original data domain and any other data domain in the cross-domain association graph, the weight of the edge is weighted according to the proportion of the number of data records contained in the first subdomain and the second subdomain to the total number of records in the original data domain, and assigned to the new edges corresponding to the first subdomain node and the second subdomain node respectively. If the original data domain in the cross-domain association graph has a self-loop edge pointing to itself or a jump pointer between other data domains, the jump pointer corresponding to the edge is cleared in the graph, and the cross-domain jump path during retrieval is re-established based on the actual storage nodes of the two subdomains.

[0033] Simultaneously, the system generates a new data domain version number and broadcasts it to all storage nodes. After receiving the broadcast, each storage node traverses the list of jump pointers in its local storage and marks all jump pointers pointing to the original data domain as invalid. If an invalid pointer is detected during subsequent retrieval jumps, the intelligent retrieval routing module recalculates the target path and updates the pointer. If a query request that arrives before the split is completed carries the original data domain identifier, the routing decision module automatically redirects it to the corresponding subdomain. If the original data domain is currently a hot data domain, the first and second subdomains will initially inherit the hot data state after creation, and each will independently build a high-frequency index tree based on its own data records. If the original data domain is currently a cold or warm data domain, the two subdomains will initially inherit the cold or warm classification state of the original data domain. Subsequently, S2 will re-determine the hot and cold classification based on the access frequency of each of the two subdomains and dynamically adjust their respective index structures.

[0034] After completing the distributed storage of data records in each data domain, a local metadata mapping table is constructed for each storage node. This local metadata mapping table stores the correspondence between the semantic feature abbreviation values ​​and physical addresses of all data records in the node in the form of key-value pairs.

[0035] The key generation method is as follows: Local Sensitive Hash (LSH) is used to reduce the dimensionality of the semantic feature vector of each data record. The LSH algorithm maps the high-dimensional vector into a fixed-length binary hash code through random projection, so that the original vectors with similar semantics have a higher probability of colliding in the hash space, while the vectors with large semantic differences have a lower probability of colliding. Locality-sensitive hashing is implemented using a variant of the SimHash algorithm. The core principle of the SimHash algorithm is to project vectors in a high-dimensional space onto a low-dimensional binary space using a random hyperplane, so that vectors with smaller angles in the original space have more identical binary bits after projection, thereby achieving hash collisions of semantically similar data.

[0036] The random projection matrix is ​​generated as follows: During the initialization phase, the system generates a random projection matrix once. The number of rows in this matrix is ​​equal to the dimension of the input semantic feature vector, and the number of columns is equal to the number of bits in the target hash code. In this embodiment, the number of bits in the target hash code is 64 bits. Each element in the projection matrix is ​​independently sampled randomly from a standard normal distribution. The sampling process uses a fixed random number seed to ensure that the same projection matrix is ​​generated each time the system is initialized, ensuring that the same data is hashed at different times and that a consistent hash code is obtained. After the projection matrix is ​​generated, it is stored in the system memory. The same projection matrix is ​​used for hash mapping of all data records.

[0037] The encoding rule for the hash code is as follows: For any input semantic feature vector, first perform matrix multiplication with the above projection matrix to obtain a 64-dimensional real vector, where each dimension corresponds to a hash bit; then, perform a threshold determination on each dimension of the real vector. If the value of the dimension is greater than or equal to 0, the corresponding hash bit is encoded as 1; if the value of the dimension is less than 0, the corresponding hash bit is encoded as 0; arrange all 64 determination results in order to form a 64-bit binary hash code, which is stored as a zero-one string in the key field of the local metadata mapping table.

[0038] The meaning of a 64-bit binary hash code is: each bit represents the symbol information of a semantic feature vector in a random projection direction. The number of bits that differ in value between two hash codes is called the Hamming distance. The smaller the Hamming distance, the higher the symbol consistency of the two original semantic feature vectors in each projection direction, the smaller the angle between the corresponding vectors, and the higher the semantic similarity.

[0039] To further improve the recall rate of nearest neighbor retrieval, the system adopts a multi-table concatenation strategy, which maintains multiple independent random projection matrices, each generating a set of 64-bit hash codes independently. In this embodiment, four independent projection matrices are maintained, each generated using a different random number seed, and the corresponding hash codes are stored in four independent mapping tables. During a query, the system passes the query semantic feature vector through the four projection matrices to obtain four sets of hash codes, searches for candidate data records in the four mapping tables, merges all found candidate records, sorts them in ascending order of Hamming distance, and takes the top few records with the smallest distance as the final matching result. The multi-table concatenation strategy effectively reduces the probability of missing similar vectors due to a single projection matrix by increasing the diversity of independent projection directions.

[0040] The SimHash algorithm and its multi-table concatenation scheme described above complete the generation and storage of all projection matrices during the system initialization phase, and the projection matrices remain fixed during operation. Locality-sensitive hash mapping is based entirely on linear operations of fixed projection matrices, which is fast and the results are repeatable, making it suitable for real-time retrieval needs in large-scale data scenarios. In this embodiment, a 64-bit hash code is generated as a semantic abbreviation value, which significantly reduces storage space while maintaining a high probability of hash collisions for semantically similar data.

[0041] The value is generated by recording the physical storage address of the data record in the storage node. The physical storage address includes the disk block number, offset, or object storage key name, and the specific form depends on the implementation of the underlying storage engine.

[0042] This local metadata mapping table is stored in the local memory or cache of each storage node. When performing intra-node retrieval, the hash code is obtained by performing local sensitive hash calculation on the query semantic vector. Then, the physical address of the candidate data record is quickly matched and located in this mapping table. There is no need to traverse the full data or access the global metadata center, which significantly reduces the metadata query latency during the retrieval process.

[0043] S2: After the dynamic partitioning and distributed storage of the data domain are completed in step S1, each storage node continuously records relevant information for each data access operation, generating a historical access log. The historical access log includes the following fields: the unique identifier of the accessed data record, the domain identifier of the data domain to which the data record belongs, the timestamp of the access, the access type, and the user identifier or query identifier that performed the access. Each storage node periodically reports the locally generated historical access log to the log aggregation node, which then organizes them in chronological order to form a global historical access log database.

[0044] For each data domain, access records of all data records within that domain within a preset time window are extracted from the global historical access log database. The length of the preset time window is set according to the data scale and application scenario; in this embodiment, it is the most recent thirty calendar days. The extracted access records are aggregated according to a preset time granularity, and the total number of times the data domain is accessed within each time granularity is counted. The setting of the time granularity depends on the system's sensitivity to changes in popularity; in this embodiment, the time granularity is one hour, resulting in twenty-four time points per day.

[0045] Arrange the access counts at each time point in chronological order to form the access popularity time series for that data field; for the j-th data field, its access popularity time series is represented as follows: , ..., ,in This represents the total length of the time series, i.e., the total number of time points contained within the preset time window. Indicates the first The number of times this data field was accessed within a given time point.

[0046] For newly created data domains, when there are not enough historical access logs in the early stages of their creation, the system assigns a default initial popularity time series to the data domain. The value of each time point in the initial series is a preset baseline access frequency, which is determined based on the historical average access level of similar data domains.

[0047] While generating the access popularity time series of each data domain, joint query records are extracted from the global historical access log library; a joint query record refers to a data access operation involving multiple data domains in a single user query request; for each joint query record, all data domain identifiers hit by the query are parsed to form the data domain set corresponding to the query.

[0048] The co-occurrence frequency of any two data domains being simultaneously hit in the same query is calculated. Specifically, all joint query records are traversed, and for the set of data domains corresponding to each record, the co-occurrence count of all possible pairwise combinations in that set is incremented by one. In the above statistical process, the definition of the same query is: an independent search request initiated by the user is used as the basic statistical unit. Specifically, each time a user submits a query statement and receives a search result set returned by the system, it is counted as an independent query event. Two different query requests submitted by the same user within a consecutive time period, even if the time interval between submissions is very short, are counted as two independent query events respectively. This definition ensures that the co-occurrence relationship counted each time comes from the user's real joint demand for data from multiple data domains within a single search intent, avoiding the conflation of multiple queries under different search intents.

[0049] The method for identifying simultaneous hits is as follows: For any independent query event, during the process of completing the retrieval and generating the returned result set, the system records all data field identifiers hit by the query; specifically, the system parses each data record in the retrieval result set returned by the query, extracts the data field identifier to which the data record belongs, and forms the set of hit data fields for the query after deduplicating all extracted data field identifiers; any two data fields in this set of hit data fields are identified as being simultaneously hit in the query, and the corresponding co-occurrence frequency count is incremented by one.

[0050] At the log recording level, when executing the retrieval process for each query, the system immediately records the globally unique identifier of this query and the corresponding set of hit data fields after the final result set is generated. This log entry is written to the cross-domain query log database in a structured format. Subsequent co-occurrence frequency statistics directly use the set of hit data fields recorded in this log database as the data source, without relying on repeated re-parsing of the original query statement, ensuring the consistency of statistical caliber and the reproducibility of statistical results. After traversal, the co-occurrence frequency statistics between any two data fields are obtained.

[0051] Using all data domains as the set of nodes, and the co-occurrence frequency between any two data domains as the edge weight connecting these two nodes, an undirected weighted graph is constructed. This graph is the cross-domain association graph. The nodes of the cross-domain association graph correspond one-to-one with the data domains, and the edge weights reflect the degree to which the two data domains are simultaneously requested by users in historical queries.

[0052] For each data domain, a comprehensive popularity score is calculated based on its access popularity time series for the current moment. The comprehensive popularity score measures the access activity level of the data domain at the current moment, and its calculation takes into account both recent access frequency and the decay effect of historical access.

[0053] The recent access frequency is the sum or average of the access counts at the most recent time points in the access popularity time series, specifically the sum of the access counts at the most recent six time points; the access decay factor assigns a lower weight to access records that are further away from the current time, in order to reflect the timeliness characteristics of the access pattern.

[0054] The formula for calculating the overall popularity score is as follows: To avoid dimensional conflicts and eliminate boundary ambiguities, the comprehensive popularity score adopts a sliding window mechanism with the same length as the threshold statistical window; let the index of the time point corresponding to the current moment be... Take the nearest one Access data at various time points, in this embodiment =30, which is consistent with the sliding window length of the historical average, then the first... The formula for calculating the overall popularity score for each data domain is as follows:

[0055] in, Indicates the time elapsed since the current time The number of times this data field is accessed at each time granularity. Set a fixed length for the sliding window. The preset access attenuation factor is 0.05 in this embodiment. The base of the natural logarithm is denominator. This is a normalization factor used to eliminate the dimensional amplification effect caused by the summation of weights; the corrected... Represents the time before the current moment The weighted average access frequency within a time granularity, with dimensions of access count and time granularity, consistent with the dimensions of the historical average; access decay factor. Control the rate at which historical access weight decays over time. The larger the value, the lower the weight assigned to access records that are further removed from the current time; in this embodiment... Take 0.05, so that =30, meaning that the weight of visits 30 hours ago is reduced to about 22% of the current weight, which retains the influence of recent visit trends and eliminates the dimensional interference caused by long-term historical totals on the score.

[0056] Overall popularity rating The larger the value, the higher the access activity of the data domain at the current moment; this score will serve as the core basis for subsequent hot and cold classification; the hot threshold is equal to 150% of the historical average, and the cold threshold is equal to 50% of the historical average.

[0057] The historical average is calculated using a sliding window approach, with the window length set at the most recent 30 time points, each representing one hour. This represents the average access frequency of the data domain over the past 30 hours. The system maintains a circular queue of length 30 for each data domain, storing the access counts of the most recent 30 time points in chronological order. The historical average at the current moment is the arithmetic mean of all 30 values ​​in the queue. The sliding window moves forward hourly over time, discarding the oldest value in the queue and adding the access count of the latest time point to the end of the queue each time it moves, ensuring that the historical average always reflects the access level over the past 30 hours. Through this sliding window mechanism, the historical average of each data domain can adapt to changes in access patterns at different times of the day, avoiding the use of all historical data which would result in an overly smooth average that fails to respond to real-time access fluctuations.

[0058] Before a new data field accumulates 30 hours of records, it is marked as warm data, temporarily stored in a buffer, and no index is built.

[0059] The hot and cold thresholds are not set once and then fixed, but are continuously and dynamically updated according to the access popularity of each data domain. The hot and cold thresholds for each data domain are recalculated every 6 hours, that is, 4 times a day. Each recalculation uses a fixed length within the sliding window. =30. The historical average number of accesses over the last 30 time points is recalculated, and then the new historical average is multiplied by the corresponding percentage to obtain the new hot threshold and cold threshold. When the overall access popularity of a data domain increases, its historical average value also increases, and the hot threshold and cold threshold increase simultaneously. Some data domains that were originally hot data are downgraded to warm data or cold data because they have not reached the new hot threshold, so that the first-level index area always stores the data with the highest access popularity. Conversely, when the overall access popularity decreases, the threshold decreases simultaneously, and some cold data can re-enter the hot data category and obtain index building resources. The dynamic update of hot and cold thresholds and the dynamic splitting and merging of high-frequency index trees together constitute a complete closed loop of adaptive index building in S2, ensuring that the allocation of index resources always matches the current access pattern.

[0060] Regarding the first Each data field is used to determine its current overall popularity score. Compare with both the hot threshold and the cold threshold, and perform the following grading judgment: when When the data is greater than or equal to the hot threshold, all data records within the data domain are marked as hot data. Hot data is used to construct a high-frequency index tree in the first-level index area, which adopts a B+ tree structure. In the B+ tree, each leaf node stores a pointer to the physical location of the data record, and non-leaf nodes store index key values ​​to accelerate the location of the retrieval path. The index key of the high-frequency index tree is a comparable ordered integer value obtained by scalar quantization of the semantic feature vector of the data record. The specific implementation of scalar quantization is as follows: each dimension of the 128-dimensional semantic feature vector is quantized separately. For each floating-point dimension, the maximum and minimum values ​​of that dimension are determined according to its range of values ​​in all data records. This range is evenly divided into 256 quantization intervals, each interval corresponding to an 8-bit unsigned integer value between 0 and 255. The floating-point value of that dimension is then determined. The integer value within a given interval is used as the quantization result for that dimension. The 8-bit quantization results for each of the 128 dimensions are concatenated end-to-end according to dimension order to form a 1024-bit binary integer string, which serves as the index key for the data record. Since the quantization results for each dimension maintain the numerical order of that dimension, and the dimensions are arranged in a fixed order, a lexicographical comparison between two index keys is equivalent to a comparison of the original 128-dimensional vector in each dimension. Therefore, this index key possesses the property of total order comparability and can be used as an ordered index key for building a B+ tree and for path retrieval. Each leaf node also stores the locality-sensitive hash code corresponding to the data record as an auxiliary matching key. In addition to storing a data pointer, each leaf node also maintains an access counter, recording the number of times the leaf node is matched.

[0061] when When the data is less than or equal to the cold threshold, all data records in the data field are marked as cold data. Cold data are used to build a low-frequency compressed index in the secondary index area. The low-frequency compressed index uses a combination of bitmap index and Bloom filter. The bitmap index maps each attribute label to a bitmap. Each bit in the bitmap corresponds to a data record. If the record has the label, the corresponding bit is set to 1, otherwise it is set to 0. A Bloom filter is used to quickly determine whether a query is likely to have a matching result in the current data domain. The length of the Bloom filter's bit array is determined based on the total amount of data expected to be stored in the cold data domain and the preset false positive rate. In this embodiment, the preset false positive rate is 1%, and the expected total amount of data is one million records. Accordingly, the bit array length is set to ten million bits, and the number of hash functions is seven. The hash function adopts the MurmurHash3 algorithm, which calculates seven independent hash positions for the same input using seven different seed values. When a new data record is added to the cold data domain, the attribute tag of the record is concatenated into a string and input into the seven hash functions, and the corresponding seven bit positions in the bit array are set to 1. When a data record is deleted from the cold data domain, the Bloom filter is not updated. It is only further confirmed by bitmap indexing when the Bloom filter determines that it may exist. This avoids invalid scanning of irrelevant cold data.

[0062] when When the data falls between the cold and hot thresholds, it is marked as warm data and temporarily stored in a buffer. Data in the buffer is not indexed and is only arranged in order of arrival time. The system periodically recalculates the comprehensive heat score of each data field in the buffer. When its score deviates into the cold or hot range, it is removed from the buffer and indexed according to the corresponding cold and hot classification strategy.

[0063] As the high-frequency index tree continuously receives newly inserted data, the load on its leaf nodes gradually increases. The load is defined as the number of index entries stored in the leaf node, with each index entry corresponding to one data record. A split threshold is pre-set for each leaf node of the high-frequency index tree. The value of the split threshold is determined based on the storage page size and retrieval performance requirements. In this embodiment, the split threshold is set to allow each leaf node to store a maximum of 256 index entries.

[0064] Each time new data is inserted into the high-frequency index tree, it is checked whether the current load of the leaf node to which the data falls exceeds the splitting threshold. If it does, the splitting operation of the leaf node is triggered: all index entries in the leaf node are sorted according to their index key values ​​and divided into two groups. The first group retains the original leaf node position, and the second group is assigned to the newly created leaf node. The index key range in the parent node is updated, and the new leaf node is linked to the corresponding pointer position of the parent node. The splitting operation is passed up the tree. If the parent node also exceeds its capacity, the splitting continues upward until the root node. If the root node splits, the height of the tree increases by one level.

[0065] When deleting data from the high-frequency index tree, it checks whether the fill rate of the leaf node corresponding to the data after the deletion operation is lower than a preset merging threshold. The fill rate is calculated by dividing the current number of index entries by the splitting threshold. The merging threshold is set to prevent excessively sparse leaf nodes from wasting storage space and reducing retrieval efficiency. In this embodiment, the merging threshold is 40%. If the fill rate of a leaf node is lower than the merging threshold, it attempts to merge the leaf node with its adjacent sibling leaf nodes. If the total number of index entries after merging does not exceed the splitting threshold, the merging operation is performed, the merged leaf node is released, and the index key range of the parent node is updated. If the total number after merging still exceeds the splitting threshold, a redistribution operation is performed to evenly distribute the index entries of the two nodes.

[0066] After the cross-domain association graph is constructed, all edges in the graph are traversed to obtain the weight of each edge, i.e., the co-occurrence frequency between the two data domains. A jump threshold is preset, which is set based on the statistical distribution of the co-occurrence frequency. In this embodiment, the jump threshold is the average value of all edge weights plus one standard deviation, ensuring that only node pairs with a co-occurrence relationship significantly higher than the average level can establish jump pointers, thus avoiding excessive jump pointers that increase maintenance overhead.

[0067] For nodes whose edge weights exceed the jump threshold, jump pointers are established: bidirectional pointers are established between hot data domains; unidirectional pointers are established between hot data domains pointing to cold data domains or between hot data domains pointing to warm data domains; no pointers are established between cold data domains or between cold and warm data domains; when a cold or warm data domain is used as the current target domain and the hit result is lower than the expected threshold, the system rolls back to the routing decision stage, selects the candidate data domain with the second highest relevance from the routing decision set to reissue the query, or traverses the global data domains to perform the retrieval, and cold and warm data domains are no longer excluded from the jump system; the bidirectional cross-domain jump pointer is stored as follows: a pointer to the root node of the high-frequency index tree of data domain B is stored in the root node of the high-frequency index tree of data domain A, and a pointer to the root node of the high-frequency index tree of data domain A is stored in the root node of the high-frequency index tree of data domain B.

[0068] In addition to storing the address of the target retrieval entry point, the jump pointer data structure also records the pointer type identifier, the corresponding edge weight value, and the timestamp of the most recent use. The pointer type identifier is used to distinguish between three types of targets: high-frequency index tree root node, low-frequency compressed index root node, and warm data buffer entry point. When performing cross-domain jumps, the system selects the corresponding retrieval method based on the type identifier. If the target is a high-frequency index tree root node, an index tree retrieval is performed; if the target is a low-frequency compressed index root node, a combined bitmap index and Bloom filter retrieval is performed; and if the target is a warm data buffer entry point, a sequential scan retrieval is performed. The pointer weight and the timestamp of the most recent use are used for subsequent jump priority sorting and pointer eviction.

[0069] During the retrieval process of executing a user query, after the query request is routed to the storage node of the current data domain, the high-frequency index tree of that node performs a retrieval operation to obtain the set of hit results within the current data domain; a preset expected threshold is set, which means the minimum number of hits required to determine whether the retrieval results of the current data domain are sufficient. In this embodiment, the expected threshold is set to 10.

[0070] If the number of hit results in the current data domain is lower than the expected threshold, it means that the data related to the query in the current data domain is insufficient to meet the user's needs, triggering a cross-domain jump. At this time, all cross-domain jump pointers stored in the root node of the high-frequency index tree of the current data domain are read, sorted according to the weight of each pointer, and the jump pointer with the highest weight is selected first. The user jumps directly to the root node of the high-frequency index tree of the corresponding related data domain through the pointer, and continues to perform the search in the related data domain. If the hit results in the related data domain are still insufficient, the user continues to jump along the jump pointer of the related domain until the number of hit results reaches the expected threshold or the number of related data domains traversed reaches the preset maximum jump depth. The maximum jump depth is set to 3 in this embodiment to prevent the retrieval delay from getting out of control due to excessively long jump links.

[0071] Each time a cross-domain search is performed via a jump pointer, the number of hits for that jump is accumulated. That is, the number of times the jump pointer is successfully used and generates additional hit results is used as a gain value. This gain value is added to the edge weight of the corresponding edge in the cross-domain association graph, realizing dynamic positive updates of edge weights. At the same time, all edge weights are periodically multiplied by a decay coefficient according to a preset time decay period. The decay coefficient is less than 1, so that the weight of jump pointers that have not been used for a long time gradually decreases. When the weight drops below the jump threshold, the corresponding jump pointer is marked as ready for elimination and deleted during off-peak periods. Through the above dynamic update mechanism, the cross-domain association graph and jump pointers can continuously evolve with changes in user query patterns, always maintaining a high degree of matching between the jump path and the user's actual joint query needs.

[0072] S3: To alleviate the real-time performance pressure caused by repeated semantic expansion and multi-domain operations in high-concurrency scenarios, the system is configured with a query semantic cache layer and a routing result cache layer: The semantic cache stores semantic vectors and intent tags with standardized query hash values ​​as keys. Data can be directly reused when the hash is hit, eliminating the need for knowledge base retrieval and BERT model inference; The routing cache combines semantic vector locality-sensitive hashing and data domain load snapshot identifiers to generate keys, storing data domain sorting and routing results. Queries with similar semantics and whose load fluctuations do not exceed limits can directly retrieve cached results without repeatedly calculating domain relevance and detecting load overhead; Both cache layers adopt an LRU eviction policy, and the expiration time is dynamically adjusted according to changes in data domain popularity; The system receives raw query requests from users, which can be in the form of natural language questions, keyword combinations, or structured query statements; The system first performs syntax parsing on the raw query requests to identify core words, modifiers, logical operators, and conditional constraint phrases in the query.

[0073] After the syntax parsing is completed, the query semantic expansion operation is performed. Semantic expansion is implemented based on a pre-trained semantic expansion model or a thesaurus, expanding each core word in the original query into a set of semantically related synonyms, hypernyms, and hyponyms; for example, if the original query contains the word "vehicle", the semantic expansion will expand it into a set of semantically related words such as "car", "motor vehicle", "sedan", and "truck". The semantic extension module uses WordNet as its knowledge base for hypernyms and hyponyms. WordNet is a large-scale English lexical semantic network that organizes semantic relationships between words based on word meaning. It explicitly defines the hypernyms (more general and abstract words) and hyponyms (more specific and granular words) for each word meaning. The system calls WordNet's application programming interface. For the input core vocabulary, it first obtains all the hypernyms and hyponyms corresponding to each word meaning in WordNet. The extension depth of two means that the first layer of extension obtains the direct hypernyms and direct hyponyms of the core vocabulary, and the second layer of extension continues to obtain the hypernyms of each hypernym obtained in the first layer and the hyponyms of each hyponym obtained in the first layer. All the words obtained in the two layers are included in the extension set. The acquisition of synonyms is also based on the synonym set corresponding to word meanings in WordNet. The synonym set refers to all words in WordNet that belong to the same word meaning group. These words can be semantically interchanged without changing the basic meaning of the sentence.

[0074] Building upon the WordNet knowledge base, the system further introduces the BERT mask language model as a supplementary tool for synonym expansion. The core words from the original query are substituted into a pre-defined mask template, which replaces the core words with mask markers while preserving the contextual information of the words in the original query. The constructed template is then input into the BERT model, which predicts the most likely words to appear at the mask position based on the contextual semantic information. The candidate words with the highest probability output by the model are selected as context-related synonym supplements. This supplementary method can capture words not included in WordNet but semantically similar in the current query context. For example, "car" in the context of "purchase" can be expanded to include words like "select" and "purchase".

[0075] The union of the expanded results of the WordNet knowledge base and the BERT masked language model is taken as the final semantic extended vocabulary set. During the initialization phase, the WordNet vocabulary file is pre-stored in local storage, and the system can complete the query operations of hypernyms, hyponyms and synonyms without the need for network connection during runtime. The BERT masked language model is also loaded into local memory during the system initialization phase. During the operation, the model parameters are kept fixed, and only forward inference is performed to complete the vocabulary prediction of the mask position.

[0076] For uncommon words or proper nouns not included in WordNet, the system retains the word itself without expansion, while recording the frequency of its occurrence. When the same uncommon word appears repeatedly in multiple queries, the system adds the word and other words that frequently co-occur with it as new synonym groups to the local expansion cache, thus achieving a gradual expansion of the knowledge base. The breadth of semantic expansion is controlled by a preset expansion depth parameter. In this embodiment, the expansion depth is set to 2, which means expanding to two layers: the synonyms of the original word and the synonyms of the synonyms of the synonyms.

[0077] The original query terms and semantically expanded terms are used together as input to the query representation and fed into the same large-scale pre-trained semantic representation model as in S1. This model encodes the expanded query term sequence into a fixed-dimensional vector representation to obtain the query semantic vector. The dimension of the query semantic vector is consistent with the dimension of the semantic feature vector of the data record in S1, both being 128-dimensional, to ensure that the two can perform vector distance calculation in the same semantic space.

[0078] Simultaneously, query intent tags are extracted from the results of syntax parsing and semantic expansion. These tags include data domain range indicators, which indicate the business domain or data type to which the user expects to retrieve the data, such as "sales data," "operation logs," and "customer feedback"; operation type tags, which indicate the type of operation the user expects to perform, including exact queries, fuzzy matching, range queries, and sorting queries; time constraint tags, which indicate the time range conditions specified in the query; and entity tags, which indicate the entity objects mentioned in the query. The query intent tags are represented as a set and serve as the matching basis for subsequent candidate data domain filtering.

[0079] After obtaining the query semantic vector and query intent label, a preliminary filtering operation of the candidate data domain is performed to narrow down the data domain range for subsequent routing decisions and reduce computational overhead.

[0080] The query intent tags are matched against the domain tag index tables of each data domain, and the degree of matching between the query intent tags and each domain tag index table is calculated; specifically, for the first... For each data field, the number of tags appearing in the tag index table of the query intent tag set is counted. If the number of hit tags is greater than or equal to a preset hit count threshold, the data field is included in the initial selection set of candidate data fields. In this embodiment, the hit count threshold is 50% of the total number of query intent tags.

[0081] Through the above preliminary screening, data domains that are completely irrelevant or have very low relevance to the query intent are excluded, resulting in a preliminary set of candidate data domains. The size of this preliminary set is much smaller than the size of all data domains, effectively reducing the amount of processing required for subsequent fine calculations.

[0082] For each data domain in the initial candidate data domain set, calculate the relevance score of that data domain relative to the current query. The calculation of the relevance score takes into account both semantic vector distance and the hot / cold classification status.

[0083] First, calculate the basic vector distance between the query semantic vector and the centroid of the domain features of the candidate data domain. Consistent with the cosine similarity used when data is entered into the domain in S1, the basic vector distance is measured by cosine similarity. However, it should be noted that before calculating the basic vector distance, different dimensional weights should be assigned to each dimension component of the query semantic vector according to the attention of each dimension in the query intent tag.

[0084] The dimensional weights of each component of the query semantic vector are determined as follows: A mapping relationship between intent labels and vector dimensions is pre-established. Each intent label is associated with a set of dimensional weight vectors, where each element corresponds to a dimension of the query semantic vector, with values ​​between 0 and 1. When the query intent label contains multiple labels, the average of the dimensional weight vectors corresponding to each label is taken as the final dimensional weight vector. The query semantic vector is then multiplied element-wise by the dimensional weight vectors to obtain the weighted query semantic vector. Similarly, the centroids of the domain features of the candidate data domain are also subjected to the same dimensional weighting process to obtain the weighted centroid vector. Cosine similarity is then calculated based on the weighted vectors to obtain the weighted vector distance score. .

[0085] Then, a heat weighting coefficient is introduced based on the hot / cold classification of the candidate data domain in S2; the heat weighting coefficient for hot data domains is higher than that for cold data domains; in this embodiment, if the candidate data domain is a hot data domain, the heat weighting coefficient is 1.2; if it is a cold data domain, the heat weighting coefficient is [missing value]. 0.8; If it is a temperature data domain, the heat weighting coefficient is 1.0, that is, no additional weighting is performed.

[0086] The final relevance score is determined by the product of the base vector distance score and the popularity weighting coefficient; the higher the relevance score, the more relevant the data domain is to the user's query; the formula for calculating the relevance score is as follows:

[0087] in Indicates the first Relevance scores for each candidate data domain. Indicates the first The heat-weighted coefficients of each candidate data domain This represents the dimension-weighted query semantic vector. Represents the dimensional weighted number of dimensions. The domain feature centroids of each candidate data domain This represents the cosine similarity between the two.

[0088] The candidate data domains are sorted from highest to lowest according to their relevance scores, and the top N candidate data domains are selected to enter the routing decision set. In this embodiment, N is set to ten, that is, the ten data domains with the highest relevance are selected from all candidate data domains to enter the fine routing decision stage, which ensures a full evaluation of high-quality candidate domains while controlling the computational scale of routing decisions.

[0089] For each candidate data domain in the routing decision set, obtain the following three types of evaluation metrics.

[0090] The first type of indicator is the relevance score. This metric has been calculated in S3 and reflects the degree of matching between the data domain content and the query semantics. The higher the relevance score, the more suitable it is as a target route.

[0091] The second type of metric is the load cost corresponding to the current load status. Each storage node monitors its own resource usage in real time, including CPU utilization, memory utilization, and disk I / O queue depth. The utilization of each resource is weighted and summed to obtain the comprehensive load value of the storage node. The load cost is defined as a monotonically increasing function of the comprehensive load value; the higher the load, the greater the load cost. In this embodiment, the load cost is calculated as follows: CPU utilization multiplied by 0.4, memory utilization multiplied by 0.3, and the normalized value of disk I / O queue depth multiplied by 0.3 to obtain the comprehensive load value. The comprehensive load value itself serves as the load cost.

[0092] The third type of metric is network transmission cost; network transmission cost depends on the network path length from the routing decision node to the target storage node and the current network congestion status; the round-trip latency of each storage node is measured by periodically sending network probe packets, and the currently measured round-trip latency value is used as the base value of network transmission cost; if a storage node is located in the same physical rack or the same availability zone as the routing decision node, the network transmission cost is additionally multiplied by a discount factor of 0.5.

[0093] The comprehensive routing cost value is obtained by multiplying the three evaluation indicators by their respective weighting coefficients and then summing the results. The formula for calculating the comprehensive routing cost value is as follows:

[0094] in Indicates the first The comprehensive routing value of each candidate data domain Indicates the first The relevance scores of each candidate data domain are used to convert them into cost values ​​by subtracting one from the relevance score, in order to maintain consistency with the cost directions of the other two factors. Indicates the first The load cost of the storage node where each candidate data domain is located Indicates up to the The network transmission cost of the storage node where each candidate data domain is located. This is the first weighting coefficient corresponding to the relevance score. This is the second weighting coefficient corresponding to the load cost. This is the third weighting coefficient corresponding to the network transmission cost; all three weighting coefficients are preset positive numbers. In this embodiment... Take 0.5, Take 0.3, Setting it to 0.2 reflects the dominant role of correlation in routing decisions, while also taking into full account system load balancing and network transmission efficiency.

[0095] The candidate data domain and its storage node with the lowest overall routing cost are selected as the optimal target routing path, and the query request is directed to the target data domain and its corresponding storage node. The routing rules for warm data domains include: if the storage size of the warm data domain does not exceed 100,000 records, it is allowed to be included in the routing decision; if it exceeds 100,000 records, it is marked as a large warm data domain, and other non-large warm data domains are selected first. Only when all candidate domains are large warm data domains is the one with the smallest storage size selected as the target. During retrieval, a sequential scan is performed on the buffer of the warm data domain, and the cosine similarity between each record and the query vector is calculated as a relevance score. Only records with a score of not less than 0.5 are retained. The scan stops when the number of results reaches the maximum number of records returned in a single query, which is 100 by default.

[0096] To avoid input / output bottlenecks caused by repeated sequential scans due to frequent queries on warm data domains, the system sets up an independent query result cache for warm data domains. When a sequential scan is performed on a warm data domain for the first time and a hit result is obtained, the query semantic vector hash value of that query and the corresponding hit result set are cached in the warm data cache as key-value pairs, and a cache validity period is set. During the cache validity period, when the same or highly similar queries hit the warm data domain again, the result is returned directly from the warm data cache without performing a sequential scan again. The cache validity period is dynamically adjusted according to the access frequency of the warm data domain; the higher the access frequency, the longer the cache validity period.

[0097] After the query request is sent to the target storage node via the routing decision and the retrieval is performed, the storage node returns the retrieval results and the actual response latency value to the routing decision node. The routing decision node records the expected response latency value of each candidate data field in this routing decision. The expected response latency value is the theoretical response time estimated based on the load cost and network transmission cost.

[0098] If the deviation rate Delta between the actual delay and the expected delay is greater than or equal to 0.2, then... Multiply d by 1.1; if the deviation rate is less than or equal to -0.1, then... d multiplied by 0.9.

[0099] Specifically, the delay deviation rate is defined as follows:

[0100] in The delay deviation rate, This represents the actual response delay value. This represents the expected response delay value.

[0101] like If the actual latency is greater than or equal to the preset positive adjustment threshold, it indicates that the actual latency significantly exceeds the expected latency, and the load cost weighting coefficient needs to be increased. d is multiplied by an increment coefficient greater than 1; if If the actual latency is less than or equal to the preset negative adjustment threshold, it indicates that the actual latency is significantly lower than the expected latency, and the load cost weighting coefficient needs to be reduced. d is multiplied by a reduction factor less than 1; in this embodiment, the positive adjustment threshold is 0.2, the negative adjustment threshold is -0.1, the increment factor is 1.1, and the reduction factor is 0.9.

[0102] To prevent the weighting coefficients from diverging to extreme values ​​during repeated adjustments and causing routing decisions to fail, the system sets upper and lower boundary constraints on the value range of each weighting coefficient; the weighting coefficient corresponding to the load cost... The value of d must not be lower than 0.1 and not higher than 0.6; the weighting coefficient corresponding to the relevance score. The value must not be lower than 0.3 and not higher than 0.8; the weighting coefficient corresponding to the network transmission cost. The value of must not be lower than 0.1 and not higher than 0.4. After each multiplicative adjustment is completed according to the incremental or decremental coefficient, the system immediately checks whether the current value of each weight coefficient falls within the above constraints. If it is lower than the lower bound, the weight coefficient is truncated to the lower bound value. If it is higher than the upper bound, it is truncated to the upper bound value. Through the above boundary pruning, each weight coefficient is limited to a reasonable range in any adjustment cycle, which fundamentally avoids the risk of divergence where the weight coefficient tends to infinity or zero.

[0103] After completing the independent boundary pruning of each weight coefficient, the system performs normalization on the three weight coefficients to ensure that the sum of the three is always equal to 1. The specific operation of the normalization process is as follows: first, the three weight coefficients after boundary pruning are added together to obtain the total; then, each weight coefficient is divided by the above sum to obtain the final normalized weight coefficient. After normalization, the relative proportion of the three weight coefficients is preserved, and the sum of the three is exactly equal to 1, so that the comprehensive routing cost value always maintains a consistent dimension and comparability, avoiding the problem of distortion of the routing cost calculation result when the total weight coefficient deviates. Normalization and boundary pruning together constitute a complete security constraint mechanism for weight coefficient adjustment. Boundary pruning prevents individual weight coefficients from going out of control, while normalization ensures the stability of the overall weight coefficient structure and the effectiveness of the routing cost function.

[0104] Boundary pruning and normalization are strictly performed in the order of pruning first and then normalization in each weight coefficient adjustment cycle. That is, each weight coefficient is first truncated independently by upper and lower bounds, and then the three truncated weight coefficients are normalized to ensure that the weight coefficients that finally participate in the routing decision satisfy both the single coefficient boundary constraint and the constant sum condition.

[0105] The dynamic adjustment of the weight coefficients is performed in units of sliding time windows. In this embodiment, the length of the sliding time window is 100 query requests. Within a sliding time window, the delay deviation rate of all completed queries is counted, and the average value is used as the basis for weight adjustment of that window.

[0106] The weight adjustment adopts an asynchronous non-blocking method: after each query, the deviation rate is written to a lock-free circular buffer with a capacity of 2048. A background independent thread reads all deviation rates in the buffer in batches every ten seconds and calculates the average value. Based on this, multiplicative adjustment, boundary pruning and normalization are performed, and the adjustment does not block the main query path.

[0107] Through the above feedback adjustment mechanism, the load cost weight coefficient can be adaptively optimized as the system operating status changes, so that the multi-objective routing decision always maintains the best overall performance.

[0108] S4: Hot data fields only initiate searches in the first-level index area, while cold data fields only initiate searches in the second-level index area; In S4, the retrieval results from the first-level index area, i.e., the high-frequency index tree, are collectively referred to as the first-level hit result set, and the retrieval results from the second-level index area, i.e., the low-frequency compressed index, are collectively referred to as the second-level hit result set. When the target data domain is a hot data domain, the result obtained from the first-level index area is a hit result set, and the second-level index area does not produce any results in this round of retrieval. When the target data domain is a cold data domain, the result obtained from the second-level index area is the second-level hit result set. The hit result set is represented as a hit result set in the hot data domain scenario.

[0109] When searching the warm data domain, the cosine similarity is calculated for each record in the buffer as a relevance score. Only records with a score of not less than 0.55 are retained and sorted in descending order of score. The first N records are taken as the result set, where N is the maximum number of records returned in a single query preset by the system, with a default value of 100.

[0110] Meanwhile, the system will uniformly include all additional hit records retrieved from the associated data domain through the cross-domain jump pointer described in S2 during this query into the auxiliary result set for subsequent hierarchical merging processing; the additional hit records obtained from the cross-domain jump and the hit records obtained from the warm data domain scan will be uniformly included into the auxiliary result set.

[0111] For different target data domain types, the following hierarchical merging strategy is executed: When the target data domain is a hot data domain: all data records in the first-level hit result set are precisely sorted according to their relevance scores to the query semantic vector to generate a first sort result, and a relevance threshold is set based on the relevance score of the last data record in the first sort result; then, the auxiliary result set is traversed, including each data record in the additional hit records retrieved from the associated data domain through cross-domain jump pointers, and their relevance scores are calculated respectively. Only data records with scores higher than the relevance threshold are retained, and they are inserted one by one into the corresponding sorting position of the first sort result in descending order of their respective scores; after the insertion is completed, a merged sort result set is formed; When the target data domain is a cold data domain: directly use the secondary hit result set as the basic sorting set, sort all its records from high to low according to the relevance score, and then insert the records in the auxiliary result set whose scores are higher than the last score of the basic sorting set into the corresponding positions to form a merged sorting result set.

[0112] When the target data domain is a warm data domain: the hit result set obtained by scanning directly in the truncation order is sorted from high to low according to the relevance score, and used as the merged sort result set. The total number of outputs is limited by the maximum number of returns preset by the system.

[0113] First, a consistency check is performed to verify whether each data record in the merge sort result set still exists in the corresponding storage node. For data records that have been deleted or migrated, they are removed from the merge sort result set, and an exception log is recorded for subsequent auditing.

[0114] Then, deduplication is performed to check if there are duplicate data records in the merged sort result set. Duplicate data may come from the same data record existing in both the first-level index area and the second-level index area, or the same record being indexed multiple times due to index update delays. The deduplication process uses the unique identifier of the data record as the key to build a temporary hash set, traverses the merged sort result set, and only retains the entry when each unique identifier appears for the first time, and deletes subsequent duplicate entries.

[0115] After completing consistency verification and deduplication, the final search results are output to the user interface in descending order of relevance score.

[0116] After the final search results are presented to the user, the user's click behavior on the search results is collected as a feedback signal. The scope of user click feedback collection is the data records in the final search results that the user actually clicked to view details. The collected information includes the unique identifier of the clicked data record, the data domain identifier to which the clicked data record belongs, the query semantic vector corresponding to the click, and the position of the data record in the merged and sorted result set.

[0117] For each user click event, the data domain to which the clicked data record belongs and the query semantic vector at the time of the click are statistically analyzed; the positive feedback difference between the relevance score of the clicked data record and the query semantic vector is used as the pulse update amount; the pulse update amount is calculated as follows:

[0118] in Indicates the pulse update amount. This indicates the relevance score of the clicked data record during sorting. This indicates the baseline relevance threshold for this query, which is the score of the last position in the first ranking result. For a sign function, when The value is 1 when it is positive, -1 when it is negative, and 0 when it is zero. Since the clicked data record must exist in the merged sort result set, its relevance score must be higher than the relevance threshold. Therefore, the difference is always positive and the pulse update amount is positive.

[0119] A positive pulse increment is applied to the edge weights corresponding to all routing links traversed by the query in the cross-domain association graph. Specifically, the pulse update amount is accumulated to the edge weights corresponding to all routing links traversed by the query. The pulse increment is not accumulated to the access popularity time series. The update of the popularity time series is based solely on objective access records in historical access logs to avoid the user's subjective click behavior and objective access records being counted in the same data domain, which would mean that the data domain has received a positive reinforcement due to the user's click behavior. If data records from multiple different data domains are clicked in the same query, the popularity value of each corresponding data domain receives its own corresponding positive pulse increment.

[0120] At the same time, the same positive pulse increment is applied to the edge weights corresponding to all routing links traversed by this query in the cross-domain association graph; the routing link refers to the path between each data domain accessed sequentially from the initial data domain through the jump pointer; the same pulse update amount is added to the edge weights corresponding to each pair of adjacent data domains on these paths, which strengthens the association strength between data domains in the joint query scenario.

[0121] The popularity value and edge weight of unclicked data fields are synchronously decayed according to the decay factor. The decay factor works by multiplying the popularity value of all data fields by the decay factor at the end of the current popularity value update cycle. ,in In this embodiment, the preset value is greater than 0 and less than 1. The value is set to 0.995, allowing the popularity value to gradually decrease without additional click boosting, thus ensuring the dynamic adaptability of the access mode.

[0122] The synergistic effect of pulse updates and decay factors ensures that frequently clicked data fields and data fields frequently used in joint queries are continuously positively reinforced, while data fields that are not of interest to users gradually fade out of the active state. The feedback loop drives the system's index structure and routing strategy to continuously align with users' actual search preferences.

[0123] Example 2 Please see Figure 2 This embodiment provides a data storage and retrieval system based on big data, including four core modules: a semantic domain storage module, an adaptive index construction module, an intelligent retrieval routing module, and a hierarchical merging feedback module. The modules communicate and exchange data with each other through an asynchronous message queue and a remote procedure call (RPC) framework.

[0124] Semantic domain storage module: This module serves as the system's data access layer and is deployed at each data source access node. It is responsible for executing all semantic parsing, vectorization, and dynamic data domain management functions described in S1. The module consists of three parts: a heterogeneous data parsing engine, a semantic encoding unit, and a domain management component.

[0125] The heterogeneous data parsing engine maintains three independent data channels: the structured channel has a built-in JDBC connection pool and table parser, the semi-structured channel has a built-in JSON / XML streaming parser, and the unstructured channel loads a fine-tuned BERT-base pre-trained model; each channel processes the input data stream in parallel and passes the initial feature vector to the semantic encoding unit through a shared memory region.

[0126] The semantic coding unit implements the linear transformation of the two-stage coding strategy and the shared semantic mapping matrix. The shared semantic mapping matrix is ​​stored in the GPU memory as a 256×768 two-dimensional floating-point array. The matrix multiplication is accelerated by the CUDA kernel to output a 128-dimensional standardized semantic feature vector. The attribute tag set is stored in the form of an inverted index table in memory, with the tag name as the key and the value as a set of pointers to data record identifiers.

[0127] The domain management component maintains a global data domain registry, which is implemented using a consistent hashing ring. The domain feature centroids, label index tables, and capacity status of each data domain are stored in the distributed coordination service ZooKeeper in Protocol Buffers serialization format. When new data arrives, the domain management component traverses all data domains to calculate a comprehensive attribution score. The scoring process is accelerated in parallel using a Single Instruction Multiple Data (SIMD) instruction set. When a domain split is triggered, the domain management component calls the distributed implementation of the K-Means clustering operator, uses the AllReduce communication protocol to aggregate the centroids of each sub-cluster, and atomically updates the registry to ZooKeeper after the split. The access popularity time series of the original data domain is split and inherited into two sub-domains according to the number of records. After the split, the globally unique identifier of the original data domain is removed from the registry, and the first and second sub-domains obtain new domain identifiers. The metadata records of both sub-domains record their source parent domain identifiers as fields for operation and maintenance tracking and cross-generational association analysis.

[0128] During module operation, the local metadata mapping table maintained locally by each storage node is stored in memory-mapped file mode. The key is a 64-bit SimHash binary code, and the value is a physical address pointer. The mapping table is implemented using an open addressable hash table, and the load factor is maintained below 0.7 to ensure query hit efficiency. When adding or deleting data records, the mapping table implements lock-free updates through a double buffer mechanism to ensure read and write concurrency safety.

[0129] Adaptive Index Building Module: This module is connected to the semantic domain storage module via a Kafka message queue, subscribes to data domain change events and access log streams, and is responsible for executing the functions described in step S2: popularity calculation, hot / cold classification, index building, and cross-domain jump pointer management. Internally, the module consists of four parts: a log aggregation unit, a popularity evaluation engine, an index building subsystem, and a graph manager.

[0130] The log aggregation unit periodically pulls historical access logs from each storage node, performs time window aggregation at a five-minute granularity, and writes the aggregation results into the time series database InfluxDB. The access popularity time series of each data domain is stored in label-value format, with the label containing the domain identifier and the value being a JSON array of access counts per hour.

[0131] The heat assessment engine runs on an independent daemon thread, triggering a full scan every six hours to read access data from InfluxDB for the most recent 30 time points of each data domain. It uses a sliding window to calculate the comprehensive heat score and hot / cold threshold. The calculation process uses a streaming accumulation algorithm, which only requires a single traversal of the data to complete the weighted average, with a time complexity of O(N). When a newly created data domain accumulates less than 30 time points, the engine marks it as warm data and attaches it to a buffer linked list. This linked list is maintained with a doubly circular structure, supporting node insertion and removal with a time complexity of O(1).

[0132] When a data field is determined to be hot data, the index building subsystem allocates a first-level index area on the local disk and constructs a high-frequency index tree using a B+ tree structure. The order of the B+ tree is dynamically determined based on the storage page size, and by default, each node can hold a maximum of 256 index entries. When data is added or deleted, the index tree performs split or merge operations, and all tree structure changes are persisted through the write-ahead log (WAL) to ensure that the index state can be restored after a system restart. When data is determined to be cold data, a bitmap index and a Bloom filter are constructed in the second-level index area. The bitmap index is compressed and stored in Roaring Bitmap format, and the Bloom filter bit array is allocated in off-heap memory. The hash function uses the MurmurHash3_x64_128 algorithm and reuses 7 different seeds.

[0133] The graph manager maintains a cross-domain association graph, which is stored in Redis as an adjacency list. Nodes are data domain identifiers, and edges are co-occurrence frequency weights. The graph manager scans the adjacency list every half hour, identifies node pairs whose edge weights exceed the jump threshold, and writes a cross-domain jump pointer to the root node structure of the high-frequency index tree in the corresponding data domain via RPC calls. The jump pointer is embedded in the metadata page of the B+ tree root node after being serialized as protobuf. The pointer field contains the target domain identifier, the target retrieval entry type, and the target node RPC address. The retrieval entry type uses three predefined constant identifiers in the specific implementation; 0x01 represents the root node of the high-frequency index tree. 0x02 represents the root node of the low-frequency compressed index, and 0x03 represents the warm data buffer entry. The graph manager also listens for index structure change events. When a hot data domain is downgraded to a cold data domain, the graph manager calls the RPC interface of that domain to obtain its new retrieval entry type of 0x02, and then automatically updates the retrieval entry type in the jump pointer pointing to that domain from 0x01 to 0x02. When a warm data domain is promoted to a hot data domain, the graph manager also updates the retrieval entry type from 0x03 to 0x01. The above automatic updates prevent the jump pointer from becoming invalid due to changes in the target index structure, and maintain the routing correctness of the jump system under the dynamic changes of hot and cold classification.

[0134] Intelligent retrieval and routing module: This module serves as the system's query access layer, exposing a unified RESTful API interface to receive user query requests. It is responsible for executing the query parsing, semantic expansion, routing decision-making, and weight adaptive adjustment functions described in S3. The module consists of three parts: a query parsing engine, a routing decision-maker, and a weight adaptive adjuster.

[0135] The query parsing engine first performs syntactic analysis on the original query, constructing an Abstract Syntax Tree (AST) based on the Apache Calcite framework to extract core vocabulary and constraints. Then, it calls the semantic extension submodule, which loads the WordNet lexicon as an in-memory hash table, where the key is a word and the value is a joint structure of the synonym set, hypernym set, and hyponym set, with a default extension depth of 2. For words not covered by WordNet, the engine backtracks by calling the locally deployed BERT-MLM model for context prediction, selecting the top 5 probabilities as supplementary extensions. After extension, the extended vocabulary sequence is input into the same BERT encoder instance shared with the semantic domain storage module, resulting in a 128-dimensional query semantic vector. This vector is located in the same region as the semantic feature vector of the data record. Semantic space; simultaneously, this module integrates with the distributed key-value store Redis, with both the query semantic cache layer and the routing result cache layer deployed in the Redis cluster: the semantic cache uses the normalized hash value of the query statement as the key and the MsgPack serialization result of the query semantic vector and intent tag as the value, and the cache expiration time is dynamically adjusted according to the data domain popularity; the routing result cache uses the concatenated string of the 64-bit SimHash value of the query semantic vector and the current load snapshot identifier of each candidate data domain as the key, and the sorted routing decision result set as the value, the load snapshot identifier is updated by the system every five minutes, and the identifier is automatically incremented when the load fluctuation exceeds the preset threshold, so that the routing cache is automatically invalidated, ensuring that the routing decision is aware of real-time load changes.

[0136] After receiving the query semantic vector, the routing decision-maker first performs preliminary screening using the domain label index, reducing the size of the candidate data domain set to 10 to 20. Then, it calculates the relevance score and comprehensive routing cost for each candidate domain in parallel. The calculation process is executed concurrently using a thread pool, with each candidate domain assigned an independent task. For candidate domains marked as large warm data domains, whose storage size exceeds the searchable limit, the routing decision-maker prioritizes excluding them from the decision set, selecting the smallest domain as the target only when all candidate domains are large warm data domains. After completing the optimal path selection, the routing decision-maker obtains the RPC address of the target storage node through the service discovery component. The query request and query semantic vector are packaged into a Thrift binary protocol data packet and transmitted over the network to the target node. After receiving the request, the target storage node performs an index retrieval. After the retrieval is completed, the hit result set is returned to the routing decision-maker along the original RPC link. The module also has a built-in query result cache area. The retrieval results of recent high-frequency queries are cached in local off-heap memory according to the LRU strategy. The cache capacity is up to 1024 query entries. Each entry contains a query semantic vector hash, a result set pointer, and a last access timestamp. When a new query comes in, the cache hit is checked first. If a hit is found, the cached result is returned directly to avoid the overhead of full routing and retrieval links.

[0137] The weight adaptive adjuster operates asynchronously and non-blockingly, maintaining a lock-free circular buffer. Each time a single query is completed and the actual response latency value is collected, the latency deviation rate data for that query is written to the buffer. The background weight adjustment thread triggers an adjustment every 100 queries, reading the deviation rate data in batches from the buffer, calculating the average, and performing a multiplicative adjustment of the load cost weight coefficient. After adjustment, boundary pruning and normalization are performed sequentially. To prevent concurrency insecurity introduced by online weight adjustment, the adjuster employs a dual-buffered weight replica mechanism: the current routing decision uses the active weight replica, while the background thread adjusts the backup weight replica. After adjustment, the pointers to the two replicas are atomically swapped, ensuring that weight switching is instantaneous and does not block the main query path.

[0138] Hierarchical Merge Feedback Module: This module is connected to the intelligent retrieval routing module and the adaptive index building module, respectively. It is responsible for executing the hierarchical merge sorting, verification and deduplication, and user feedback collection and popularity update functions described in S4. The module consists of three parts: a merge engine, a verification and deduplication subsystem, and a feedback collector.

[0139] The merge engine receives the primary hit result set and auxiliary result set from the intelligent retrieval routing module. Internally, the engine implements a dual-path merge algorithm to quickly sort all records in the primary result set according to their relevance scores using the TimSort algorithm, with a time complexity of O(log n). , The number of first-level hit records is used; after sorting, the last score is taken as the dynamic relevance threshold, and then the auxiliary result set is traversed. Binary search is used to determine the insertion position of each auxiliary record, and only those with scores higher than the threshold are inserted into the merge set; the entire merging process is carried out in memory. For ultra-large result sets, an external sorting strategy is adopted to overflow intermediate results to local temporary files.

[0140] The deduplication verification subsystem starts after the merge is complete. First, it calls the existence check interface of each storage node via RPC to verify the physical existence status of each data record in the merged set. The existence check interface queries the local metadata mapping table of the node based on the unique identifier of the data record. If there is no physical address entry corresponding to the identifier in the mapping table, a non-existent status code is returned. If it exists, the current storage level tag of the record is returned. The verification engine compares whether the returned storage level tag matches the index type used in this retrieval. If an inconsistency is found, it means that the data record has undergone cold / hot level migration during the retrieval process. The record is removed from the merged set and the migration event log is recorded for index consistency auditing. Records that pass the verification enter the deduplication stage. Deduplication is implemented based on the hash set of unique identifiers. The identifier that appears for the first time in the traversal is retained, and duplicate identifiers are discarded. After deduplication is completed, the system truncates the results from high to low according to the relevance score. The upper limit of the truncation number is determined by the page size in the user request parameters, with a default of 20 records. The final result is encapsulated as a JSON response and returned to the user.

[0141] The feedback collector listens for click events on the user's end and captures user click behavior through JavaScript tracking code embedded in the search results page. Click event data is reported to the receiving endpoint of the feedback collector via asynchronous HTTP requests. The data payload includes the unique identifier of the clicked record, its domain identifier, the globally unique identifier of this query, and the sorting position of the record in the result set. The feedback collector writes the raw click events into a feedback topic in Kafka, which is processed in batches by a separate consumer thread: the consumer pulls a batch of feedback data every five minutes, calculates the pulse update amount, and then calls the heat update interface of the adaptive index building module via RPC to apply the pulse increment to the access heat time series of the corresponding data domain and the edge weights of the cross-domain association graph, while triggering the decay factor synchronous decay operation to form a complete feedback loop.

[0142] When the four modules work together, the entire process from data access to retrieval is completed under modular division of labor. The modules communicate across processes through well-defined gRPC interfaces. The interface contracts are uniformly defined in the form of Protocol Buffers files and included in version management, ensuring the feasibility of parallel development and independent deployment of multiple modules. The system as a whole adopts a containerized deployment approach, with each module packaged as an independent Docker image and managed through Kubernetes orchestration, supporting horizontal automatic scaling. The semantic domain storage module and adaptive index building module are stateful services, using PersistentVolumeClaim to persist index data, while the intelligent retrieval routing module and hierarchical merging feedback module are stateless services, and the number of instances can be increased or decreased at any time to cope with fluctuations in query load.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data storage and retrieval method based on big data, characterized in that, Includes the following steps: S1: Acquire multi-source heterogeneous data streams and extract semantic feature vectors and attribute tag sets. After dynamically dividing several data domains based on vector similarity and tag sets, the data is distributed and stored. At the same time, a local metadata mapping table corresponding to semantic abbreviation values ​​and physical addresses is established for each storage node. S2: Collect historical access logs to generate a heat time series and cross-domain correlation graph. Based on this, the data is classified as hot or cold. Hot data is used to build a high-frequency index tree and dynamically adjusted, while cold data is used to build a low-frequency compressed index. At the same time, cross-domain jump pointers are established based on the correlation graph. S3: Parse the user query and expand the semantics to obtain the query vector and intent label. Match the domain label index to filter candidate domains and calculate the vector distance score. Combine the candidate domain load and network cost to optimize the selection of the routing path and direct the query to the target domain and storage node. S4: Select the corresponding index for retrieval based on the hot / cold classification status of the target node, obtain the hit result set, and combine it with the additional result set obtained through cross-domain jump pointer retrieval. After accurately sorting the hit results, insert them into a merged set according to the relevance threshold. After verification and deduplication, output the result set and update the heat sequence and association weights based on user feedback.

2. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S1, multi-source heterogeneous data streams are distributed to corresponding parsing channels according to data type. Structured data is directly mapped to field semantics, semi-structured data is extracted for key tags through pattern recognition, and unstructured data is output as high-dimensional semantic vectors by a natural language processing model. The initial features output from each channel are uniformly mapped to a shared semantic space and dimensionality is reduced to generate standard-dimensional semantic feature vectors. At the same time, the attribute tag set is constructed based on data source, timestamp, business type, and entity relationship information.

3. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S1, the vector distance between the semantic feature vector of the newly arrived data record and the domain feature centroid of each existing data domain, and the label overlap rate between the attribute label set and the label index table of each domain, are weighted and fused to generate a comprehensive attribution score. When the highest comprehensive attribution score exceeds the preset domain entry threshold, it is assigned to the corresponding domain and the domain feature centroid and label index table of that domain are updated. When the highest comprehensive attribution score is lower than the domain entry threshold, a new data domain is created and the initial domain feature centroid and label index table are set with the data record as the domain center. When the number of records stored in each data domain reaches the capacity limit, a domain split is triggered, and the records in the domain are re-divided into two sub-domains and the corresponding domain feature centroids are updated.

4. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S2, data records in each data domain are calculated with a comprehensive popularity score based on the recent access frequency and access decay factor of the access popularity time series. Data with a comprehensive popularity score higher than the hot threshold are marked as hot data and stored in the first-level index area to build a high-frequency index tree. Data with a comprehensive popularity score lower than the cold threshold are marked as cold data and stored in the second-level index area to build a low-frequency compressed index. Data between the hot and cold thresholds is temporarily stored in a buffer until its score shifts into the cold or hot range. After inserting new data, the high-frequency index tree checks whether the leaf node load exceeds the splitting threshold. If it does, the node is split. When deleting data, it checks whether the leaf node fill rate is lower than the merging threshold. If it is lower, the node is merged.

5. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S2, based on the joint query records in the historical access logs of each data domain, the co-occurrence frequency of any two data domains being simultaneously hit in the same query is counted. A cross-domain association graph is constructed using the data domains as nodes and the co-occurrence frequency as edge weights. Node pairs with edge weights exceeding a preset jump threshold are selected, and bidirectional cross-domain jump pointers are established between the root nodes of the corresponding high-frequency index tree. During the retrieval process, when the hit result of the current data domain is lower than the expected threshold, the retrieval is directly jumped to the associated data domain through the cross-domain jump pointer to continue the retrieval, and the hit count of this jump is accumulated to dynamically update the edge weights of the association graph.

6. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S3, when calculating the relevance score based on the vector distance between the query semantic vector and the centroid of the domain features of the candidate data domain, a heat weighting coefficient is introduced according to the hot and cold classification status of the candidate data domain in S2. The heat weighting coefficient of the hot data domain is higher than that of the cold data domain. This makes the relevance score determined by the product or weighted sum of the basic vector distance score and the heat weighting coefficient. Furthermore, each dimension component of the query semantic vector is assigned a different dimension weight according to the attention of the corresponding dimension in the query intent tag before the vector distance is calculated.

7. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S3, each candidate data domain is sorted from high to low according to its relevance score. The top N candidate data domains are selected into the routing decision set. For each candidate data domain in the routing decision set, its relevance score, the load cost corresponding to the current load state, and the network transmission cost are multiplied by the corresponding weight coefficients and then summed to obtain the comprehensive routing cost value. The candidate data domain with the smallest comprehensive routing cost value and its storage node are selected as the target routing path. The weight coefficient of the load cost is dynamically adjusted according to the deviation between the actual delay value and the expected delay value after the storage node responds.

8. The method according to claim 1, characterized in that, In step S4, the first-level hit result set is accurately sorted according to the similarity between the query semantic vector and the semantic feature vector of each data record to generate a first sorting result, and a relevance threshold is set based on the relevance score of the last data record in the first sorting result. The secondary hit result set and the additional hit result set retrieved from the associated data domain through the cross-domain jump pointer are uniformly included in the auxiliary result set. The secondary hit result set and the auxiliary result set are traversed, and only data records with a relevance score higher than the relevance threshold are inserted into the corresponding sorting position of the first sorting result. After insertion and sorting, data entries with a relevance score lower than the relevance threshold are removed to form a merged sorting result set.

9. The data storage and retrieval method based on big data according to claim 1, characterized in that, In step S4, based on the user's click feedback on the final search result, the data domain to which the clicked data record belongs and the query semantic vector at the time of the click are statistically analyzed. The positive feedback difference between the relevance score of the clicked data record and the query semantic vector is used as the pulse update amount. A positive pulse increment is applied to the heat value of the corresponding data domain at the corresponding time point in the access heat time series. At the same time, the same positive pulse increment is applied to the edge weights corresponding to all routing links traversed by this query in the cross-domain association graph. When the data records clicked in the same query involve multiple data fields, each clicked field independently calculates the pulse update amount and adds it to its own popularity value and associated edge weight; the popularity value and edge weight corresponding to the unclicked data fields decay synchronously according to the decay factor.

10. A data storage and retrieval system based on big data, characterized in that, Includes the following modules: Semantic domain storage module; used to acquire multi-source heterogeneous data streams and extract semantic feature vectors and attribute tag sets, dynamically divide multiple data domains for distributed storage based on vector similarity and tag sets, and build a local metadata mapping table for each storage node that corresponds one-to-one with semantic abbreviation values ​​and physical addresses; The adaptive index building module, connected to the semantic domain storage module, is used to collect historical access logs to generate a hot time series and a cross-domain association graph. Based on the hot time series, it completes the hot and cold data classification, builds a dynamically adjustable high-frequency index tree for hot data, builds a low-frequency compressed index for cold data, and builds a cross-domain jump pointer based on the cross-domain association graph. Intelligent search and routing module; Connected to the adaptive index building module, it is used to parse user query statements and perform semantic expansion to generate query vectors and intent labels, match domain label indexes to filter candidate data domains and calculate weighted vector distance scores, and select the optimal routing path by taking into account node load cost and network transmission cost, and then send the query task to the target storage node. Hierarchical merging feedback module; It is connected to the intelligent retrieval routing module and the adaptive index building module respectively. It is used to retrieve the corresponding first-level index or second-level index on the target storage node according to the hot and cold classification status of the target data domain to obtain the hit result set. Combined with the additional hit results retrieved from the related data domain through the cross-domain jump pointer, the hit results are accurately sorted and then merged into the merged retrieval set according to the dynamic relevance threshold. After verification and deduplication, the retrieval content is output to the user. Collect user click feedback data and send it back to the adaptive index building module to update the weights of the popularity time series and cross-domain association graph.

Images