A Smart Indexing Method for Tuple States in Multichannel Transmission

Abstract

This invention relates to the field of information retrieval technology, specifically to an intelligent indexing method for tuple states in multi-channel transmission. The method includes: constructing a mapping relationship between physical channels and logical flows to achieve channel decoupling; extracting logical identifiers, key fields, and timestamps from tuples to generate a unique cross-channel fingerprint, and establishing a state index based on this fingerprint; collecting tuple access behavior data, constructing a passive attack-based state prediction model to predict state access frequency; using a reinforcement learning algorithm to perform dynamic sharding and cache optimization of state data based on the prediction results; introducing an index health indicator, combining the false positive rate of a Bloom filter and the query timeout rate to monitor channel health, and triggering an automatic reconstruction process to restore abnormal indexes. This invention can effectively improve the accuracy of multi-channel data state access, system response performance, and fault tolerance, and is suitable for large-scale real-time data processing scenarios.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to an intelligent indexing method for tuple states in multi-channel transmission. Background Technology

[0002] In current mainstream streaming data processing frameworks, such as Apache Flink, Apache Storm, and Apache Samza, multi-channel input mechanisms are widely used to improve the system's concurrent processing capabilities and data throughput efficiency. These frameworks typically achieve load balancing and horizontal scaling by dividing the data source into multiple physical input channels (such as Kafka partitions, Socket channels, or internal Task channels) and assigning these channels to different concurrent subtasks.

[0003] While these frameworks possess some state management capabilities—such as Flink's Keyed State and RocksDB backend, and Storm's State API—they still face several key challenges in multi-channel concurrent processing scenarios. First, the explicit binding between physical channels and processing logic results in state management logic being highly coupled to the channel structure, making flexible adjustments difficult. Second, data across different channels cannot effectively share state index information, making cross-channel state association and consistency maintenance challenging, impacting the globality and accuracy of state management.

[0004] Furthermore, the state indexing and sharding strategies in existing frameworks are mostly statically configured (such as key-based partitioning, range-based partitioning, etc.), lacking the ability to dynamically adjust based on access patterns. When traffic is uneven or state access hotspots change, the system cannot respond in a timely manner, which can easily lead to problems such as concentrated state hotspots, decreased cache hit rates, and frequent remote accesses, affecting overall processing performance and resource utilization efficiency.

[0005] To address this, a smart indexing method for tuple states in multi-channel transmission is proposed. Summary of the Invention

[0006] This invention provides an intelligent indexing method for tuple states in multi-channel transmission. This method decouples physical channels from logical business flows by constructing channel mapping relationships and extracts key fields from tuples to generate unique fingerprints for state indexing. The system collects access behavior data, combines it with a passive attack algorithm to predict state access frequency, and then dynamically adjusts the distribution of state data between local cache and remote storage through reinforcement learning. Furthermore, the index health is periodically evaluated, and an index rebuilding process is automatically triggered when it falls below a threshold, ensuring efficient and stable system operation.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A smart indexing method for tuple states in multi-channel transmission includes:

[0009] Construct a channel topology mapping to establish a mapping relationship between multiple physical input channels and one or more logical flows, so as to decouple the binding relationship between physical channels and state processing logic;

[0010] For the received tuples, the key fields in the tuples are extracted, and a hash tree structure is constructed using a hash algorithm to generate unique cross-channel identification information, which serves as the tuple fingerprint for subsequent state tracking and storage.

[0011] A state index is established based on tuple fingerprints, and tuple state information is recorded in the index structure in the form of key-value pairs, where the key is the tuple fingerprint and the value is the current tuple state.

[0012] Collect tuple access behavior data and build an access prediction model, including training a state prediction algorithm based on historical tuple flow rate, access frequency and channel status, to predict subsequent state access distribution and hot keys;

[0013] Based on the prediction results, dynamic sharding and index optimization operations are performed, and the distribution of state data between local cache and remote storage is adjusted through reinforcement learning algorithms.

[0014] Periodically perform index health monitoring operations, calculate the index health of each physical channel based on the Bloom filter false positive rate and index query timeout rate, and when the index health of a certain channel is lower than the preset threshold, it is determined to be an abnormal channel and triggers the automatic reconstruction process.

[0015] Furthermore, the decoupling steps between the physical channel and the state processing logic include:

[0016] Construct a channel mapping configuration table to describe the mapping relationship between physical channel identifiers and corresponding logical service flow identifiers;

[0017] When a tuple enters the processing flow, its physical channel information is extracted, and the corresponding logical business flow identifier is obtained according to the channel mapping configuration table.

[0018] The logical business flow identifier is appended to the tuple structure to form a tuple with a logical identifier.

[0019] Furthermore, the steps for generating unique identifier information across channels include:

[0020] Extract the logical flow identifier, key business fields, and event timestamp from the received tuple to form the information body;

[0021] The information body is concatenated according to a predetermined field order to form an original identifier string;

[0022] A hash digest is calculated by applying a hash algorithm to the original identifier string, and the calculated hash digest is used as the fingerprint of the tuple.

[0023] Furthermore, the state prediction algorithm is a passive attack algorithm, and the algorithm execution steps are as follows:

[0024] Collect streaming tuple access behavior data and construct a feature vector for state access prediction. The feature vector includes, but is not limited to: logical flow identifier, channel load, access frequency, tuple time interval, state size, and cache hit rate.

[0025] The feature vectors are input into the passive-attack classifier in mini-batch form, and the model is updated online through the incremental training interface of local fitting.

[0026] The updated model is used to predict the access frequency of each tuple state in subsequent time periods.

[0027] Furthermore, the steps for performing dynamic sharding and index optimization operations based on the prediction results include:

[0028] The current state of the system is represented as a state vector, which includes the tuple access popularity prediction result, local cache utilization, remote storage access latency, and cache hit rate.

[0029] Define the action space, including migrating tuple state data between local cache and remote storage and maintaining the current storage location;

[0030] Design a comprehensive reward function based on improved cache hit rate, reduced access latency, and migration overhead;

[0031] Using a reinforcement learning algorithm, an optimal strategy for dynamic sharding is trained based on the state vector and action space. Based on the optimal strategy, the distribution of tuple state data between local cache and remote storage is dynamically adjusted.

[0032] Furthermore, the formula for calculating the index health is as follows:

[0033] ;

[0034] in, Indicates the health of the index. This indicates the false positive rate of the Bloom filter. This indicates the index query timeout rate.

[0035] Furthermore, the automated reconstruction process includes:

[0036] Isolate the index structure corresponding to the abnormal channel;

[0037] Based on the adjacent channel hash chain structure, reconstruct the tuple state index data of the abnormal channel;

[0038] Erasure coding technology is used to recover lost or damaged data in order to restore index integrity and availability.

[0039] The beneficial effects of this invention are:

[0040] First, by constructing a channel topology mapping, the binding relationship between physical channels and logical business processing is effectively decoupled, enhancing the system's flexibility and scalability. Second, by using tuple key fields to generate unique fingerprints across channels, accurate tracking and efficient indexing of state are achieved, significantly reducing query conflicts and location overhead. Furthermore, a passive attack algorithm is employed to model and predict access behavior, identifying state hotspots in advance, which helps to achieve load-oriented dynamic sharding and cache optimization. Simultaneously, the integration of reinforcement learning strategies dynamically adjusts the distribution of state data between local cache and remote storage, effectively improving cache hit rate and system response performance.

[0041] Furthermore, by periodically calculating index health, the system can promptly detect index anomalies and automatically rebuild them, ensuring the stability and availability of the index structure, thereby improving the system's robustness and processing efficiency in large-scale concurrent environments. In summary, this invention achieves intelligent, adaptive, and high-performance state management while ensuring state consistency and index integrity. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a flowchart of an intelligent indexing method for tuple states in multi-channel transmission provided by the present invention. Detailed Implementation

[0044] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1

[0045] A smart indexing method for tuple states in multi-channel transmission includes:

[0046] S100: Construct channel topology mapping to establish mapping relationships between multiple physical input channels and one or more logic flows, so as to decouple the binding relationship between physical channels and state processing logic;

[0047] Furthermore, the decoupling steps between the physical channel and the state processing logic include:

[0048] Construct a channel mapping configuration table to describe the mapping relationship between physical channel identifiers and corresponding logical service flow identifiers;

[0049] When a tuple enters the processing flow, its physical channel information is extracted, and the corresponding logical business flow identifier is obtained according to the channel mapping configuration table.

[0050] The logical business flow identifier is appended to the tuple structure to form a tuple with a logical identifier.

[0051] Specifically, during system initialization, a channel mapping configuration table is loaded from the configuration center or scheduling platform. This table stores data in key-value pairs, such as {Physical Channel ID → Logical Flow ID}. Whenever a new data tuple enters the system through a physical channel (e.g., Kafka consumption, Netty access), its physical channel ID is first extracted in the tuple parsing module. The configuration table interface is called, using the physical channel ID as the key, to look up the corresponding logical flow ID in the mapping table, and the obtained logical flow ID is added as a new field to the original tuple structure. In subsequent state processing and routing modules, only the logical flow identifier is used for traffic splitting and state index binding, not the original physical channel ID. In this way, when physical channels change, migrate, or expand, only the configuration table needs to be adjusted to achieve seamless adaptation without changing the processing logic.

[0052] The above implementation scheme separates physical channels from logical state processing logic through a channel mapping configuration table, eliminating the direct impact of channel changes on business logic and enhancing the system's module decoupling, topology flexibility, and maintainability. It is particularly suitable for multi-source access scenarios in stream computing frameworks such as Apache Storm, Flink, and Spark Streaming. This mechanism not only improves the system's flexibility and scalability in multi-channel environments but also makes the scheduling and reconstruction of logical streams more efficient, facilitating rapid adaptation to dynamic changes in channel topology and reducing the risk of anomalies caused by state mismatches and coupled processing logic.

[0053] S200: For the received tuple, extract the key fields in the tuple, use a hash algorithm to build a hash tree structure, thereby generating unique identification information across channels, which serves as the tuple fingerprint for subsequent state tracking and storage;

[0054] Furthermore, the steps for generating unique identifier information across channels include:

[0055] Extract the logical flow identifier, key business fields, and event timestamp from the received tuple to form the information body;

[0056] The information body is concatenated according to a predetermined field order to form an original identifier string;

[0057] A hash digest is calculated by applying a hash algorithm to the original identifier string, and the calculated hash digest is used as the fingerprint of the tuple.

[0058] Whenever a new tuple enters the system, several key fields are first extracted from the tuple in the preprocessing module to form an identification information body. These fields include, but are not limited to:

[0059] logical_stream_id: The identifier of the logical stream to which the tuple belongs (e.g., "stream_1");

[0060] business_key: A field that can uniquely represent the meaning of business data, such as order number, user ID, etc. (e.g., "order_123456");

[0061] event_timestamp: indicates the time when the data occurred (e.g., "2025-06-26T10:20:00Z").

[0062] The above fields are concatenated in a predefined order to form the original identifier string. Fields can be separated by "|" or other special symbols to ensure the uniqueness and reconstructability of the concatenation. For example: "stream_1|order_123456|2025-06-26T10:20:00Z". A high-performance hash algorithm (such as MurmurHash, SHA-256, CityHash, or Blake3) is used to calculate the digest of the original identifier string, generating a fixed-length hash value as the unique identifier information (i.e., tuple fingerprint) for the tuple. The generated tuple fingerprint will be used as the primary key of the state index table, combined with the tuple state value, and recorded in the state manager to achieve consistent state access and location capabilities across channels, threads, or clusters.

[0063] By using a hash algorithm to generate a unique fingerprint identifier through an information body composed of logical flow identifiers, key business fields, and event times, the accuracy and efficiency of tuple state identification and positioning in a distributed, multi-channel environment are significantly improved, avoiding state management errors caused by channel overlap, field conflicts, or time ambiguity.

[0064] S300: Establish a state index based on tuple fingerprints, and record tuple state information in the index structure in the form of key-value pairs, where the key is the tuple fingerprint and the value is the current tuple state.

[0065] Specifically, the generated tuple fingerprint (usually a hash digest generated by processing key fields such as logical identifiers, business fields, and timestamps using a hash algorithm) is received and used as the unique key for subsequent indexing.

[0066] For each tuple, extract its current state information based on business logic. This state information may include, but is not limited to: the current processing step (e.g., whether parsing is complete, whether routing has been completed), state version number, last modification time, and intermediate business states or context data associated with that tuple. The state information can be encapsulated as a JSON object or a structure format and used as the index value.

[0067] S400: Collects tuple access behavior data and builds an access prediction model, including training a state prediction algorithm based on historical tuple flow rate, access frequency and channel status, which is used to predict the subsequent state access distribution and hot keys.

[0068] Furthermore, the state prediction algorithm is a passive attack algorithm, and the algorithm execution steps are as follows:

[0069] Collect streaming tuple access behavior data and construct a feature vector for state access prediction. The feature vector includes, but is not limited to: logical flow identifier, channel load, access frequency, tuple time interval, state size, and cache hit rate.

[0070] The feature vectors are input into the passive-attack classifier in mini-batch form, and the model is updated online through the incremental training interface of local fitting.

[0071] The updated model is used to predict the access frequency of each tuple state in subsequent time periods.

[0072] Specifically, the system first continuously collects access behavior data for each tuple during streaming processing, including but not limited to logical stream identifier, channel load, access frequency, tuple time interval, state size, and cache hit rate. This information forms a feature vector. As input to the state access prediction model, where The dimension representing the feature.

[0073] To meet the system's requirements for real-time prediction and model adaptation, the state access prediction model employs a passive attack algorithm for online learning and prediction. Its training and update process is as follows:

[0074] The system periodically extracts access features from the latest batch of processed tuples and constructs a feature vector. Each component in the vector is a standardized numerical feature.

[0075] Based on historical statistics or business rules, set tags according to access popularity. ,For example This indicates that the current tuple is in a high-heat state; This indicates that the current tuple is in a low-popularity state. It can also be expanded to multi-category labels (such as three popularity levels) or continuous popularity values ​​(used in regression models).

[0076] Use the current weight vector For the sample Make a prediction: ,in Indicates predicted popularity tags, The sign function is represented. The Hinge loss function is introduced to evaluate the current prediction error. ,like This indicates that the current prediction is correct and the model remains unchanged. The basic PA-I update formula performs the following model update: ,in The algorithm, implemented in Java, uses a local fitting incremental training interface deployed within the tuple processing module. It supports learning the model while receiving data, eliminating the need for batch offline training. The predicted access popularity level is used for subsequent state data migration strategies and cache optimization.

[0077] The aforementioned state prediction method based on a passive attack algorithm fully utilizes real-time access behavior data from streaming tuples. By constructing feature vectors containing multi-dimensional features and employing a mini-batch incremental training approach, the model acquires online learning and adaptive adjustment capabilities. This method eliminates the need for large-scale offline training, dynamically capturing changes in state access patterns and accurately predicting the access popularity of each tuple's state. This facilitates early detection of hot states and optimized resource scheduling.

[0078] S500: Performs dynamic sharding and index optimization operations based on prediction results, and adjusts the distribution of state data between local cache and remote storage through reinforcement learning algorithms;

[0079] Furthermore, the steps for performing dynamic sharding and index optimization operations based on the prediction results include:

[0080] The current state of the system is represented as a state vector, which includes the tuple access popularity prediction result, local cache utilization, remote storage access latency, and cache hit rate.

[0081] Define the action space, including migrating tuple state data between local cache and remote storage and maintaining the current storage location;

[0082] Design a comprehensive reward function based on improved cache hit rate, reduced access latency, and migration overhead;

[0083] Using a reinforcement learning algorithm, an optimal strategy for dynamic sharding is trained based on the state vector and action space. Based on the optimal strategy, the distribution of tuple state data between local cache and remote storage is dynamically adjusted.

[0084] Specifically, the system collects current status indicators of each channel periodically or by triggering, and constructs an environmental state vector. The state vector is used to characterize the system's operating status. It consists of: ,in: This indicates the predicted access popularity of tuples within the current time window; Indicates local cache usage; Indicates remote storage access latency; This indicates the cache hit rate.

[0085] The system defines a finite set of actions. Indicates the executable migration operation:

[0086] ;

[0087] : Keep the current data distribution unchanged;

[0088] Migrate hotspot status data to local cache;

[0089] Migrate low-hot or long-term unhits to remote storage.

[0090] To guide the model in learning a reasonable transfer strategy, the following comprehensive reward function can be designed. To simultaneously consider system performance and overhead:

[0091] ;

[0092] in, This indicates the extent of the improvement in cache hit rate. This indicates the amount of change in access latency. This indicates the data migration overhead incurred by the current action. , and This represents the weighting parameter, used to balance the importance of various indicators.

[0093] Policy training is implemented using deep reinforcement learning algorithms (such as DQN or Actor-Critic). The process is as follows:

[0094] Policy function Represented by a neural network or policy table, used to analyze states. Choose the optimal action ;

[0095] value function or Used to evaluate the long-term benefits of a state or state-action pair;

[0096] Update mechanism: Use time difference method or policy gradient method, combined with experience replay and batch update to achieve training stability.

[0097] After processing each tuple state access request, the system determines the state based on the current state. Calling the strategy function Select Action And receive rewards based on the execution results. At the same time, the reinforcement learning model is updated.

[0098] By modeling the system's operational state as a state vector and defining a reasonable action space and reward function, the system can adaptively learn the optimal state data migration strategy. Compared to static or rule-driven sharding methods, this approach can dynamically adjust the distribution of state data between local cache and remote storage by combining multi-dimensional information such as access popularity prediction results, cache usage, and storage latency. This effectively improves cache hit rate, reduces access latency, and optimizes overall resource allocation.

[0099] S600: Periodically performs index health monitoring operations, calculates the index health of each physical channel based on the Bloom filter false positive rate and index query timeout rate, and determines the abnormal channel when the index health of a certain channel is lower than the preset threshold and triggers the automatic reconstruction process.

[0100] Furthermore, the formula for calculating the index health is as follows:

[0101] ;

[0102] in, Indicates the health of the index. This represents the false positive rate of the Bloom filter, which is the ratio of the number of false positives to the total number of indexes indexed. This indicates the index query timeout rate. Its value is the ratio between the number of times the ultrasonic index was used to the total number of indexes. The closer to 1, the healthier the channel index. If the index health falls below a set threshold, it is considered an abnormality in the index structure. The system can set a health threshold according to actual business needs and fault tolerance requirements. The threshold can be statically configured or dynamically and adaptively adjusted by the system based on historical fluctuations. In this embodiment, 0.8 is preferred. It should be noted that there can be other methods for calculating index health; the calculation method proposed in this invention is only for reasonable reference.

[0103] By introducing an index health metric, which combines the false positive rate of the Bloom filter with the index query timeout rate, the reliability and response performance of the indexes in each physical channel can be quantitatively evaluated. This mechanism helps the system to promptly detect issues such as index degradation, missing data, or abnormal access during operation, and trigger corresponding rebuilding and optimization operations.

[0104] Furthermore, the automated reconstruction process includes:

[0105] Isolate the index structure corresponding to the abnormal channel;

[0106] Based on the adjacent channel hash chain structure, reconstruct the tuple state index data of the abnormal channel;

[0107] Erasure coding technology is used to recover lost or damaged data in order to restore index integrity and availability.

[0108] Specifically, first, execute the abnormal channel. The index structure is isolated to prevent its erroneous state from continuing to affect query results or pollute surrounding states. This includes: pausing index writes and queries corresponding to the channel; setting the channel status to "isolated"; and synchronously recording the timestamp of the exception occurrence. This is to define the time frame for fault recovery.

[0109] For the isolated passage The system utilizes its logically adjacent channels and The existing hash chain structure is cross-completed and restored. The specific implementation is as follows:

[0110] Compare time windows in adjacent channels The tuple record within;

[0111] Status backfilling is performed by matching the same hash fingerprint or some fields (such as stream ID + timestamp ± sliding window);

[0112] If both channels share a common tuple state, the newest one is selected as the recovery value.

[0113] If only one side exists, it is marked as "pending recovery" and proceeds to the next step of the erasure coding recovery process.

[0114] For state units marked "pending recovery," the system further applies erasure coding technology to repair the data. Details are as follows:

[0115] When each piece of state data is written, it is encoded using a k+n erasure coding scheme (such as Reed-Solomon coding), dividing the state into k original fragments and n redundant check fragments;

[0116] State fragments lost in an abnormal channel can be recovered by decoding redundant fragments retained in adjacent channels;

[0117] As long as the number of available fragments during the recovery process is greater than or equal to k, the state content can be completely restored;

[0118] The recovered state is rewritten into the isolated index structure and marked as "available".

[0119] Once the index reconstruction is complete and the health indicators recover to above the threshold, the system automatically removes the channel isolation status and restores its index read and write functions.

[0120] By isolating the index structure of abnormal channels and combining the redundant information of the hash chain of adjacent channels to achieve rapid completion of status data, and introducing erasure coding technology to perform fault-tolerant recovery of damaged or lost data, the system's recoverability and fault self-healing ability in a multi-channel environment are effectively improved. Example 2

[0121] This embodiment demonstrates the practical application of the intelligent indexing method for tuple states in multi-channel transmission in the Apache Flink stream processing platform. It supports unified state management and access optimization for large-scale heterogeneous data channels and is suitable for high-throughput, low-latency scenarios such as ad click monitoring, IoT sensor streams, and real-time financial monitoring.

[0122] The system is deployed in an Apache Flink cluster and needs to receive user behavior log streams (such as web, mobile, and third-party proxies) from multiple Kafka Topics (representing different physical data sources). Each Topic belongs to a different data channel, and the data structures are not entirely consistent, but they belong to a unified logical processing flow in terms of business logic.

[0123] First, a channel mapping table is established through configuration. For example, the three Kafka topics "web_events", "mobile_events", and "proxy_events" are mapped to a unified logical flow "L_CLICK_STREAM" for subsequent unified modeling. In Flink's SourceFunction module, after receiving each piece of data, the system extracts its source channel identifier, looks up the mapping relationship, and dynamically adds a logical flow identifier field to the tuple, thus forming a "tuple with a logical label" for subsequent unified processing.

[0124] Extract key business fields (such as user ID, event type, and object ID) and event timestamps from each tuple, and concatenate these fields in a preset order to form the original string. Then, use a consistent hash algorithm (such as MurmurHash or SHA256) to generate a hash digest of this string, producing a unique identifier used to identify the tuple's cross-channel fingerprint. This approach ensures that tuples with the same business meaning retain the same fingerprint across different channels, facilitating state aggregation and consistent tracing.

[0125] The Flink system uses this fingerprint as the key for state storage, and the corresponding state content (such as user behavior summaries, session information, behavior windows, etc.) as the value, storing them in MapState or RocksDBStateBackend as key-value pairs. Because the fingerprint is unique, the index can maintain data from multiple channels without duplication, supporting unified access.

[0126] Real-time collection of tuple behavior data related to state access is used to construct feature vectors, including logical flow ID, channel load, access frequency, time interval between two accesses, state data size, and cache hit rate. This vector is then used as input to a passive attack classification model, which continuously updates its prediction parameters using an incremental learning method based on local fitting. A sigmoid activation function is used to output a predicted state access heat value, reflecting the likelihood of the state being frequently accessed in the future, which is then used to optimize storage strategies.

[0127] The current state environment is encoded as a state vector, including metrics such as the popularity prediction value obtained in the previous step, local cache utilization, remote storage access latency, and current hit rate. A set of actions is also defined, including "stay in current position," "migrate to cache," and "migrate to remote storage." The system employs a reinforcement learning algorithm, designing a reward function that comprehensively considers improving cache hit rate, reducing access latency, and migrating overhead to learn the optimal action strategy.

[0128] The health of the index structure for each physical channel is periodically assessed. This is calculated using two metrics: the false positive rate of the Bloom filter, reflecting the proportion of invalid queries; and the timeout rate during status queries. These two metrics together constitute the index health value. When the health of a certain channel is lower than the set threshold (such as 0.7), the system automatically determines that the channel index is abnormal.

[0129] Subsequently, the system isolates the corresponding index structure and reconstructs the potentially missing tuple state index data for that channel using the hash chain information of time-aligned adjacent channels. Simultaneously, erasure coding is used for data block recovery and verification, ultimately restoring the integrity and availability of the channel index without manual intervention.

[0130] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart indexing method for tuple states in multi-channel transmission, characterized in that, include: Construct a channel topology mapping to establish a mapping relationship between multiple physical input channels and one or more logical flows, so as to decouple the binding relationship between physical channels and state processing logic; For the received tuples, the key fields in the tuples are extracted, and a hash tree structure is constructed using a hash algorithm to generate unique cross-channel identification information, which serves as the tuple fingerprint for subsequent state tracking and storage. A state index is established based on tuple fingerprints, and tuple state information is recorded in the index structure in the form of key-value pairs, where the key is the tuple fingerprint and the value is the current tuple state. Collect tuple access behavior data and build an access prediction model, including training a state prediction algorithm based on historical tuple flow rate, access frequency and channel status, to predict subsequent state access distribution and hot keys; Based on the prediction results, dynamic sharding and index optimization operations are performed, and the distribution of state data between local cache and remote storage is adjusted through reinforcement learning algorithms. Periodically perform index health monitoring operations, calculate the index health of each physical channel based on the Bloom filter false positive rate and index query timeout rate, and when the index health of a certain channel is lower than the preset threshold, it is determined to be an abnormal channel and triggers the automatic reconstruction process. The state prediction algorithm is a passive attack algorithm, and its execution steps are as follows: Collect streaming tuple access behavior data and construct a feature vector for state access prediction. The feature vector includes, but is not limited to: logical flow identifier, channel load, access frequency, tuple time interval, state size, and cache hit rate. The feature vectors are input into the passive attack algorithm in mini-batch form, and the model is updated online through the incremental training interface of local fitting. The updated model is used to predict the access popularity of each tuple state in subsequent time periods; The steps for performing dynamic sharding and index optimization based on the prediction results include: The current state of the system is represented as a state vector, which includes the tuple access popularity prediction result, local cache utilization, remote storage access latency, and cache hit rate. Define the action space, including migrating tuple state data between local cache and remote storage and maintaining the current storage location; Design a comprehensive reward function based on improved cache hit rate, reduced access latency, and migration overhead; Using a reinforcement learning algorithm, an optimal strategy for dynamic sharding is trained based on the state vector and action space. Based on the optimal strategy, the distribution of tuple state data between local cache and remote storage is dynamically adjusted.

2. The intelligent indexing method for tuple states in multi-channel transmission according to claim 1, characterized in that, The steps to decouple the physical channel from the state processing logic include: Construct a channel mapping configuration table to describe the mapping relationship between physical channel identifiers and corresponding logical service flow identifiers; When a tuple enters the processing flow, its physical channel information is extracted, and the corresponding logical business flow identifier is obtained according to the channel mapping configuration table. The logical business flow identifier is appended to the tuple structure to form a tuple with a logical identifier.

3. The intelligent indexing method for tuple states in multi-channel transmission according to claim 1, characterized in that, The steps to generate unique identifiers across channels include: Extract the logical flow identifier, key business fields, and event timestamp from the received tuple to form the information body; The information body is concatenated according to a predetermined field order to form an original identifier string; A hash digest is calculated by applying a hash algorithm to the original identifier string, and the calculated hash digest is used as the fingerprint of the tuple.

4. The intelligent indexing method for tuple states in multi-channel transmission according to claim 1, characterized in that, The formula for calculating the health of the index is: ； in, Indicates the health of the index. This indicates the false positive rate of the Bloom filter. This indicates the index query timeout rate.

5. The intelligent indexing method for tuple states in multi-channel transmission according to claim 1, characterized in that, The automatic reconstruction process includes: Isolate the index structure corresponding to the abnormal channel; Based on the adjacent channel hash tree structure, reconstruct the tuple state index data of the abnormal channel; Erasure coding technology is used to recover lost or damaged data in order to restore index integrity and availability.

Images