Data development, transmission, optimization and storage method based on big data processing

Abstract

The invention discloses a data development, transmission and optimization storage method based on big data processing. The method comprises the following steps: acquiring original key data and processing the original key data to generate a state activation vector; constructing a full-state space structure of the finite-state transduction device; executing state observability cutting processing to obtain an observable state subspace; establishing a semantic mapping relation and determining a legal state transduction path sequence; executing data transmission processing on the target data according to the legal state transduction path sequence; recording a time slice identifier, a key derived fragment and a historical path access mark, and establishing a binding relationship; executing path uniqueness verification processing, judging a failure legal state transduction path sequence and stopping data transmission; and after failure, state observability cutting processing is executed again until the data closed-loop transmission process is completed. According to the method, a shared key driven finite state transduction method is adopted, rapid, safe and stable safe closed-loop transmission of data is realized, and the method has the advantages of unique path and dynamic encryption.

Description

Technical Field

[0001] This invention relates to the field of data communication security technology, and in particular to a data development, transmission and storage optimization method based on big data processing. Background Technology

[0002] With the increasing demands for data development and cross-system data interaction, existing data transmission and storage technologies typically rely on fixed encrypted channels or single-path transmission mechanisms. They achieve data security through traditional key negotiation and session encryption, and maintain communication reliability through conventional integrity checks and replay detection. However, these technologies are mostly based on static transmission structure designs. They lack unified modeling of the data's state evolution, path structure constraints, and semantic association features during communication, making it difficult to simultaneously achieve transmission efficiency, structural security, and process controllability in complex data development scenarios.

[0003] Under current technological conditions, problems such as data transmission path reuse, unobservable state transitions, and decoupling between key derivation and transmission phases still exist. These issues can easily lead to increased predictability of the communication structure and reduced security boundaries. Furthermore, the lack of an adaptive reconstruction mechanism under path failure or conflict conditions makes it difficult to form a continuously controlled closed-loop data transmission process. Therefore, there is an urgent need for a novel data development, transmission, and storage optimization method that can achieve state space pruning, path uniqueness verification, and dynamic reconstruction of transmission under shared key constraints.

[0004] Therefore, how to provide a data development, transmission, and storage optimization method based on big data processing is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a data development, transmission, and storage optimization method based on big data processing. This invention employs a shared key-driven finite state transduction method to achieve fast, secure, and stable secure closed-loop transmission of data, possessing the advantages of unique path and dynamic encryption.

[0006] A data development, transmission, and storage optimization method based on big data processing according to an embodiment of the present invention includes the following steps: Obtain the original key data shared by both communicating parties, and perform hash mapping and bit-level expansion processing to generate a state activation vector; Construct the full state space structure of the finite state transducer; Based on the correlation between the state activation vector and the transition weights in the full state space structure of the finite state transducer, a state observability pruning process of structural coupling is performed to obtain the observable state subspace. Establish a semantic mapping relationship between data content and state transition paths within the observable state subspace, and determine the corresponding legal state transition path sequence; Perform data transmission processing on the target data according to the legal state transduction path sequence; During data transmission, time slice identifiers, key derivation fragments, and historical path access markers are recorded, and a binding relationship is established between them and the legitimate state transition path sequence. Based on the binding relationship, the path uniqueness verification process is performed. When the path is duplicated or the state is conflicted, the valid state is deduced to be invalid and the data transmission is stopped. After the valid state transduction path sequence fails, a new state activation vector is regenerated, and state observability pruning is performed until the data closed-loop transmission process under shared key control is completed.

[0007] Optionally, generating the state activation vector corresponding to the shared key includes: It receives the original key data provided by the communication initiator and the communication receiver respectively, performs consistency verification on the original key data, and uses the original key data as input when the consistency verification passes. The original key data that has passed the consistency check is input into the cryptographic hash function to generate a fixed-length hash digest; Perform bit-level expansion on the hash digest to convert the hash digest into a continuous bit sequence in binary bit order; Perform sequential segmentation on the continuous bit sequence according to the segment length to generate multiple bit segments; Numerical mapping is performed on each segment to convert each segment into a decimal value, and the activation value corresponding to each segment is determined based on the correspondence between the decimal value and the threshold interval. Arrange the activation values ​​corresponding to each segment in the order of the segments in the continuous bit sequence to generate an activation value sequence; Perform sequential combination processing on the activation value sequence to generate a state activation vector corresponding to the shared key.

[0008] Optionally, the generation of the full state space structure of the finite state transducer includes: Multiple states are determined based on the phase division rules of the data transmission process based on shared keys, and a unique corresponding state identifier is generated for each state. All state identifiers are then collected in order to form a state set. In the data transmission process based on shared keys, the sequential constraints of the stages determine the reachability relationships between each state, and each state is sequentially associated with at least one successor state according to the reachability relationships to generate multiple state transition paths, which are then aggregated to form a transition set. Establish a weighted transduction relationship for each state transition path in the transition set; The transition weights for each state transition path are calculated and processed uniformly. By structurally associating and integrating the state set, transition set, and weighted transduction relationship, a full state space structure of a finite state transducer is generated.

[0009] Optionally, the generation of the observable state subspace includes: Read the state set, transition set, and weighted transduction relation in the full state space structure of the finite state transducer, and establish a one-to-one sequential correspondence between the state set and the state activation vector according to the arrangement order of the state set; For each state in the state set, extract the corresponding state activation vector value, and extract the set of transition weight values ​​for all state transition paths in the transition set in which the state participates. Perform aggregation calculation on the set of transition weight values ​​to generate state structure associated values. The state activation vector values ​​and the state structure associated values ​​are coupled and processed to obtain the structural coupling values. Based on the structural coupling values ​​of all states, a sorting and pruning process is performed, and the states that are in the first interval of the sorting result are determined as observable states. Retain all observable states and only the state transition paths connecting the observable states and their corresponding weighted transduction relationships to generate an observable state subspace corresponding to the current shared key.

[0010] Optionally, the establishment of the legal state transition path sequence includes: Acquire the target data and perform content parsing processing on the target data to generate a data content description; Traverse the set of states and the set of transitions within the observable state subspace, extract all state transition paths formed by continuous connection from the start state to the end state, and generate a path description for each state transition path. The path description includes the number of states and the number of transitions contained in the state transition path. Semantic matching processing is performed based on the data content description and path description to establish a semantic mapping relationship between the data content and the state transition path, so that the state transition path that meets the data content description constraints is determined as a candidate state transition path. The path selection value is generated based on the shared key, and the target state transition path is determined from the candidate state transition paths according to the path selection value. Generate a sequence of valid state transition paths corresponding to the target data according to the connection order of each state in the target state transition path.

[0011] Optionally, the data transmission process includes: Obtain the sequence of valid state transition paths, and determine the execution order of the paths based on the state order in the sequence of valid state transition paths; Perform sequential segmentation processing on the target data to generate a sequence of data segments corresponding to the execution order of the path; Based on the path execution order, the current state and the next state are determined sequentially from the legal state transformation path sequence, and the current state is determined as the current state of the target data during the transmission of the current data segment; The sequential determination of the current state and the next state ensures that the transmission of each data segment is within a defined state interval. This state interval not only describes the transmission stage of the data but also serves as the state input condition for generating the session encryption key. By binding the data segment with the state interval, it can be ensured that the data encryption results under different states are independent of each other, forming a dynamic encryption structure that changes with the state evolution. Based on the shared key and the current state, perform key derivation processing to generate a session encryption key that uniquely corresponds to the current state; The session encryption key is used to encrypt the data segment corresponding to the current state, generating an encrypted data segment. The encrypted data segment is sent to the communication receiver, and after the transmission is completed, the current state corresponding to the target data is updated to the next state to complete a state transition and encrypted transmission. The current state determination, key derivation, encryption, transmission, and state update are repeatedly executed in the order of the path execution until all states in the valid state transition path sequence complete the state transition and encrypted transmission in sequence, thus completing the data transmission processing of the target data.

[0012] Optionally, establishing the binding relationship includes: During data transmission, the current time value is obtained, and time quantization processing is performed on the current time value to generate a time slice identifier corresponding to the legal state transition path sequence. During data transmission, sequential segmentation is performed based on the shared key, and continuous key fragments are extracted from the sequential segmentation results to generate key derivation fragments corresponding to the legal state transduction path sequence. During data transmission, path access marker generation is performed on the valid state transformation path sequence, and hash mapping is performed on the sequential combination processing result to generate historical path access markers corresponding to the valid state transformation path sequence. The time slice identifier, key derivation fragment, and historical path access mark are associated with the legal state transition path sequence through a record-associating process. This creates a one-to-one correspondence between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence in the same record, thereby establishing a binding relationship between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence.

[0013] Optionally, stopping the continued transmission of the target data includes: Obtain the binding relationship corresponding to the legal state transduction path sequence, and read the time slice identifier, key derivation fragment, and historical path access mark from the binding relationship; When performing subsequent state transitions, the current state and the next state are extracted, and the next state is compared with the state at the corresponding position in the legal state transition path sequence to generate a state consistency determination result. Perform sequential combination processing based on time slice identifier, key derivation fragment and historical path access mark, and perform hash mapping processing on the result of sequential combination processing to generate path verification value; Based on the path verification value, a matching search is performed in the historical records corresponding to the binding relationship. When the same path verification value is found or the status consistency judgment result is inconsistent, the valid status transformation path sequence is judged as invalid. When a valid state transduction path sequence is determined to be invalid, the continued transmission of target data is stopped.

[0014] Optionally, the process of completing the closed-loop data transmission under shared key control includes: After the valid state transduction path sequence fails, the shared key is obtained, and hash mapping and bit-level expansion are performed on the shared key to generate a new state activation vector. Read the full state space structure of the finite state transducer, and perform state observability pruning on the full state space structure of the finite state transducer again based on the new state activation vector to obtain a new observable state subspace. Within the new observable state subspace, determine the new legitimate state transduction path sequence corresponding to the target data based on the shared key; The data transmission process is re-executed on the target data according to the new legal state transduction path sequence; While the target data has not been fully transmitted, the process of generating new state activation vectors, pruning state observability, determining the sequence of legal state transition paths, and transmitting data is repeated until the closed-loop data transmission process under shared key control is completed.

[0015] The beneficial effects of this invention are: This invention constructs a full-state space structure of a finite state transducer under shared key constraints and performs state observability pruning of structural coupling using state activation vectors. This ensures that data transmission is confined to an observable state subspace highly correlated with the current key before entering the communication phase. By eliminating states and paths with low coupling or invalidity, redundant computations in path search and structure judgment are reduced, lowering scheduling overhead and improving data transmission efficiency. Simultaneously, a semantic mapping relationship is established between data content and legitimate state transducer path sequences based on the observable state subspace. This allows data semantics to be implicitly expressed through the path structure. Combined with a dynamic key derivation mechanism that changes with state evolution, the data encryption structures in different state intervals are independent, significantly enhancing the structural concealment and attack resistance of the transmission process and ensuring data security during transmission.

[0016] Furthermore, this invention establishes a binding relationship between time slice identifiers, key derivation fragments, historical path access markers, and legitimate state transition path sequences, and performs path uniqueness verification. This enables real-time identification and blocking of path reuse and state conflicts during transmission, preventing the accumulation and spread of risks during communication. When a legitimate state transition path sequence fails, the system can regenerate the state activation vector based on the shared key and rapidly reconstruct the observable state subspace and path structure. The transmission process can be restored without re-establishing a complete communication environment, thus ensuring the continuity and stability of data transmission. Therefore, this invention achieves efficient operation, security, and long-term stability of data transmission under controlled structural conditions, forming a fast, secure, and stable closed-loop data transmission mechanism. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a data development, transmission, and storage optimization method based on big data processing proposed in this invention; Figure 2 This is a schematic diagram of the full state space structure of the finite state transducer and the pruning relationship of its observable state subspace in a data development, transmission, optimization, and storage method based on big data processing proposed in this invention. Figure 3 This is a schematic diagram of the dynamic key derivation and data segmentation encryption transmission process based on state evolution in the data development, transmission optimization and storage method based on big data processing proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-3 A data development, transmission, and storage optimization method based on big data processing includes the following steps: Obtain the original key data shared by both communicating parties, and perform hash mapping and bit-level expansion processing on the original key data to generate a state activation vector corresponding to the shared key; Construct the full state space structure of the finite state transducer, which includes the state set, the transition set, and the weighted transduction relationship. Based on the correlation between the state activation vector and the transition weights in the full state space structure of the finite state transducer, the state observability pruning process of the structural coupling is performed on the full state space structure of the finite state transducer to obtain the observable state subspace corresponding to the current shared key; Establish a semantic mapping relationship between data content and state transition paths within the observable state subspace, and determine the sequence of legitimate state transition paths corresponding to the target data based on the shared key; Data transmission processing is performed on the target data according to the legal state transition path sequence, so that the target data completes state transition and encrypted transmission sequentially along the legal state transition path sequence during the communication process; During data transmission, the time slice identifier, key derivation fragment, and historical path access mark corresponding to the legitimate state transduction path sequence are recorded, and the binding relationship between the time slice identifier, key derivation fragment, and historical path access mark and the legitimate state transduction path sequence is established. Based on the binding relationship, the path uniqueness verification process is performed on the subsequent state transit. When path duplication or state conflict is detected, the valid state transit path sequence is determined to be invalid and the continued data transmission of the target data is stopped. After the valid state transduction path sequence fails, a new state activation vector is generated again based on the shared key, and the state observability pruning process is performed again on the full state space structure of the finite state transducer to obtain a new observable state subspace and a new valid state transduction path sequence. Data transmission processing is then performed again based on the new valid state transduction path sequence until the data closed-loop transmission process under the control of the shared key is completed.

[0020] In this embodiment, generating the state activation vector corresponding to the shared key includes: It receives the original key data provided by the communication initiator and the communication receiver respectively, performs consistency verification on the original key data, and uses the original key data as input when the consistency verification passes. The original key data that has passed the consistency check is input into the cryptographic hash function to generate a fixed-length hash digest. The fixed length is determined by a preset length parameter, which is used to limit the number of bits in the output hash digest so that the dimension of the state activation vector can be determined during the system design phase. Perform bit-level expansion on the hash digest to convert the hash digest into a continuous bit sequence in binary bit order; Bit-level expansion is used to convert the hash digest from a byte-level representation to a continuous binary bit representation, so that all the information in the hash digest can participate in subsequent calculations at the smallest granularity. The continuous bit sequence is arranged in a uniform order from high bit to low bit or from low bit to high bit to ensure the consistency of bit order between the two communicating parties in the process of generating the state activation vector. Perform sequential segmentation on the continuous bit sequence according to the segment length to generate multiple bit segments; Sequential segmentation is used to divide a continuous bit sequence into multiple bit segments according to a fixed segment length. The segment length can be determined based on the number of states in the full state space structure of the finite state transducer or the trimming accuracy requirements. Numerical mapping is performed on each segment to convert each segment into a decimal value, and the activation value corresponding to each segment is determined based on the correspondence between the decimal value and the threshold interval. Arrange the activation values ​​corresponding to each segment in the order of the segments in the continuous bit sequence to generate an activation value sequence; Perform sequential combination processing on the activation value sequence to generate a state activation vector corresponding to the shared key.

[0021] In this embodiment, the generation of the full state space structure of the finite state transducer includes: Multiple states are determined based on the phase division rules of the data transmission process based on shared keys, and a unique corresponding state identifier is generated for each state. All state identifiers are then collected in order to form a state set. The phase division rules are determined based on the functional execution order in the data transmission process based on shared keys. Each phase corresponds to the execution intervals such as data preparation, path selection, state transition, encryption processing, and transmission completion. Each phase is mapped to a discrete state in a finite state transducer. By assigning a fixed-length state identifier code to each state, different states can be uniquely identified in the full state space structure. In the data transmission process based on shared keys, the sequential constraints of the stages determine the reachability relationships between each state, and each state is sequentially associated with at least one successor state according to the reachability relationships to generate multiple state transition paths, which are then aggregated to form a transition set. The reachability relationship is determined based on the stage sequence constraints and the allowed jump rules in the data transmission process. The stage sequence constraints are used to limit the basic direction of state transition, while the allowed jump rules are used to describe the optional successor states in the case of anomaly recovery or path reconstruction. By considering the sequential progression relationship and the restricted jump relationship at the same time, a state transition structure containing the main path and the backup path can be formed. For each state transition path in the transition set, a weighted transduction relationship is established. The weighted transduction relationship includes the transition weight value associated with the corresponding state transition path and the transduction output value associated with the corresponding state transition path. The transition weight values ​​for each state transition path are calculated using a unified method, which is determined as follows: the communication load value is multiplied by the first weight coefficient, the integrity verification failure count value is multiplied by the second weight coefficient, the replay risk value is multiplied by the third weight coefficient, and the three products are summed to obtain the transition weight value for the corresponding state transition path. By structurally associating and integrating the state set, transition set, and weighted transduction relationship, a full state space structure of a finite state transducer is generated.

[0022] In this embodiment, the generation of the observable state subspace includes: Read the state set, transition set, and weighted transduction relation in the full state space structure of the finite state transducer, and establish a one-to-one sequential correspondence between the state set and the state activation vector according to the arrangement order of the state set; For each state in the state set, extract the corresponding state activation vector value, and extract the set of transition weight values ​​for all state transition paths in the transition set in which the state participates. Perform aggregation calculation on the set of transition weight values ​​to generate state structure associated values. The aggregation calculation process specifically includes: summing all the transition weight values ​​corresponding to the state to obtain the cumulative weight value, and then dividing the cumulative weight value by the number of corresponding state transition paths to obtain the state structure association value; The state activation vector values ​​and the state structure associated values ​​are coupled and processed to obtain the structural coupling values. Based on the structural coupling values ​​of all states, a sorting and pruning process is performed, and the states that are in the first interval of the sorting result are determined as observable states. The state observability determination value specifically includes: performing a product calculation on the state activation vector value and the state structure association value to obtain the structure coupling value, and performing a relative sorting process on the structure coupling value and the set of structure coupling values ​​corresponding to all states. The state that is in the first interval of the sorting result is determined as an observable state. Retain all observable states and only the state transition paths connecting the observable states and their corresponding weighted transduction relationships to generate an observable state subspace corresponding to the current shared key.

[0023] In this embodiment, the establishment of the legal state transition path sequence includes: Acquire the target data and perform content parsing processing on the target data to generate a data content description that characterizes the content features of the target data; Content parsing is used to extract core feature information from target data that can stably represent the semantic structure of the data. The extracted data content description does not depend on the byte length or transmission segmentation method of the target data, but generates a unified semantic representation based on the structural composition relationship of the target data at the business level. Traverse the set of states and the set of transitions within the observable state subspace, extract all state transition paths formed by continuous connection from the start state to the end state, and generate a path description for each state transition path. The path description includes the number of states and the number of transitions contained in the state transition path. Semantic matching processing is performed based on the data content description and path description to establish a semantic mapping relationship between the data content and the state transition path, so that the state transition path that meets the data content description constraints is determined as a candidate state transition path. Semantic matching processing establishes a correspondence between data semantics and state transition path structure by mapping data content description to path structure complexity representation parameter space. This mapping process compares the projection position of semantic structure in path structure space, so that state transition paths that meet the constraints of data content description form a convergent set in structure space, thereby determining candidate state transition paths. Through this semantic mapping method based on structure projection, data semantics can be implicitly expressed in the form of path structure during transmission, avoiding the direct exposure of semantic information in the communication channel. The path selection value is generated based on the shared key, and the target state transition path is determined from the candidate state transition paths according to the path selection value. Generate a sequence of valid state transition paths corresponding to the target data according to the connection order of each state in the target state transition path.

[0024] In this embodiment, the data transmission processing includes: Obtain the sequence of valid state transition paths, and determine the execution order of the paths based on the state order in the sequence of valid state transition paths; Perform sequential segmentation processing on the target data to generate a sequence of data segments corresponding to the execution order of the path; Based on the path execution order, the current state and the next state are determined sequentially from the legal state transformation path sequence, and the current state is determined as the current state of the target data during the transmission of the current data segment; The sequential determination of the current state and the next state ensures that the transmission of each data segment is within a defined state interval. This state interval not only describes the transmission stage of the data but also serves as the state input condition for generating the session encryption key. By binding the data segment with the state interval, it can be ensured that the data encryption results under different states are independent of each other, forming a dynamic encryption structure that changes with the state evolution. Based on the shared key and the current state, a key derivation process is performed to generate a session encryption key that uniquely corresponds to the current state. The key derivation process is determined as follows: the shared key is converted into a decimal sequence value, the state number value corresponding to the current state is combined with the decimal sequence value in sequence, and the result of the sequence combination process is hashed to generate the session encryption key. The session encryption key is used to encrypt the data segment corresponding to the current state, generating an encrypted data segment. The encrypted data segment is sent to the communication receiver, and after the transmission is completed, the current state corresponding to the target data is updated to the next state to complete a state transition and encrypted transmission. The current state determination, key derivation, encryption, transmission, and state update are repeatedly executed in the order of the path execution until all states in the valid state transition path sequence complete the state transition and encrypted transmission in sequence, thus completing the data transmission processing of the target data.

[0025] In this embodiment, the establishment of the binding relationship includes: During data transmission, the current time value is obtained, and time quantization processing is performed on the current time value to generate a time slice identifier corresponding to the legal state transition path sequence. Time quantization processing divides the continuous time axis into discrete time intervals according to a fixed time granularity, so that any data transmission behavior can be mapped to a unique time slice identifier. The time granularity is determined in conjunction with the data transmission cycle and the state transition frequency to ensure that the time slice identifiers corresponding to different state transition stages do not overlap during the execution of the legal state transition path sequence. During data transmission, sequential segmentation is performed based on the shared key, and continuous key fragments are extracted from the sequential segmentation results to generate key derivation fragments corresponding to the legal state transduction path sequence. Extraction of continuous key segments is achieved by selecting segments corresponding to the current data transmission stage in the sequential segmentation structure of the shared key, so that the key-derived segments exhibit sequential characteristics that evolve with state changes during the path advancement process; During data transmission, a path access tag generation process is performed on the legal state transition path sequence. The path access tag generation process includes sequentially combining the state order and transition order in the legal state transition path sequence, and performing hash mapping on the result of the sequential combination process to generate a historical path access tag corresponding to the legal state transition path sequence. The time slice identifier, key derivation fragment, and historical path access mark are associated with the legal state transition path sequence through a record-associating process. This creates a one-to-one correspondence between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence in the same record, thereby establishing a binding relationship between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence.

[0026] In this embodiment, stopping the continued transmission of target data includes: Obtain the binding relationship corresponding to the valid state transduction path sequence, and read the time slice identifier, key derivation fragment, and historical path access mark from the binding relationship, so that the time slice identifier, key derivation fragment, and historical path access mark can be used as input data for path uniqueness verification processing; The input for path uniqueness verification processing is uniformly derived from the one-to-one correspondence records already formed in the binding relationship, so that the time slice identifier, key derivation fragment and historical path access mark are kept in a synchronized and associated state before entering the verification stage. When performing subsequent state transitions, the current state and the next state are extracted, and the next state is compared with the state at the corresponding position in the legal state transition path sequence to generate a state consistency determination result. The state consistency comparison process is used not only to determine whether the next state conforms to the order constraints of the legal state transition path sequence, but also to confirm whether the current data transmission behavior is still within the established structural path. Perform sequential combination processing based on time slice identifier, key derivation fragment and historical path access mark, and perform hash mapping processing on the result of sequential combination processing to generate path verification value; Based on the path verification value, a matching search is performed in the historical records corresponding to the binding relationship. When the same path verification value is found or the status consistency judgment result is inconsistent, the valid status transformation path sequence is judged as invalid. Historical record matching retrieval is completed by checking whether the same path exists in the existing set of binding relationships. When the retrieval result is true, it indicates that the current path structure has appeared in the historical transmission process, thus constituting path duplication. When the state consistency judgment result is inconsistent, it indicates that the current path structure has deviated from the original legal structure. Both cases indicate that the current data transmission no longer meets the structural uniqueness requirement. Therefore, the legal state transition path sequence is uniformly judged to be invalid in order to maintain the non-repeatability of the overall transmission structure. When a valid state transduction path sequence is determined to be invalid, the continued transmission of target data is stopped.

[0027] In this embodiment, the process of completing the closed-loop data transmission under shared key control includes: After the valid state transduction path sequence fails, the shared key is obtained, and hash mapping and bit-level expansion are performed on the shared key to generate a new state activation vector. Read the full state space structure of the finite state transducer, and perform state observability pruning on the full state space structure of the finite state transducer again based on the new state activation vector to obtain a new observable state subspace. Within the new observable state subspace, determine the new legitimate state transduction path sequence corresponding to the target data based on the shared key; The target data is reprocessed according to the new legal state transition path sequence, so that the target data completes state transition and encrypted transmission sequentially along the new legal state transition path sequence. While the target data has not been fully transmitted, the process of generating new state activation vectors, pruning state observability, determining the sequence of legal state transition paths, and transmitting data is repeated until the closed-loop data transmission process under shared key control is completed.

[0028] Example 1: To verify the feasibility of this invention in practice, it was applied to a regional data exchange and business collaboration environment. This environment covers the continuous data development, sharing, and cross-domain transmission needs between multiple business systems. Different systems exhibit significant differences in security control strategies, data structure organization, and communication path management mechanisms, and have long relied primarily on fixed transmission links and single session keys for data exchange. With the continuous increase in data interaction frequency and the growing complexity of business collaboration, traditional technologies have gradually revealed problems such as unavoidable path reuse, imperceptible transmission process status, and insufficient recovery capabilities under abnormal conditions. This makes it difficult to maintain the structural security boundary of data during transmission, and often requires the complete communication process to be re-established when disturbances or access conflicts occur in the communication link, thus affecting the overall continuous operation capability of the business. Based on the above operational background, a data development transmission optimization storage method based on a shared key driven finite state transducer mechanism is deployed in this environment. By performing hash mapping and bit-level expansion processing on the shared key at the beginning of communication to generate a state activation vector, a full state space structure of the finite state transducer is constructed accordingly. Then, combined with the observability pruning mechanism of structural coupling, the range of states that can participate in transmission is limited, so that data transmission is constrained within a controlled state subspace before entering actual communication. This reduces the possibility of path structure being predicted or reused from the source, laying a structural foundation for subsequent secure transmission and stable operation.

[0029] In practical applications within this operating environment, the data to be transmitted first generates a stable data content description based on its semantic content. Then, within the observable state subspace, a mapping and matching process is completed from semantics to state transition path structure. This allows data semantics to implicitly participate in transmission control through the path structure, without directly exposing semantic information in the communication channel. This enhances the security constraints of the structural layer while maintaining the original data expression method at the business layer. Subsequently, the system segments the data according to the valid state transition path sequence. Within each state interval, a session encryption key is dynamically derived based on the shared key and the current state, ensuring that the data encryption results at different stages maintain an independent evolutionary relationship. This improves the anti-replay capability and structural concealment of the communication process without altering the existing business data organization method. During continuous data transmission, the system synchronously generates time slice identifiers, key derivation fragments, and historical path access markers, establishing a strict one-to-one correspondence between these three and the valid state transition path sequence. This ensures that any subsequent state transition can complete path uniqueness verification through a unified verification entry point, guaranteeing the consistency of the entire communication structure in both the time and state dimensions. When link fluctuations, state deviations from the predetermined order, or potential duplicate path risks occur in the communication environment, the system can quickly identify path conflicts and terminate the current transmission process based on the binding relationship. Then, by regenerating the state activation vector and a new observable state subspace, the path is reconstructed, so that data transmission can be restored to a new security structure without interrupting the overall business logic, thereby forming a continuous closed-loop controlled transmission process and improving the operational resilience in complex environments.

[0030] During continuous operation, comprehensive collection and comparative analysis of system operation records across different time intervals, geographical deployment nodes, and multi-source business data types revealed that, despite prolonged communication load variations and multiple rounds of state reconstruction, the data transmission process maintained stable structural constraints, with no path reuse. Abnormal states were identified early and structural-level recovery was achieved, ensuring stable communication continuity. Further evaluation, combined with log correlation in cross-regional deployment environments, confirmed that this invention not only maintains a synergistic balance between data security control and path unpredictability in complex data exchange scenarios but also maintains adaptive adjustment capabilities of the transmission structure during continuous operation. This transforms cross-system data development transmission from a static protection-dependent approach to a dynamic closed-loop control mode constrained by state space. This verifies that the invention possesses excellent engineering feasibility, long-term operational stability, and environmental adaptability for multi-system collaborative scenarios under real-world business operation conditions. It provides a sustainable and evolving technical path for secure transmission and optimized storage during data development and lays a reliable technical foundation for its future application in broader data collaboration environments.

[0031] Table 1 Performance Comparison Report of Three Data Development and Transmission Optimization Schemes

[0032] In terms of average transmission success rate, the present invention achieves 96.6%, which is higher than traditional and improved methods. This indicates that the finite state transduction structure constraint can reduce the probability of potential failure paths in the initial stage of communication, thereby improving the overall transmission stability.

[0033] Regarding the path duplication rate, this invention reduces it to 0.2%, which is far lower than the other two methods, demonstrating the role of the triple binding verification mechanism of time slice identifier, key derivation fragment and historical path access mark in controlling structural uniqueness.

[0034] Regarding the time required for abnormal recovery, this invention can complete path reconstruction and restore communication in just 3.1 seconds, which is significantly better than the comparison method that requires re-establishing the session or renegotiating the key, verifying the engineering efficiency advantage of the fast state subspace reconstruction mechanism.

[0035] In terms of data integrity and resistance to replay attacks, this invention achieves near-complete protection. This is because the state-bound dynamic key derivation mechanism enables different transmission stages to form independent encryption structures, making it difficult for attackers to deduce the overall key evolution relationship through local information.

[0036] The continuous stable operation time has been increased to 168 hours, indicating that the present invention has a significant advantage in long-term operational robustness. This advantage comes from the combined effect of structural path unique control and closed-loop self-recovery mechanism, which effectively suppresses the cumulative spread of safety risks over time.

[0037] Based on all experimental data, it can be confirmed that the present invention has achieved systematic improvements in security, continuity, anti-attack capability, and long-term stable operation capability, fully demonstrating its practical engineering application value in complex data development and cross-system transmission environments.

[0038] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data development, transmission, and storage optimization method based on big data processing, characterized in that, Includes the following steps: Obtain the original key data shared by both communicating parties, and perform hash mapping and bit-level expansion processing to generate a state activation vector; Construct the full state space structure of the finite state transducer; Based on the correlation between the state activation vector and the transition weights in the full state space structure of the finite state transducer, a state observability pruning process of structural coupling is performed to obtain the observable state subspace. Establish a semantic mapping relationship between data content and state transition paths within the observable state subspace, and determine the corresponding legal state transition path sequence; Perform data transmission processing on the target data according to the legal state transduction path sequence; During data transmission, time slice identifiers, key derivation fragments, and historical path access markers are recorded, and a binding relationship is established between them and the legitimate state transition path sequence. Based on the binding relationship, the path uniqueness verification process is performed. When the path is duplicated or the state is conflicted, the valid state is deduced to be invalid and the data transmission is stopped. After the valid state transduction path sequence fails, a new state activation vector is regenerated, and state observability pruning is performed until the data closed-loop transmission process under shared key control is completed.

2. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The generated state activation vector corresponding to the shared key includes: It receives the original key data provided by the communication initiator and the communication receiver respectively, performs consistency verification on the original key data, and uses the original key data as input when the consistency verification passes. The original key data that has passed the consistency check is input into the cryptographic hash function to generate a fixed-length hash digest; Perform bit-level expansion on the hash digest to convert the hash digest into a continuous bit sequence in binary bit order; Perform sequential segmentation on the continuous bit sequence according to the segment length to generate multiple bit segments; Numerical mapping is performed on each segment to convert each segment into a decimal value, and the activation value corresponding to each segment is determined based on the correspondence between the decimal value and the threshold interval. Arrange the activation values ​​corresponding to each segment in the order of the segments in the continuous bit sequence to generate an activation value sequence; Perform sequential combination processing on the activation value sequence to generate a state activation vector corresponding to the shared key.

3. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The generation of the full state space structure of the finite state transducer includes: Multiple states are determined based on the phase division rules of the data transmission process based on shared keys, and a unique corresponding state identifier is generated for each state. All state identifiers are then collected in order to form a state set. In the data transmission process based on shared keys, the sequential constraints of the stages determine the reachability relationships between each state, and each state is sequentially associated with at least one successor state according to the reachability relationships to generate multiple state transition paths, which are then aggregated to form a transition set. Establish a weighted transduction relationship for each state transition path in the transition set; The transition weights for each state transition path are calculated and processed uniformly. By structurally associating and integrating the state set, transition set, and weighted transduction relationship, a full state space structure of a finite state transducer is generated.

4. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The generation of the observable state subspace includes: Read the state set, transition set, and weighted transduction relation in the full state space structure of the finite state transducer, and establish a one-to-one sequential correspondence between the state set and the state activation vector according to the arrangement order of the state set; For each state in the state set, extract the corresponding state activation vector value, and extract the set of transition weight values ​​for all state transition paths in the transition set in which the state participates. Perform aggregation calculation on the set of transition weight values ​​to generate state structure associated values. The state activation vector values ​​and the state structure associated values ​​are coupled and processed to obtain the structural coupling values. Based on the structural coupling values ​​of all states, a sorting and pruning process is performed, and the states that are in the first interval of the sorting result are determined as observable states. Retain all observable states and only the state transition paths connecting the observable states and their corresponding weighted transduction relationships to generate an observable state subspace corresponding to the current shared key.

5. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The establishment of the legal state transition path sequence includes: Acquire the target data and perform content parsing processing on the target data to generate a data content description; Traverse the set of states and the set of transitions within the observable state subspace, extract all state transition paths formed by continuous connection from the start state to the end state, and generate a path description for each state transition path. The path description includes the number of states and the number of transitions contained in the state transition path. Semantic matching processing is performed based on the data content description and path description to establish a semantic mapping relationship between the data content and the state transition path, so that the state transition path that meets the data content description constraints is determined as a candidate state transition path. The path selection value is generated based on the shared key, and the target state transition path is determined from the candidate state transition paths according to the path selection value. Generate a sequence of valid state transition paths corresponding to the target data according to the connection order of each state in the target state transition path.

6. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The data transmission processing includes: Obtain the sequence of valid state transition paths, and determine the execution order of the paths based on the state order in the sequence of valid state transition paths; Perform sequential segmentation processing on the target data to generate a sequence of data segments corresponding to the execution order of the path; Based on the path execution order, the current state and the next state are determined sequentially from the legal state transformation path sequence, and the current state is determined as the current state of the target data during the transmission of the current data segment; The sequential determination of the current state and the next state ensures that the transmission of each data segment is within a defined state interval. This state interval not only describes the transmission stage of the data but also serves as the state input condition for generating the session encryption key. By binding the data segment with the state interval, it can be ensured that the data encryption results under different states are independent of each other, forming a dynamic encryption structure that changes with the state evolution. Based on the shared key and the current state, perform key derivation processing to generate a session encryption key that uniquely corresponds to the current state; The session encryption key is used to encrypt the data segment corresponding to the current state, generating an encrypted data segment. The encrypted data segment is sent to the communication receiver, and after the transmission is completed, the current state corresponding to the target data is updated to the next state to complete a state transition and encrypted transmission. The current state determination, key derivation, encryption, transmission, and state update are repeatedly executed in the order of the path execution until all states in the valid state transition path sequence complete the state transition and encrypted transmission in sequence, thus completing the data transmission processing of the target data.

7. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The establishment of the binding relationship includes: During data transmission, the current time value is obtained, and time quantization processing is performed on the current time value to generate a time slice identifier corresponding to the legal state transition path sequence. During data transmission, sequential segmentation is performed based on the shared key, and continuous key fragments are extracted from the sequential segmentation results to generate key derivation fragments corresponding to the legal state transduction path sequence. During data transmission, path access marker generation is performed on the valid state transformation path sequence, and hash mapping is performed on the sequential combination processing result to generate historical path access markers corresponding to the valid state transformation path sequence. The time slice identifier, key derivation fragment, and historical path access mark are associated with the legal state transition path sequence through a record-associating process. This creates a one-to-one correspondence between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence in the same record, thereby establishing a binding relationship between the time slice identifier, key derivation fragment, and historical path access mark and the legal state transition path sequence.

8. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The cessation of continued data transmission of the target data includes: Obtain the binding relationship corresponding to the legal state transduction path sequence, and read the time slice identifier, key derivation fragment, and historical path access mark from the binding relationship; When performing subsequent state transitions, the current state and the next state are extracted, and the next state is compared with the state at the corresponding position in the legal state transition path sequence to generate a state consistency determination result. Perform sequential combination processing based on time slice identifier, key derivation fragment and historical path access mark, and perform hash mapping processing on the result of sequential combination processing to generate path verification value; Based on the path verification value, a matching search is performed in the historical records corresponding to the binding relationship. When the same path verification value is found or the status consistency judgment result is inconsistent, the valid status transformation path sequence is judged as invalid. When a valid state transduction path sequence is determined to be invalid, the continued transmission of target data is stopped.

9. The data development, transmission, optimization, and storage method based on big data processing according to claim 1, characterized in that, The process of completing the closed-loop data transmission under shared key control includes: After the valid state transduction path sequence fails, the shared key is obtained, and hash mapping and bit-level expansion are performed on the shared key to generate a new state activation vector. Read the full state space structure of the finite state transducer, and perform state observability pruning on the full state space structure of the finite state transducer again based on the new state activation vector to obtain a new observable state subspace. Within the new observable state subspace, determine the new legitimate state transduction path sequence corresponding to the target data based on the shared key; The data transmission process is re-executed on the target data according to the new legal state transduction path sequence; While the target data has not been fully transmitted, the process of generating new state activation vectors, pruning state observability, determining the sequence of legal state transition paths, and transmitting data is repeated until the closed-loop data transmission process under shared key control is completed.

Images