Data storage method and system for business information consultation system

Abstract

The invention relates to the technical field of business information consultation, and discloses a data storage method of a business information consultation system, comprising: S1, data acquisition and analysis; s2, character frequency statistics; s3, character combination generation; s4, influence factor calculation; s5, generating a character list; s6, coding compression; and S7, carrying out distributed storage. According to the data storage method and system of the business information consultation system, after a data acquisition module analyzes dialogue data to generate a character sequence, a frequency analysis module counts a first probability, and a character combination processing module extracts a high-frequency three-character combination by using a sliding window and combines the high-frequency three-character combination into a multi-level set; the influence factor calculation module adjusts a second probability in combination with co-occurrence and global frequency, the list generation module generates a character list according to a front probability sequence, and the coding compression module dynamically adjusts the sequence by moving to a front-end algorithm, so that the coding quantity is reduced, the compression efficiency is improved, and the occupied storage space is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of business information consulting, and in particular to a data storage method and system for a business information consulting system. Background Art

[0002] With the development of the Internet, more and more companies and institutions are providing business information consulting services through online platforms. During the consultation process, the system will generate a large number of conversation records, which need to be stored efficiently to reduce server costs. Among the commonly used data compression methods, an algorithm based on dynamically adjusting the order of characters is widely used. The core of this algorithm is to generate a character list by counting the frequency of character occurrences, and replace the original characters with the index values ​​in the list, thereby reducing storage space.

[0003] However, there are two obvious problems with existing methods:

[0004] The character list generation method is simple: the algorithm relies solely on the frequency of character occurrence, which may result in high-frequency characters being assigned larger index values. For example, if "hello" is a high-frequency word, it may appear at the end of the list due to the frequency sorting rule, resulting in a larger index value, which in turn increases the amount of encoded data.

[0005] Ignoring contextual relevance: Existing methods typically count individual characters without considering the relationship between them. For example, fixed combinations such as "hello" and "thank you" frequently appear in conversations, but traditional methods count "you," "good," and "thank you" separately, causing the statistical results to deviate from actual usage scenarios and reducing compression efficiency.

[0006] Therefore, a data storage method and system for a business information consulting system are proposed. Summary of the Invention

[0007] (1) Technical problems solved

[0008] In response to the shortcomings of the existing technology, the present invention provides a data storage method and system for a business information consulting system. By introducing multi-level character sets and dynamic influencing factor analysis, the character list generation process moved to the front-end algorithm is optimized, thereby improving data compression efficiency and reducing storage space usage, solving the problem of the existing technology of a single character list generation method and ignoring contextual relevance.

[0009] (2) Technical solution

[0010] To achieve the above-mentioned purpose of optimizing the character list generation process of the front-end algorithm by introducing multi-level character sets and dynamic impact factor analysis, thereby improving data compression efficiency and reducing storage space usage, the present invention provides the following technical solution: A data storage method for a business information consulting system, comprising:

[0011] S1. Data acquisition and parsing: Receive conversation data generated by the business information consulting system in real time through the API interface and parse it into character sequences, where each character sequence consists of letters, numbers, punctuation marks, and special symbols;

[0012] S2. Character frequency statistics: Perform frequency statistics on each character unit in the character sequence and calculate the first probability of each character unit. The first probability is determined by the ratio of the number of times the character unit appears in the character sequence to the total number of characters.

[0013] S3, character combination generation: Use a sliding window algorithm to traverse the character sequence, extract consecutive character units of length 3 as the initial character combination, and count the frequency of each initial character combination; filter high-frequency character combinations based on the frequency threshold, merge adjacent high-frequency character combinations, and generate a multi-level character set;

[0014] S4. Impact factor calculation: For each character unit in the multi-level character set, its co-occurrence frequency in the character combination is counted, and the impact factor of each character unit is calculated based on the length of the character combination and the distribution characteristics of the co-occurrence frequency. The impact factor is used to adjust the second probability of the character unit;

[0015] S5. Character list generation: Multiply the first probability of the character unit by the second probability to obtain the pre-probability of the character unit, sort the character units from large to small according to the pre-probability, and generate a character list for encoding;

[0016] S6, encoding compression: Execute the move-to-front algorithm based on the character list, perform index lookup on each character unit in the character sequence, and dynamically adjust the order of the character list; perform secondary compression on the index value sequence through context-adaptive binary arithmetic coding to generate compressed data;

[0017] S7. Distributed storage: Compressed data is divided into fixed-size data blocks and distributedly stored through the Hadoop distributed file system, where each data block is stored on three different nodes for redundant backup and metadata information is recorded for fast retrieval.

[0018] Preferably, the sliding step size of the sliding window algorithm is 1 and the window length is 3, ensuring that each character unit forms a combination with the two character units before and after it, thereby covering all possible consecutive three-character combinations and screening out high-frequency character combinations through backtracking.

[0019] Preferably, the frequency threshold is set based on the ratio of the frequency of occurrence of the character combination to the total length of the character sequence. When the frequency of the character combination exceeds 0.01 of the total length, it is marked as a high-frequency character combination, and adjacent character combinations with a merging possibility greater than 0.4 are screened through the merging possibility formula.

[0020] Preferably, the calculation of the impact factor comprises the following steps:

[0021] Counting the co-occurrence frequency of a character unit with other character units in a multi-level character set;

[0022] Calculate the ratio of the co-occurrence frequency of the character unit to the character combination length to obtain the local influence coefficient of the character unit;

[0023] The local influence coefficient is combined with the global frequency of the character unit, and the final influence factor is generated through a weighted average algorithm.

[0024] Preferably, the dynamic adjustment of the move-to-frontend algorithm comprises the following steps:

[0025] The position of each character unit in the character list is updated in real time. When a character unit is accessed, it is moved to the front of the character list.

[0026] Record the update status of the character list and synchronously update the version information of the character list during the encoding process to ensure the consistency of the list during decoding.

[0027] Preferably, the context-adaptive binary arithmetic coding adopts a dynamic probability model, and improves the compression efficiency of the index value sequence by updating the probability model every 1000 index values.

[0028] A data storage system for a business information consulting system, comprising:

[0029] Data acquisition module: used to receive and parse consultation dialogue data and generate character sequences;

[0030] Frequency analysis module: used to calculate the first probability of character units and generate high-frequency character combinations;

[0031] Character combination processing module: used to merge high-frequency character combinations, generate multi-level character sets, and calculate impact factors;

[0032] List generation module: used to generate a character list according to the first probability and the second probability;

[0033] Coding and compression module: used to execute the mobile-to-frontend algorithm and context-adaptive binary arithmetic coding to generate compressed data;

[0034] Storage management module: used to store compressed data in the Hadoop distributed file system and manage metadata information.

[0035] Preferably, the storage management module further includes:

[0036] Data sharding unit: used to divide compressed data into blocks of size, with each block size being 128MB.

[0037] Redundant storage unit: used to replicate each data block to three different nodes in the Hadoop distributed file system cluster to achieve data redundancy;

[0038] Metadata management unit: used to record the storage location, version information, and access permissions of each data block, and centrally manage it through the NameNode node.

[0039] Preferably, the frequency analysis module further includes:

[0040] Dynamic adjustment unit: used to update the character frequency and the possibility of merging character combinations based on real-time consultation dialogue data;

[0041] Machine Learning Unit: Used to train character combination generation models based on historical data and optimize the construction strategy of multi-level character sets.

[0042] Preferably, the coding compression module adopts hardware acceleration technology to achieve parallel processing of the mobile front-end algorithm and context-adaptive binary arithmetic coding through a field programmable gate array to improve the speed and efficiency of data compression.

[0043] (3) Beneficial effects

[0044] Compared with the prior art, the present invention provides a data storage method and system for a business information consulting system, which has the following beneficial effects:

[0045] 1. The data storage method and system of the business information consulting system are as follows: after the data acquisition module parses the conversation data to generate a character sequence, the frequency analysis module calculates the first probability of the character unit, and the character combination processing module extracts high-frequency continuous three-character combinations through a sliding window algorithm and merges them to generate a multi-level character set; the influence factor calculation module combines the co-occurrence frequency and the global frequency to generate the influence factor and adjust the second probability; the list generation module multiplies the first probability by the second probability to obtain the leading probability, and generates a character list by sorting the leading probability; the encoding compression module dynamically adjusts the list order through the move to the front algorithm to ensure that high-frequency character units are always at the front of the list and are assigned smaller index values, thereby reducing the amount of encoded data and improving compression efficiency.

[0046] 2. The data storage method and system of the business information consulting system are as follows: the character combination generation module uses a sliding window algorithm to extract continuous three-character combinations, and screens high-frequency combinations through a merging possibility formula to generate a multi-level character set; the influence factor calculation module counts the co-occurrence frequency of character units in the combination, combines the local influence coefficient with the global frequency to generate an influence factor, and optimizes the second probability; the list generation module merges the first probability and the second probability into a leading probability, and prioritizes character units related to high-frequency combinations; the encoding compression module executes a move-to-front-end algorithm based on a dynamic character list to ensure that character units with strong contextual relevance occupy smaller index values ​​when encoding, thereby enhancing context sensitivity, significantly improving compression efficiency, and reducing storage space occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the data storage method of the business information consulting system of the present invention;

[0048] Figure 2 This is a diagram of the data storage system architecture of the business information consulting system of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] See also Figure 1-2 , a data storage method for a business information consulting system, comprising:

[0051] S1. Data acquisition and parsing: Receive conversation data generated by the business information consulting system in real time through the API interface and parse it into character sequences, where each character sequence consists of letters, numbers, punctuation marks, and special symbols;

[0052] S2. Character frequency statistics: Perform frequency statistics on each character unit in the character sequence and calculate the first probability of each character unit. The first probability is determined by the ratio of the number of times the character unit appears in the character sequence to the total number of characters.

[0053] S3, character combination generation: Use a sliding window algorithm to traverse the character sequence, extract consecutive character units of length 3 as the initial character combination, and count the frequency of each initial character combination; filter high-frequency character combinations based on the frequency threshold, merge adjacent high-frequency character combinations, and generate a multi-level character set;

[0054] S4. Impact factor calculation: For each character unit in the multi-level character set, its co-occurrence frequency in the character combination is counted, and the impact factor of each character unit is calculated based on the length of the character combination and the distribution characteristics of the co-occurrence frequency. The impact factor is used to adjust the second probability of the character unit;

[0055] S5. Character list generation: Multiply the first probability of the character unit by the second probability to obtain the pre-probability of the character unit, sort the character units from large to small according to the pre-probability, and generate a character list for encoding;

[0056] S6, encoding compression: Execute the move-to-front algorithm based on the character list, perform index lookup on each character unit in the character sequence, and dynamically adjust the order of the character list; perform secondary compression on the index value sequence through context-adaptive binary arithmetic coding to generate compressed data;

[0057] S7. Distributed storage: Compressed data is divided into fixed-size data blocks and distributedly stored through the Hadoop distributed file system, where each data block is stored on three different nodes for redundant backup and metadata information is recorded for fast retrieval.

[0058] A data storage system for a business information consulting system, comprising:

[0059] Data acquisition module: used to receive and parse consultation dialogue data and generate character sequences;

[0060] Frequency analysis module: used to calculate the first probability of character units and generate high-frequency character combinations;

[0061] Character combination processing module: used to merge high-frequency character combinations, generate multi-level character sets, and calculate impact factors;

[0062] List generation module: used to generate a character list according to the first probability and the second probability;

[0063] Coding and compression module: used to execute the mobile-to-frontend algorithm and context-adaptive binary arithmetic coding to generate compressed data;

[0064] Storage management module: used to store compressed data in the Hadoop distributed file system and manage metadata information.

[0065] Example 1:

[0066] This embodiment is designed for the core processes of data acquisition and parsing, and character frequency statistics, focusing on solving the problems of efficient parsing of raw conversation data and accurate modeling of character unit frequency distribution. By combining a memory buffer mechanism and a dynamic hash table expansion strategy, it ensures that the system maintains stability in high-throughput scenarios.

[0067] The data acquisition module receives the conversation data of the business information consultation system through the RESTful API interface. The interface supports HTTP / HTTPS protocol transmission, and the maximum concurrent processing capacity is 1000 conversation records per second; the original data is encapsulated in JSON format and contains fields such as timestamp, user identification, conversation content, etc.

[0068] The parsing module uses the regular expression library of Python to split the conversation content into characters, splitting letters, numbers, punctuation marks, and special symbols into independent character units; for example, the conversation content "您好!感谢咨询。" will be parsed into ["您","好","!","感","谢","咨","询","。"]; the parsed character sequence is stored in the memory buffer in UTF-8 encoding format, and the buffer capacity is dynamically adjusted according to the system load, with a maximum of no more than 1GB.

[0069] The character frequency statistics module uses a hash table structure to record the occurrence times of each character unit. The key of the hash table is the Unicode code value of the character unit, and the value is its cumulative occurrence times.

[0070] The initial capacity of the hash table is 1024, and it will automatically expand when the load factor exceeds 0.75; for example, if the current hash table has stored 800 character units, the expansion operation will be triggered and the capacity will be doubled to 2048.

[0071] The calculation of the first probability is obtained by dividing the occurrence times of each character unit by the total number of characters, and the result is reserved to 6 decimal places; for example, if the character "您" appears 500 times and the total number of characters is 100,000, its first probability is 0.005000.

[0072] After the statistics are completed, the data in the hash table is sorted by the Unicode code value of the character unit to generate an initial frequency table; this table is persistently stored through the memory mapping file mechanism to support high-frequency reading operations.

[0073] Example 2:

[0074] This embodiment focuses on the implementation of character combination generation and influence factor calculation, aiming to improve the compression efficiency through context relevance modeling; the design of the sliding window algorithm and the combination possibility formula needs to consider both computational efficiency and combination quality, and at the same time, the calculation of the influence factor needs to balance local and global features.

[0075] The character combination generation module is implemented using the sliding window algorithm. The length of the sliding window is fixed at 3, and the step size is 1. The window slides character by character on the character sequence. For example, for the character sequence ["您","好","!","感","谢","咨","询","。"], the sliding window will generate combinations such as ["您","好","!"],["好","!","感"],["!","感","谢"], etc.

[0076] After all combinations are counted, the system filters out the high-frequency combinations whose frequency exceeds 0.01% of the total number of characters. For example, if the total number of characters is 100,000, the frequency threshold is 1000 times. The filtered high-frequency combinations are judged whether to be merged through the merging possibility formula:

[0077] If the length of the overlapping part of adjacent combinations is greater than or equal to 2 and the frequency of the combined combination is higher than 80% of the frequency of the individual combinations, the merging operation is performed.

[0078] For example, the overlapping part of the combinations ["您","好","!"] and ["好","!","感"] is ["好","!"]. The frequency of the combined combination ["您","好","!","感"] needs to satisfy freq([您,好,!,感])>0.8*(freq([您,好,!])+freq([好,!,感])). The combined combinations form a multi-level character set, and the length of the highest-level character set can reach 10 character units.

[0079] The influence factor calculation module includes three sub-modules: co-occurrence frequency statistics, local influence coefficient calculation, and global influence factor generation.

[0080] Co-occurrence frequency statistics traverses the multi-level character set and records the co-occurrence times of each character unit with other character units. For example, the character "您" co-occurs with "好" 500 times, and "好" co-occurs with "!" 300 times. Local influence coefficient calculation divides the co-occurrence frequency by the average length of the character combination. For example, if the co-occurrence frequency of the character "您" is 500 and the average combination length is 3, the local influence coefficient is 166.6667.

[0081] Global influence factor generation uses the weighted average algorithm, with the weight being the global frequency of the character unit. The finally generated influence factor ranges from 0 to 1 and is used to adjust the calculation of the second probability.

[0082] Example three:

[0083] This example focuses on the implementation of character list generation and coding compression, and ensures the real-time performance and consistency of the dynamic character list through the move-to-front algorithm and version synchronization mechanism. The calculation of the prior probability needs to comprehensively consider the independent frequency and context relevance of the character units, and the selection of the sorting algorithm needs to balance efficiency and stability.

[0084] The working process of the character list generation module includes:

[0085] First, multiply the first probability and the second probability to obtain the pre - probability. The calculation of the pre - probability takes into account the independent frequency and context relevance of character units; for example, if the first probability of the character "您" is 0.005 and the second probability is 0.8, then the pre - probability is 0.004.

[0086] All character units are sorted from largest to smallest according to the pre - probability to generate a dynamic character list; the quick - sort algorithm is used in the sorting process, and the time complexity is O(nlog n); for example, if the character list contains 4096 character units, about 10000 comparison operations are required in the sorting process.

[0087] The generated character list is stored in binary format. Each character unit occupies 2 bytes. The first 2 bytes represent the index value, and the last 2 bytes represent the Unicode encoding of the character unit.

[0088] The encoding compression module is implemented based on the move - to - front algorithm; when a character unit is accessed, its position in the character list is immediately moved to the front; for example, if the character "您" is at the 100th position, after access, it is moved to the 1st position, and the remaining character units are shifted backward in turn.

[0089] During the dynamic update process, the system maintains an access log to record the timestamp and character unit information of each move operation; the version synchronization mechanism is implemented through hash verification. After each update of the character list, the hash value is calculated, and the encoding module verifies the hash value consistency before compression; if inconsistency is found, the encoding module will restore the character list version from the nearest backup point.

[0090] Example 4:

[0091] This example is designed for the implementation of context - adaptive binary arithmetic coding and hardware acceleration. The coding efficiency is improved through a dynamic probability model and an FPGA parallel processing architecture; the design of the interval divider needs to balance accuracy and computational complexity, and the resource allocation of the hardware acceleration module needs to optimize the number of parallel channels.

[0092] The context - adaptive binary arithmetic coding module adopts a dynamic probability model and updates the probability distribution every 1000 index values.

[0093] The initial probability distribution is generated based on the pre - probability of the character list, and subsequent updates are adjusted according to the actual coding data; for example, if the occurrence frequency of the index value "1" is higher than expected, its probability weight is increased; during the coding process, the system maintains an interval divider to divide the current interval [low, high) into sub - intervals according to the probability distribution.

[0094] During decoding, the original index value sequence is restored through inverse calculation; this module uses FPGA hardware acceleration, mapping the core operation unit to the programmable logic gate array to achieve parallel processing; the hardware acceleration module contains dedicated multipliers and adders, which can process the data streams of 8 encoding channels simultaneously.

[0095] The hardware acceleration module is based on the Xilinx Virtex UltraScale+ series chip, containing 1 million logic units and 40 DSP modules; the firmware implements parallel processing of mobile front-end algorithms and context-adaptive coding, with each processing channel occupying 8 DSP modules.

[0096] The hardware acceleration module is connected to the main control server via the PCIe 4.0 interface, with a data transmission rate of up to 10Gbps.

[0097] The temperature monitoring module detects the FPGA chip temperature in real time and automatically reduces the operation frequency when the temperature exceeds 85°C.

[0098] The power management module supports dynamic voltage regulation, reducing power consumption to below 5W in idle state and 40W at full load.

[0099] Embodiment 5:

[0100] This embodiment focuses on the implementation of distributed storage and redundancy checking, ensuring data reliability through the Hadoop distributed file system and RAID 5 technology. Metadata management must support version rollback and fine-grained access control, and the redundancy checking mechanism must balance efficiency and fault tolerance.

[0101] The distributed storage module is implemented based on the Hadoop distributed file system; the data sharding unit divides the compressed data into 128MB blocks, and each data block is stored on different nodes in the cluster through a three-copy strategy.

[0102] The redundant storage unit uses RAID 5 technology to store parity information in three nodes.

[0103] The metadata management unit centrally manages the storage location, version information, and access permissions of data blocks through the NameNode node; metadata includes fields such as data block hash value, creation time, modification time, and number of accesses.

[0104] When a data block needs to be updated, the system generates a new version and retains the historical version. The version rollback operation is achieved by reading the metadata of the old version.

[0105] The redundancy check mechanism is implemented through periodic checksum calculation. The system performs a hash check on all data blocks once an hour to detect data integrity.

[0106] If a data block is found to be damaged, the system automatically recovers from other copies. The fault recovery process is divided into two stages:

[0107] The first phase reads data blocks from the remaining replicas.

[0108] The second phase writes the new copy.

[0109] The recovery time is within 30 seconds, ensuring service continuity; the metadata management unit supports fine-grained access control and limits data access rights of different users through ACL (access control list).

[0110] Example 6:

[0111] This example focuses on the implementation of dynamic adjustment and machine learning optimization, improving the system's adaptability through real-time monitoring and reinforcement learning strategies. The training of the LSTM neural network requires a combination of historical conversation data and simulated test samples, and the design of the reward function must balance compression rate and computational complexity.

[0112] The dynamic adjustment unit includes a real-time monitoring module and an adaptive optimization module; the real-time monitoring module collects character frequency change data every second, and triggers adaptive optimization when it detects that the character combination frequency fluctuates by more than 15%.

[0113] The optimization process includes recalculating the impact factor, adjusting the character list sorting, and updating the probability model; the machine learning unit predicts the character combination generation trend based on the LSTM neural network model, and the model training data set contains 100,000 historical conversation records.

[0114] The Adam optimizer is used during training, with a learning rate of 0.001 and a batch size of 256. The optimized multi-level character set construction strategy is implemented using a reinforcement learning algorithm, and the reward function comprehensively considers both compression rate and computational complexity.

[0115] The reinforcement learning algorithm adopts the Q-learning strategy. The state space is defined as the previous probability distribution of the current character list, and the action space is the adjustment operation of the character list (such as moving to the front, merging and combining, etc.). The reward function is designed to be 0.7 times the compression rate plus 0.3 times the computational complexity.

[0116] During the training process, the system provides test samples through a simulated conversation data generator to evaluate the compression efficiency under different strategies.

[0117] The optimized model is deployed to the edge device, exported in ONNX format, and run locally to reduce cloud dependency. The model inference latency is less than 50ms, meeting real-time requirements.

[0118] In summary, the data storage method and system of the business information consulting system are as follows: after the data acquisition module parses the conversation data to generate a character sequence, the frequency analysis module counts the first probability of the character unit, and the character combination processing module extracts high-frequency continuous three-character combinations through a sliding window algorithm and merges them to generate a multi-level character set; the impact factor calculation module combines the co-occurrence frequency and the global frequency to generate an impact factor and adjust the second probability; the list generation module multiplies the first probability by the second probability to obtain the leading probability, and generates a character list by sorting the leading probability; the encoding compression module dynamically adjusts the list order through the move to the front algorithm to ensure that the high-frequency character unit is always at the front of the list and is assigned a smaller index value, thereby reducing the amount of encoded data and improving compression efficiency.

[0119] In addition, the data storage method and system of the business information consulting system are as follows: the character combination generation module adopts a sliding window algorithm to extract continuous three-character combinations, and screens high-frequency combinations through a merging possibility formula to generate a multi-level character set; the influence factor calculation module counts the co-occurrence frequency of character units in the combination, combines the local influence coefficient with the global frequency to generate an influence factor, and optimizes the second probability; the list generation module merges the first probability and the second probability into a leading probability, and prioritizes character units related to high-frequency combinations; the encoding compression module executes a move-to-front-end algorithm based on a dynamic character list, ensuring that character units with strong context relevance occupy smaller index values ​​when encoding, thereby improving context sensitivity, significantly improving compression efficiency, and reducing storage space occupancy, thereby solving the problem that the existing technology has a single character list generation method and ignores context relevance.

[0120] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the existing technology. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements to this system. The improvements to this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0121] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data storage method for a business information consulting system, characterized in that: include: S1. Data acquisition and parsing: Receive conversation data generated by the business information consulting system in real time through the API interface and parse it into character sequences, where each character sequence consists of letters, numbers, punctuation marks, and special symbols; S2. Character frequency statistics: Perform frequency statistics on each character unit in the character sequence and calculate the first probability of each character unit. The first probability is determined by the ratio of the number of times the character unit appears in the character sequence to the total number of characters. S3, character combination generation: Use the sliding window algorithm to traverse the character sequence, extract consecutive character units of length 3 as the initial character combination, and count the frequency of each initial character combination; Filter high-frequency character combinations according to the frequency threshold, merge adjacent high-frequency character combinations, and generate a multi-level character set; S4. Impact factor calculation: For each character unit in the multi-level character set, its co-occurrence frequency in the character combination is counted, and the impact factor of each character unit is calculated based on the length of the character combination and the distribution characteristics of the co-occurrence frequency. The impact factor is used to adjust the second probability of the character unit; S5. Character list generation: Multiply the first probability of the character unit by the second probability to obtain the pre-probability of the character unit, sort the character units from large to small according to the pre-probability, and generate a character list for encoding; S6, encoding compression: Execute the move-to-front algorithm based on the character list, perform index lookup on each character unit in the character sequence, and dynamically adjust the order of the character list; Performing secondary compression on the index value sequence through context-adaptive binary arithmetic coding to generate compressed data; S7. Distributed storage: Compressed data is divided into fixed-size data blocks and distributedly stored through the Hadoop distributed file system, where each data block is stored on three different nodes for redundant backup and metadata information is recorded for fast retrieval.

2. The data storage method of a business information consulting system according to claim 1, characterized in that: The sliding step size of the sliding window algorithm is 1 and the window length is 3, ensuring that each character unit forms a combination with the two character units before and after it, thereby covering all possible consecutive three-character combinations and screening out high-frequency character combinations through the backtracking method.

3. The data storage method of a business information consulting system according to claim 1, characterized in that: The frequency threshold is set based on the ratio of the frequency of occurrence of a character combination to the total length of the character sequence. When the frequency of a character combination exceeds 0.01 of the total length, it is marked as a high-frequency character combination, and adjacent character combinations with a merging possibility greater than 0.4 are screened using the merging possibility formula.

4. The data storage method of a business information consulting system according to claim 1, characterized in that: The calculation of the impact factor includes the following steps: Counting the co-occurrence frequency of a character unit with other character units in a multi-level character set; Calculate the ratio of the co-occurrence frequency of the character unit to the character combination length to obtain the local influence coefficient of the character unit; The local influence coefficient is combined with the global frequency of the character unit, and the final influence factor is generated through a weighted average algorithm.

5. The data storage method of a business information consulting system according to claim 1, characterized in that: The dynamic adjustment of the move-to-frontend algorithm includes the following steps: The position of each character unit in the character list is updated in real time. When a character unit is accessed, it is moved to the front of the character list. Record the update status of the character list and synchronously update the version information of the character list during the encoding process to ensure the consistency of the list during decoding.

6. The data storage method of a business information consulting system according to claim 1, characterized in that: The context-adaptive binary arithmetic coding adopts a dynamic probability model and improves the compression efficiency of the index value sequence by updating the probability model every 1000 index values.

7. A data storage system for a business information consulting system, characterized in that: include: Data acquisition module: used to receive and parse consultation dialogue data and generate character sequences; Frequency analysis module: used to calculate the first probability of character units and generate high-frequency character combinations; Character combination processing module: used to merge high-frequency character combinations, generate multi-level character sets, and calculate impact factors; List generation module: used to generate a character list according to the first probability and the second probability; Coding and compression module: used to execute the mobile-to-frontend algorithm and context-adaptive binary arithmetic coding to generate compressed data; Storage management module: used to store compressed data in the Hadoop distributed file system and manage metadata information.

8. The data storage system of a business information consulting system according to claim 7, characterized in that: The storage management module further includes: Data sharding unit: used to divide compressed data into blocks of size, with each block size being 128MB. Redundant storage unit: used to replicate each data block to three different nodes in the Hadoop distributed file system cluster to achieve data redundancy; Metadata management unit: used to record the storage location, version information, and access permissions of each data block, and centrally manage it through the NameNode node.

9. The data storage system of a business information consulting system according to claim 7, characterized in that: The frequency analysis module further includes: Dynamic adjustment unit: used to update the character frequency and the possibility of merging character combinations based on real-time consultation dialogue data; Machine Learning Unit: Used to train character combination generation models based on historical data and optimize the construction strategy of multi-level character sets.

10. The data storage system of a business information consulting system according to claim 7, characterized in that: The coding compression module adopts hardware acceleration technology and realizes parallel processing of mobile front-end algorithm and context-adaptive binary arithmetic coding through field programmable gate array to improve the speed and efficiency of data compression.

Images