An ingredient order data storage method and system

Abstract

The present application relates to the technical field of data processing, and more particularly to a kind of food material order data storage method and system.The method comprises the following steps: collecting order data to be compressed, obtaining the initial variation degree of each field according to the type of data in each field and the discrete degree of data value, correcting the initial variation degree of each field according to the repeatability of data arrangement in each field and the variation amplitude between data, and obtaining the final variation degree of each field;According to the final variation degree of each field, the sliding window size of each field is adaptively obtained, and the data in each field is individually compressed and stored according to the sliding window size of each field, which improves the accuracy of temperature control in the reaction kettle.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for storing food order data. Background Technology

[0002] In industries such as food supply chain and fresh food e-commerce, massive amounts of food order data are generated daily. This data is usually stored in a structured form, containing multiple fields such as order ID, product name, quantity, and unit price. It needs to be stored for a long time to support key business operations such as inventory management and market analysis. As the data volume grows exponentially, the pressure of storage costs is becoming increasingly prominent.

[0003] To address this issue, the industry typically employs data compression algorithms to reduce storage size. The LZ77 compression algorithm, due to its fast compression and decompression speeds, is a widely used technique. Its core idea is to use a "sliding window" to find recurring strings in the data stream and replace them with a short pointer to the previous occurrence, thus achieving compression.

[0004] When traditional LZ77 compression algorithms are directly applied to structured order data, they have significant limitations. The standard LZ77 algorithm is "structure-blind," treating the entire order record as an indiscriminate byte stream and failing to identify field boundaries and data types (e.g., "quantity" is numeric, while "ingredient type" is character). Furthermore, traditional LZ77 compression algorithms use a uniform sliding window when compressing data in each field category, ignoring the patterns within the data in each category. For example, the ingredient type field may have high repetition, or the quantity field may exhibit numerical fluctuations. When using a uniform sliding window, traditional LZ77 compression algorithms cannot employ targeted and more efficient compression strategies for highly repetitive fields or numerical fields with numerical patterns. Summary of the Invention

[0005] To address the technical problem that the LZ77 compression algorithm cannot perceive the field boundaries and data types of structured order data, and cannot apply different compression strategies to numeric and character fields, resulting in poor compression efficiency, this invention provides a method and system for storing food order data.

[0006] In a first aspect, the present invention provides a method for storing food ingredient order data, employing the following technical solution:

[0007] A method for storing food order data, comprising the following steps:

[0008] Retrieve order data to be compressed, which contains fields of various types, and divide the fields into character fields and numeric fields;

[0009] If any type of field is numeric, the initial degree of change of the field is obtained based on the type of data in the field and the dispersion of the data values; the initial degree of change is corrected based on the repetition of the data arrangement in the field and the variation between adjacent data to obtain the final degree of change of the numeric field.

[0010] If any type of field is a character type, the initial degree of change of the field is obtained according to the types of strings in the field; the initial degree of change is corrected according to the repetition of the string arrangement in the field and the edit distance between adjacent strings to obtain the final degree of change of the field.

[0011] Based on the final degree of change of each type of field, the sliding window size of each type of field is adaptively obtained; the order data to be compressed is compressed and stored independently for each type of field according to the corresponding sliding window size of each type of field.

[0012] The innovation of this invention lies in its adaptive sliding window for each type of field based on the final degree of change of each type of field. This solves the problem of the traditional LZ77 algorithm using a fixed sliding window. By aggregating the data corresponding to each type of field and compressing each type of field separately according to the sliding window of each type of field, fields with high variability can obtain smaller windows to improve speed, while fields with low variability can obtain larger windows to capture long-distance repetitions, thus achieving the best balance between compression efficiency and speed. Furthermore, the final degree of change of each type of field first considers the types of data values ​​and the degree of fluctuation of data values ​​in the field, and then considers the repetition of continuous data or characters in the field as well as the variation range between continuous data or self-reading, thereby accurately assessing the actual compression difficulty of the data in each type of field.

[0013] Preferably, if any type of field is numeric, obtaining the initial degree of change of that type of field based on the type of data and the dispersion of data values ​​in that type of field includes:

[0014] ;

[0015] In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of data types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; This represents the standard deviation of all data in the i-th class field; This represents the average value of all data in the i-th category field; `norm()` represents the preset hyperparameters; `norm()` represents the normalization function; `log()` represents the logarithmic function.

[0016] By considering the number of data types in the field and the standard deviation of the data, the initial degree of change of the field can be evaluated, which can accurately reflect the global statistical characteristics of the data.

[0017] Preferably, the step of correcting the initial degree of change based on the repetition of data arrangement in this type of field and the magnitude of change between adjacent data to obtain the final degree of change of the numerical field includes:

[0018] ;

[0019] In the formula, This represents the final degree of change of the i-th type of field; This represents the initial degree of change in the i-th type of field; This represents the number of data items in the i-th type field; This represents the value of the j-th data in the i-th field; This represents the value of the (j+1)th data in the i-th field. This represents an indicator function; its value is 1 if the condition inside the parentheses is true, and 0 otherwise. This represents the absolute value of the difference between the j-th data and the (j+1)-th data in the i-th field. This represents the preset hyperparameters.

[0020] By considering both the number of data types in the field and the arrangement characteristics of the data within the field, the bias caused by single-dimensional evaluation is avoided, making the assessment of the difficulty of field compression more comprehensive and accurate.

[0021] Preferably, if any type field is of character type, obtaining the initial degree of change of that type field based on the types of strings in that type field includes:

[0022] ;

[0023] In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of string types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; norm() represents the normalization function; log() represents the logarithmic function.

[0024] Preferably, the step of correcting the initial degree of change based on the repetition of string arrangements in the field and the edit distance between adjacent strings to obtain the final degree of change for the field includes:

[0025] ;

[0026] In the formula, This represents the final degree of change of the i-th type of field; This represents the initial degree of change in the i-th type of field; Represents the number of strings in the i-th type of field; Represents the j-th string in the i-th field; This represents the (j+1)th string in the i-th field. This represents an indicator function; its value is 1 if the condition inside the parentheses is true, and 0 otherwise. This represents the edit distance between the j-th string and the (j+1)-th string in the i-th field. This represents the preset hyperparameters.

[0027] Preferably, the step of adaptively obtaining the sliding window size for each type of field based on the final degree of change of each type of field includes:

[0028]

[0029] In the formula, The sliding window size represents the i-th type of field; This represents the final degree of change of the i-th type of field; This represents the preset initial sliding window size; This represents the minimum setting value for the preset sliding window; This represents the floor symbol.

[0030] Based on the final degree of change of each type of field, the sliding window size of each type of field is adaptively obtained, so that fields with high degree of change can get smaller windows to improve speed, and fields with low degree of change can get larger windows to capture long-distance repetitions, thus achieving the best balance between compression efficiency and speed.

[0031] Preferably, the step of independently compressing and storing the data for each type of field according to the corresponding sliding window size includes:

[0032] Using the LZ77 compression algorithm, all data in each field is compressed according to the sliding window size of each field, resulting in compressed data for each field, which is then stored.

[0033] It improves compression efficiency.

[0034] Preferably, the step of obtaining the order data to be compressed, which contains multiple types of fields, includes:

[0035] By using SQL queries, the required fields are selected from the order table in the database and exported to obtain the order data to be compressed. For example, the required fields are order ID, ingredient type, quantity, price, unit price, etc.

[0036] Preferably, the step of dividing the field into character fields and numeric fields includes:

[0037] The fields are divided into numeric fields and character fields. For example, order ID, quantity, and price are numeric fields, while food type is a character field.

[0038] Secondly, this invention provides a food ingredient order data storage system, which adopts the following technical solution:

[0039] A food ingredient order data storage system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the aforementioned food ingredient order data storage method is implemented.

[0040] By adopting the above technical solution, a computer program is generated from the above-mentioned method for storing food order data, and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0041] This invention has the following technical advantages: It solves the structural blindness and fixed sliding window problems of the traditional LZ77 algorithm by employing two core methods: columnar reconstruction and adaptive sliding window. By aggregating data corresponding to each type of field and adaptively obtaining the sliding window for each type of field based on the final degree of change in the data, each type of field is compressed individually. This allows fields with high variability to obtain smaller windows to improve speed, while fields with low variability can obtain larger windows to capture long-distance repetitions, thus achieving an optimal balance between compression efficiency and speed. Furthermore, the final degree of change for each type of field first considers the types and fluctuations of data values ​​in the field, then considers the repetition of continuous data or characters in the field, as well as the variation amplitude between continuous data or self-reading, thereby accurately assessing the actual compression difficulty of the data in each type of field, facilitating subsequent adaptive sliding window selection for each type of field. Attached Figure Description

[0042] Figure 1 This is a flowchart of a method for storing food order data according to an embodiment of the present invention;

[0043] Figure 2 A diagram showing the comparison of compression time before and after optimization of the LZ77 algorithm. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0045] This invention discloses a method for storing food order data, referring to... Figure 1 This includes steps S1-S4:

[0046] S1: Collect order data to be compressed.

[0047] In this embodiment of the invention, the required fields are selected from the order table of the database and exported through SQL query to obtain the order data to be compressed. For example, the required fields are categories such as order ID, ingredient type, quantity, price, and unit price. The fields are divided into numeric fields and character fields. For example, order ID, quantity, and price are numeric fields, and ingredient type is a character field. See Table 1, which illustrates the collected order data to be compressed.

[0048] Table 1

[0049]

[0050] S2: Based on the types of data in each field and the dispersion of data values, obtain the initial degree of change for each field.

[0051] It should be noted that in the order data to be compressed, the business attributes and data characteristics of different types of fields vary greatly. If the number of data types in any type of field is smaller (for example, the data in the quantity field in Table 1 is 50, 500, 100, and the data type in the quantity field is 3), then the data redundancy of that type of field is relatively high, and its compression difficulty is relatively small. In this case, the initial degree of change of that type of field is smaller. However, for numeric fields, if the dispersion of data values ​​in the numeric field is greater, it means that the data values ​​in the numeric field are unstable. This means that the compression difficulty of the numeric field is greater, and the initial degree of change is greater.

[0052] Therefore, for character-type fields, this invention obtains the initial degree of change based on the type of string in the field, while for numeric-type fields, this invention obtains the initial degree of change based on the type of data in the field and the dispersion of the data values.

[0053] In this embodiment of the invention, if the i-th type of field is a numeric field, the initial degree of change of the i-th type of field is obtained:

[0054] ;

[0055] In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of data types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; This represents the standard deviation of all data in the i-th class field; This represents the average value of all data in the i-th category field; Representing preset hyperparameters, in this embodiment of the invention, the preset... To prevent the denominator from being 0; norm() represents the normalization function;

[0056] The larger the value, the more data types there are in the i-th field. In this case, the content of the i-th field is more diverse, and therefore the initial degree of change of the i-th field is greater. This represents the dispersion of data in the i-th field. The larger the value, the more unstable the value of the i-th field is. In this case, the amount of information in the i-th field is greater and the compression is more difficult. At this time, the initial change of the i-th field is greater.

[0057] If the i-th type field is a character field, obtain the initial degree of change of the i-th type field:

[0058] ;

[0059] In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of string types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; norm() represents the normalization function; log() represents the logarithmic function.

[0060] S3: Based on the repetition of data arrangement in each field and the magnitude of change between data, correct the initial degree of change of each field and obtain the final degree of change of each field; based on the final degree of change of each field, adaptively obtain the sliding window size of each field.

[0061] It should be noted that the initial degree of change obtained from the above steps is a measure based on global statistical features. It focuses on the types and dispersion of data in each type of field, but completely ignores the order of these values. The effectiveness of the LZ77 compression algorithm is closely related to the order of data because it relies on the detection of repeating patterns to compress data. If the data values ​​in a type of field are repeated and continuous, the LZ77 compression algorithm can quickly identify the repetition and compress it. If the data values ​​in a type of field appear alternately without a clear pattern, LZ77 will have difficulty identifying the repeating pattern, resulting in poor compression. For example, if the first 500 strings in the food category field are "potato" and the last 500 strings are "vegetable", or if "potato" and "vegetable" appear alternately in the food category field, then the initial degree of change in the food category field is equal. However, the former is much easier to compress than the latter. Therefore, this invention needs to analyze the repetition of data arrangement in each type of field (the analysis of the repetition of data arrangement in character fields and data fields is consistent).

[0062] The above analysis of the global statistical characteristics and data repetition of each type of field can only analyze the compression difficulty of the field based on the macroscopic characteristics of the data in the field, but cannot capture the degree of change between the data in each type of field in detail. Therefore, this invention also needs to analyze the magnitude of change between the data in each type of field. For numeric fields, the greater the difference between the data values ​​in the numeric field, the greater the compression difficulty of the numeric field. For character fields, the greater the edit distance between the strings in the character field, the greater the compression difficulty of the character field. Therefore, this invention also needs to analyze the repetition of the data arrangement and the magnitude of change between the data in each type of field, correct the initial degree of change of each type of field, and obtain the final degree of change of each type of field.

[0063] In this embodiment of the invention, if the i-th type of field is a character field, the final degree of change of the i-th type of field is obtained:

[0064] ;

[0065] In the formula, This represents the final degree of change of the i-th type of field; This represents the initial degree of change in the i-th type of field; Represents the number of strings in the i-th type of field; Represents the j-th string in the i-th field; This represents the (j+1)th string in the i-th field. This represents an indicator function; its value is 1 if the condition within the parentheses is true, and 0 otherwise. In other words, if... ,at this time Conversely, it is 0; This represents the edit distance between the j-th string and the (j+1)-th string in the i-th field. Representing preset hyperparameters, in this embodiment of the invention, the preset... To prevent the data from being 0;

[0066] The larger the value, the less repetitive the character arrangements in the i-th field are, and the greater the final degree of change in the i-th field; it is known that edit distance can quantify the degree of difference between strings, representing the minimum number of operations required to convert one string into another, therefore The larger the value, the greater the variation between consecutive characters in the i-th field, resulting in greater compression difficulty, and the greater the final degree of change in the i-th field.

[0067] If the i-th type of field is a numeric field, obtain the final degree of change of the i-th type of field:

[0068] ;

[0069] In the formula, This represents the final degree of change of the i-th type of field; This represents the initial degree of change in the i-th type of field; This represents the number of data items in the i-th type field; This represents the value of the j-th data in the i-th field; This represents the value of the (j+1)th data in the i-th field. This represents an indicator function; its value is 1 if the condition within the parentheses is true, and 0 otherwise. In other words, if... ,at this time Conversely, it is 0; This represents the absolute value of the difference between the j-th data and the (j+1)-th data in the i-th field. Representing preset hyperparameters, in this embodiment of the invention, the preset... To prevent the data from being 0;

[0070] The larger the value, the less repetitive the data arrangement in the i-th field is, and the greater the final degree of change in the i-th field. The larger the value, the greater the variation between consecutive data in the i-th field. In other words, the more drastic the variation between consecutive data in the i-th field, the greater the difficulty of compression, and the greater the final degree of change in the i-th field.

[0071] It should be noted that the greater the final change in any type of field, the less continuous repetition there is in the data within that field. The repeating patterns typically exist within a short distance, so a smaller sliding window can quickly capture these local similarities. At the same time, a smaller window improves matching speed and avoids processing too much irrelevant content. Conversely, the smaller the final change in any type of field, the greater the continuous repetition there is in the data within that field. The repeating patterns typically exist within a longer distance. To capture these repeating patterns, the LZ77 compression algorithm needs to use a larger sliding window, which can effectively find repeating patterns over a longer range, improving compression efficiency.

[0072] In this embodiment of the invention, the sliding window size for each type of field is obtained:

[0073]

[0074] In the formula, The sliding window size represents the i-th type of field; This represents the final degree of change of the i-th type of field; This represents the preset initial sliding window size. In this embodiment of the invention, the preset... In other embodiments, implementers may pre-determine specific implementation methods. The value; The minimum set value representing the preset sliding window, in this embodiment of the invention, is the preset value. In order to avoid The value is too large, causing Setting it to 0 ensures that even fields with a high degree of change have a basic matching window; Represents the floor function;

[0075] The larger the value, the greater the final change of the i-th type of field, and the more difficult it is to compress the i-th type of field. In this case, a smaller sliding window needs to be set to improve the matching speed.

[0076] S4: Compress and store the data in each field separately based on the sliding window size of each field.

[0077] It should be noted that in this step, the original row-based storage of the order data to be compressed (the original order data to be compressed in Table 1 was stored row by row) is reconstructed into a columnar structure that is more conducive to compression (the order data to be compressed in Table 1 is stored column by column).

[0078] In this embodiment of the invention, the LZ77 compression algorithm is used to compress all data in each field according to the sliding window size of each field, so as to obtain compressed data for each field and store it.

[0079] If you need to query any order data, use the LZ77 compression algorithm to decompress the compressed data of each type of field and use it as the data of each column. Combine all the column data together to get the original row-by-row order data to be compressed, as shown in Table 1.

[0080] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for storing food ingredient order data, characterized in that, include: Retrieve order data to be compressed, which contains fields of various types, and divide the fields into character fields and numeric fields; If any field is numeric, the initial degree of change for that field is obtained based on the type of data and the dispersion of the data values. The initial degree of change is then corrected based on the repetition of the data arrangement and the magnitude of change between adjacent data points to obtain the final degree of change for the numeric field, satisfying the following: ； This represents the final degree of change of the i-th type of field in a numeric field; This represents the initial degree of change of the i-th type of field in a numeric field; This represents the number of data items in the i-th type field of a numeric field. This represents the value of the j-th data in the i-th type of field within a numeric field. This represents the value of the (j+1)th data in the i-th type of field within a numeric field; This represents an indicator function; its value is 1 if the condition inside the parentheses is true, and 0 otherwise. This represents the absolute value of the difference between the j-th data point and the (j+1)-th data point in the i-th type of field within a numeric field. The preset hyperparameters represent numerical fields; the final degree of change takes into account both the number of data types in the field and the arrangement characteristics of the data within the field. If any field is of type character, the initial degree of change for that field is obtained based on the types of strings within it. This initial degree of change is then adjusted based on the repetition of string arrangements and the edit distance between adjacent strings to obtain the final degree of change for that field, satisfying the following: ； This represents the final degree of change of the i-th type of field in a character field; This represents the initial degree of change of the i-th type of field in a character field; This represents the number of strings in the i-th type field of a character field; This represents the j-th string in the i-th type of character field; This represents the (j+1)th string in the i-th type field of a character field; This represents the edit distance between the j-th string and the (j+1)-th string in the i-th type of character field; This represents the preset hyperparameters in a character field; Based on the final degree of change of each type of field, the sliding window size of each type of field is adaptively obtained; the order data to be compressed is compressed and stored independently for each type of field according to the corresponding sliding window size of each type of field.

2. The method for storing food order data according to claim 1, characterized in that, If any type of field is numeric, the initial degree of change of that type of field is obtained based on the type of data in that field and the dispersion of the data values, including: ； In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of data types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; This represents the standard deviation of all data in the i-th class field; This represents the average value of all data in the i-th category field; `norm()` represents the preset hyperparameters; `norm()` represents the normalization function; `log()` represents the logarithmic function.

3. The method for storing food order data according to claim 1, characterized in that, If any type of field is a character type, the initial degree of change of that type of field is obtained based on the types of strings in that field, including: ； In the formula, This represents the initial degree of change in the i-th type of field; This represents the number of string types in the i-th field; This represents the total number of data in all class fields of the order data to be compressed; norm() represents the normalization function; log() represents the logarithmic function.

4. The method for storing food order data according to claim 1, characterized in that, The step of adaptively obtaining the sliding window size for each type of field based on the final degree of change of each type of field includes: In the formula, The sliding window size represents the i-th type of field; This represents the final degree of change in the i-th type of field. include and ; This represents the preset initial sliding window size; This represents the minimum setting value for the preset sliding window; This represents the floor symbol.

5. The method for storing food order data according to claim 1, characterized in that, The step of independently compressing and storing data for each type of field based on the corresponding sliding window size includes: Using the LZ77 compression algorithm, all data in each field is compressed according to the sliding window size of each field, resulting in compressed data for each field, which is then stored.

6. The method for storing food order data according to claim 1, characterized in that, The process of obtaining order data to be compressed, which contains multiple types of fields, includes: By using SQL queries, the required fields are selected from the order table in the database and exported to obtain the order data to be compressed. The required fields include order ID, ingredient type, quantity, price, and unit price.

7. The method for storing food order data according to claim 1, characterized in that, The process of dividing fields into character fields and numeric fields includes: The fields are divided into numeric fields and character fields. Numeric fields include order ID, quantity, and price, while character fields include ingredient type.

8. A food ingredient order data storage system, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement a method for storing food order data according to any one of claims 1-7.

Images