Data optimization method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data optimization method and device, equipment, a storage medium and a program product. The method comprises the following steps: in response to a storage event of to-be-stored original data, performing redundancy feature analysis on the to-be-stored original data based on a storage strategy analysis model, and determining redundancy features in the to-be-stored original data; based on redundancy characteristics, determining a redundancy generation strategy and non-redundant data in the to-be-stored original data, the redundancy generation strategy being used for deriving redundant data in the to-be-stored original data; generating optimized data based on the relationship between the non-redundant data and the redundancy generation strategy; and storing the optimized data instead of the to-be-stored original data. According to the data optimization scheme provided by the embodiment of the invention, the utilization rate of the storage space can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data optimization method, apparatus, device, storage medium and program product. Background Technology

[0002] In data storage applications, storage efficiency is crucial. Data undergoes acquisition, encoding, and encapsulation before storage. However, data compression, a vital component of data storage, may suffer from insufficient compression granularity, impacting storage space utilization.

[0003] Therefore, a solution for optimizing data storage is needed. Summary of the Invention

[0004] This application provides a data optimization method, apparatus, device, storage medium, and program product that can improve the utilization rate of storage space.

[0005] In a first aspect, embodiments of this application provide a data optimization method, the method comprising: In response to a storage event of the original data to be stored, a redundancy feature analysis is performed on the original data to be stored based on a storage strategy analysis model to determine the redundancy features in the original data to be stored. Based on the redundancy characteristics, a redundancy generation strategy and non-redundant data in the original data to be stored are determined. The redundancy generation strategy is used to derive the redundant data in the original data to be stored. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated. The optimized data is used to replace the original data to be stored.

[0006] In one feasible implementation, the redundancy feature is the semantic dependency relationship between multiple original sub-data in the original data to be stored, and the step of determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature includes: Based on the semantic dependency relationship, the non-redundant data in the multiple original sub-data are determined, and a logical redundancy derivation rule for deriving the redundant data is determined, and the logical redundancy derivation rule is used as the redundancy generation strategy.

[0007] In one feasible implementation, the optimized data includes data content information and tag attributes. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated, including: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the logical redundancy derivation type.

[0008] In one feasible implementation, the redundancy generation strategy further includes the number of times the optimized data references the logical redundancy derivation rule; if the number of times the optimized data references the logical redundancy derivation rule is zero, the method further includes clearing the redundancy generation strategy.

[0009] In one feasible implementation, the redundancy feature is the structural similarity between multiple original sub-data in the original data to be stored, and the step of determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature includes: Extract the structural commonalities among the multiple original sub-data to generate a common structural template; The difference between the original sub-data and the common structure template is used as the non-redundant data, and the common structure template is determined and used as the redundancy generation strategy.

[0010] In one feasible implementation, the optimized data includes data content information and tag attributes. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated, including: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the structural commonality extraction type.

[0011] In one feasible implementation, the redundancy generation strategy further includes the number of times the optimized data references the common structure template; if the number of times the optimized data references the common structure template is zero, the method further includes clearing the redundancy generation strategy.

[0012] Secondly, embodiments of this application provide a data optimization apparatus, the apparatus comprising: The redundancy feature analysis module is used to respond to the storage event of the original data to be stored, and perform redundancy feature analysis on the original data to be stored based on the storage strategy analysis model to determine the redundancy features in the original data to be stored. The determination module is used to determine a redundancy generation strategy and non-redundant data in the original data to be stored based on the redundancy features. The redundancy generation strategy is used to derive the redundant data in the original data to be stored. An optimized data generation module is used to generate optimized data based on the relationship between the non-redundant data and the redundancy generation strategy; The data storage module is used to store the optimized data in place of the original data to be stored.

[0013] Thirdly, embodiments of this application provide a data optimization device, the device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] The data optimization method, apparatus, device, storage medium, and program product of this application embodiment, in response to a storage event of original data to be stored, performs redundancy feature analysis on the original data to be stored based on a storage strategy analysis model. It can determine a redundancy generation strategy and non-redundant data in the original data to be stored based on the analyzed redundancy features. Based on the non-redundant data and the redundancy generation strategy, it generates optimized data, which includes the non-redundant data and reference information pointing to the redundancy generation strategy. Thus, redundant data in the original data can be removed, and based on the non-redundant data and the reference information pointing to the redundancy generation strategy, data integrity can be maintained, without losing information from the original data, and saving space occupied by redundant data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This paper illustrates the overall architecture of a data optimization scheme provided in one embodiment of the present application. Figure 2 A flowchart illustrating a data optimization method provided in one embodiment of this application is shown; Figure 3 A flowchart illustrating a data optimization method provided in another embodiment of this application is shown; Figure 4 This illustration shows a storage diagram of an in-memory database provided in one embodiment of the present application; Figure 5 This illustration shows a storage diagram of a memory database provided in another embodiment of this application; Figure 6 This illustration shows a schematic diagram of the data structure for optimizing data and logical redundancy derivation rules provided in one embodiment of this application; Figure 7 This illustration shows a schematic diagram of the data structure of the optimized data and common structure template provided in one embodiment of this application; Figure 8 A flowchart illustrating a data optimization method provided in yet another embodiment of this application is shown; Figure 9 This paper shows a schematic diagram of the structure of a data optimization device provided in an embodiment of this application; Figure 10 A schematic diagram of the hardware structure of the data optimization device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0021] Existing technologies have two significant drawbacks in data storage optimization: First, they lack business semantic understanding capabilities. Their general compression algorithms treat business data as meaningless binary streams, failing to identify deep logical redundancy in conjunction with actual business scenarios (such as order total price being derived through business rules, or IoT device status being determined based on original indicators), making it difficult to achieve more refined storage space optimization. Second, they cannot proactively identify and extract potential optimal (or near-optimal) common structures in the data. They struggle to identify common parts between data records (general compression can only eliminate local repetitive segments in the underlying byte stream) and cannot adapt to new common patterns in data evolution (such as binary encoding requiring predefined templates, and object sharing only supporting the reuse of identical objects). Due to the lack of global intelligent statistical analysis capabilities, shareable common parts in the data are redundantly stored, resulting in wasted space.

[0022] To address the problems of the prior art, embodiments of this application provide a data optimization method, apparatus, device, storage medium, and program product.

[0023] Figure 1 The diagram shows the overall architecture of a data optimization scheme provided in one embodiment of this application.

[0024] As shown in the figure, one embodiment of this application provides a structure optimizer 100 and an intelligent analysis unit 200. The structure optimizer, which can be called an intelligent data storage structure optimizer, is used to optimize the space of data successfully written to the database. The intelligent analysis unit is the intelligent analysis brain unit for data storage optimization. In this embodiment, the structure optimizer is installed in the database as a plug-in. The database can be an in-memory database, specifically a key-value database. The structure optimizer sends the data written to the in-memory data storage pool of the database to the intelligent analysis unit, i.e., realizes data collection and reporting. The intelligent analysis unit can perform redundancy feature analysis on the original data to be stored, obtain redundancy generation strategy or data storage optimization strategy, and feed back the redundancy generation strategy or data storage optimization strategy to the structure optimizer. The structure optimizer responds to the redundancy generation strategy or data storage optimization strategy and reconstructs the original data in the database according to the redundancy generation strategy or data storage optimization strategy, i.e., data storage optimization and reconstruction, which can effectively reduce the memory space occupied by the data. When reading data, the structure optimizer restores the original data according to the storage optimization strategy, and can also restore the reconstructed data to the original data, i.e., realize data content parsing and restoration.

[0025] In this embodiment, the structure optimizer can monitor the time when business data is successfully written to the database and asynchronously report the business data to the intelligent analysis unit, enabling the intelligent analysis unit to perform intelligent analysis. After the intelligent analysis unit feeds back the redundancy generation strategy or data storage optimization strategy generated by the analysis to the structure optimizer, the structure optimizer reconstructs and stores the original data according to the fed-back redundancy generation strategy or data storage optimization strategy. During the data query phase, the structure optimizer restores the original data according to the storage optimization strategy.

[0026] In this embodiment, the intelligent analysis unit constructs a training dataset by collecting multi-source heterogeneous data, including real-time data traffic, business logic rules, business log data, and business logic link topology data. It utilizes a machine learning model to perform deep feature mining and behavioral pattern learning on the multi-source data, automatically generating data optimization storage rules. This machine learning model is referred to as the storage strategy analysis model. Simultaneously, professional operations and maintenance personnel can configure and maintain business data optimization storage rules through the management software interface. Thus, through a dual collaborative mechanism of automatic generation via machine training and manual strategy configuration, a dynamic optimization storage rule knowledge base is constructed, including redundancy generation strategies or data storage optimization strategies. Based on this knowledge base, the intelligent analysis unit performs multi-dimensional intelligent analysis of the business data and feeds back the derived redundancy generation strategies or data storage optimization strategies to the structure optimizer, which then executes data storage optimization operations. Ultimately, this achieves intelligent dynamic optimization of data storage, resulting in a more intelligent and accurate improvement in storage space utilization.

[0027] In this embodiment, the intelligent analysis unit includes an analysis engine for performing logical redundancy field analysis and data common feature analysis. Logical redundancy field analysis is a method for identifying redundancy within a single data record based on business semantics and logical rules; it is used to discover redundant fields that can be derived from other fields or business rules. Data common feature analysis is a method for identifying redundancy among multiple data records based on patterns in the global dataset; it is used to discover common structures, repetitive patterns, or common features among batches of data records, enabling the reuse and storage of common parts.

[0028] The data optimization method provided in the embodiments of this application is described below.

[0029] Figure 2 A flowchart illustrating a data optimization method according to an embodiment of this application is shown. As shown, the method may include the following steps: S201 to S204.

[0030] S201. In response to the storage event of the original data to be stored, perform redundancy feature analysis on the original data to be stored based on the storage strategy analysis model.

[0031] S202. Based on redundancy characteristics, determine the redundancy generation strategy and the non-redundant data in the original data to be stored.

[0032] In this embodiment, the redundancy generation strategy is used to derive redundant data based on the original data to be stored, which consists of non-redundant data and redundant data.

[0033] In this embodiment, the storage strategy analysis model is a neural network model used to perform logical redundancy field analysis and data common feature analysis. For example, the storage strategy analysis model can be a convolutional neural network model or a graph neural network model. A storage event in response to the original data to be stored refers to the event when the storage of a certain piece of original data occurs. Based on the storage strategy analysis model, redundancy feature analysis of the original data to be stored can be performed to obtain redundancy features, and a redundancy generation strategy can be determined based on these features.

[0034] In this embodiment, redundant features are attributes or inherent patterns existing in the original data that do not need to be stored. Redundant data is unnecessary information existing in the original data that does not need to be directly stored. Non-redundant data refers to the data in the original data other than the redundant data. That is, the original data to be stored consists of non-redundant data and redundant data. The redundancy generation strategy refers to the rules or strategies for deriving redundant data based on the original data to be stored.

[0035] In this embodiment, the original data samples, redundant feature samples, and redundant generation strategy samples can be used as a training set, and any machine learning model can be trained using the training set. This enables the machine learning model to perform in-depth feature mining and behavior pattern learning, and automatically generate redundant features and redundant generation strategies.

[0036] In some embodiments, the redundancy feature is the semantic dependency relationship between multiple original sub-data in the original data to be stored. Based on the redundancy feature, determining the redundancy generation strategy and the non-redundant data in the original data to be stored may include: determining the non-redundant data among the multiple original sub-data based on the semantic dependency relationship, and determining a logical redundancy derivation rule for deriving the redundant data, and using the logical redundancy derivation rule as the redundancy generation strategy.

[0037] In this embodiment, the semantic dependencies between multiple original sub-data in the original data to be stored can be analyzed based on the storage strategy analysis model, and the semantic dependencies can be used as redundant features.

[0038] In this embodiment, the semantic dependency relationship between multiple original sub-data refers to the derivation logic rules between multiple original sub-data. Specifically, the semantic dependency relationship means that among a set of original sub-data, multiple original sub-data have a clear relationship at the business semantic level, and based on this relationship, other original sub-data can be deduced from some original sub-data. Original sub-data can be fields or a portion of the original data.

[0039] In this embodiment, semantic dependencies can be determined by constructing a graph of inferential dependencies among the fields in the original data. Furthermore, semantic dependencies can be treated as redundant features.

[0040] In this embodiment, if there is a semantic dependency relationship between multiple original sub-data, that is, if there is a semantic dependency relationship between multiple original sub-data, then non-redundant data that cannot be derived from other original sub-data can be determined based on the semantic dependency relationship. The logical redundancy derivation rule refers to the rule or strategy for deriving redundant data. In this embodiment, the logical redundancy derivation rule can be used as a redundancy generation strategy.

[0041] For example, the original data SL is: { "orderId": "2025090512545216", / / Order number "userId": "236", / / Order user ID "totalPrice": "40", / / Total order price (derivative field) "skuItems": [ / / All product details] { "sku": "milk", / / Product name "price": "3.0", / / Unit price of the product "quantity": "10" / / Quantity of goods }, { "sku": "candy", / / Product name "price": "2.0", / / Unit price of the product "quantity": "5" / / Quantity of goods } ] }

[0042] In this embodiment, the logical redundancy derivation rule can be `totalPrice = sum(price * quantity)`, and the memory address storing the logical redundancy derivation rule can be 0x000026. In this embodiment, the logical redundancy derivation rule refers to the total price being equal to the sum of (unit price of goods × quantity of goods) of all goods. The redundant data is "totalPrice": "40".

[0043] Figure 3 A flowchart illustrating a data optimization method provided in another embodiment of this application is shown.

[0044] As shown in the figure, in some embodiments, the redundancy feature is the structural similarity between multiple original sub-data in the original data to be stored. Based on the redundancy feature, determining the redundancy generation strategy and the non-redundant data in the original data to be stored may include the following steps: S401 to S402.

[0045] S401. Extract the structural commonalities among multiple original sub-data and generate a common structural template.

[0046] In this embodiment, assuming structural similarity exists, the common and repetitive structural parts of multiple original sub-data are extracted to form a unified template, namely the common structural template. This common structural template is the common skeleton or common structure of all similar sub-data, and does not need to be stored repeatedly in each data entry.

[0047] In this embodiment, prior to S401, the structural similarity between multiple original sub-data in the original data to be stored can be analyzed as a redundancy feature based on the storage strategy analysis model.

[0048] In this embodiment, the process involves identifying similarities in data organization or structure among batches of original sub-data, and using this structural similarity as a redundancy feature. Specifically, for key-value databases, multiple sets of data with the same key value can be identified as multiple original sub-data sets with structural similarity. When structural similarity exists among multiple original sub-data sets, step S401 is executed.

[0049] S402. Treat the difference data between the original sub-data and the common structure template as non-redundant data, and determine the common structure template, and use the common structure template as a redundancy generation strategy.

[0050] In this embodiment, the difference data refers to unique, specific content outside the common structure template of the original sub-data. The difference data is treated as non-redundant data, and the common structure template is used as a redundancy generation strategy. In other words, the redundancy generation strategy may include the common structure template; in another embodiment, the redundancy generation strategy may be the common structure template itself.

[0051] For example, if the original data SS includes four records: Record 1: Chinese people love to drink green tea because it is good for their health. Record 2: Americans love to drink cola because it tastes good. Record 3: {"skuid":"10", "name":"green tea", "price":"80"}. Record 4: {"skuid":"11", "name":"cola", "price":"2.5"}. For records 1 and 2, the common structure template 1 is: %s loves to drink %s because it %s. %s is a placeholder, indicating that this position is filled with difference data. The address of common structure template 1 is 0x000016. For records 3 and 4, the common structure template 2 is: {"skuid":"%s", "name":"%s", "price":"%s"}. %s is a placeholder, indicating that this position is filled with difference data. The address of common structure template 2 is 0x000017.

[0052] S203. Based on the relationship between non-redundant data and redundant generation strategies, generate optimized data.

[0053] In this embodiment, the optimized data includes non-redundant data and reference information pointing to the redundancy generation strategy.

[0054] In this embodiment, non-redundant data and reference information pointing to the redundancy generation strategy can be merged, and the merged data can be used as optimized data.

[0055] For example, for the original data SL, the corresponding optimized data is: { "optimized_data": { "orderId": "2025090512545216", "userId": "236", "skuItems": [ { "sku": "milk", "price": "3.0", "quantity": "10" }, { "sku": "candy", "price": "2.0", "quantity": "5" } ] }, "strategy_pointer": 0x000026 / / A reference to the redundancy generation strategy } In this embodiment, strategy_pointer represents a reference to the redundancy generation strategy, and "strategy_pointer": 0x000026 indicates that the address of the redundancy generation strategy is 0x000026.

[0056] In some embodiments, for the original data SS, the corresponding optimized data is: { different_data: [Chinese people, green tea, good for health], *common_model_pointer: 0x000016 / / A reference to the common model template } { different_data: [Americans, Coca-Cola, popular taste], *common_model_pointer: 0x000016 / / A reference to the common model template } { different_data: [10, Green Tea, 80], *common_model_pointer: 0x000017 / / A reference to the common model template } { different_data: [11, Cola, 2.5], *common_model_pointer: 0x000017 / / A reference to the common model template }

[0057] In this embodiment, different_data represents differential data, *common_model_pointer represents reference information pointing to the common structure template, and *common_model_pointer: 0x000017 indicates that the address of the common structure template is 0x000017.

[0058] In some embodiments, the optimized data includes data content information and tag attributes. Based on the relationship between non-redundant data and redundancy generation strategies, the optimized data is generated, which may include: generating reference information pointing to the redundancy generation strategy corresponding to the non-redundant data based on the relationship between the non-redundant data and the redundancy generation strategy; using the non-redundant data and the reference information pointing to the redundancy generation strategy as the data content information of the optimized data; and setting the tag attribute of the optimized data to a logical redundancy derivation type.

[0059] In this embodiment, there is a correspondence between non-redundant data and redundancy generation strategies. Specifically, if, for a given set of non-redundant and redundant data of original data to be stored, the redundant data is derived or generated based on a redundancy generation strategy, then the non-redundant data corresponds to that redundancy generation strategy. Multiple different sets of non-redundant data can all correspond to a single redundancy generation strategy, meaning that the redundant data corresponding to each of these multiple different sets of non-redundant data is derived or generated by a single redundancy generation strategy.

[0060] Data content information refers to the core actual content within the optimized data. Specifically, data content information can include non-redundant data and reference information pointing to redundancy generation strategies. Tag attributes refer to the data attributes of the optimized data, used to identify how the corresponding redundant data was obtained. Specifically, the tag attribute of the optimized data is a logical redundancy derivation type, which indicates that the redundant data corresponding to this optimized data was obtained through logical redundancy analysis or derivation. In some embodiments, tag attributes can be data metadata of the optimized data. Data metadata is data that describes the data; it is not the business content of the data itself, but rather information used to explain data attributes or tag data.

[0061] In some embodiments, the optimized data includes data content information and tag attributes. Based on the non-redundant data and the redundancy generation strategy, the optimized data is generated, including: using the non-redundant data and reference information pointing to the redundancy generation strategy as the data content information of the optimized data; and setting the tag attributes of the optimized data to a structural commonality extraction type.

[0062] In this embodiment, data content information refers to the core actual content of the optimized data. Specifically, data content information may include non-redundant data and reference information pointing to the redundancy generation strategy. Tag attributes refer to the data attributes of the optimized data, used to identify how the corresponding redundant data was obtained. Specifically, the tag attribute of the optimized data is a structural commonality extraction type, which can indicate that the redundant data corresponding to this optimized data was generated based on the structural commonality extraction type. In some embodiments, tag attributes may be data metadata of the optimized data. Data metadata is data that describes the data; it is not the business content of the data itself, but information used to explain data attributes or tag data.

[0063] S204. Use optimized data to replace the original data to be stored.

[0064] In this embodiment, after generating optimized data, the optimized data can be used to replace the original data to be stored, thereby optimizing data storage and improving the space utilization of stored data.

[0065] Figure 4 A storage diagram of an in-memory database provided in one embodiment of this application is shown.

[0066] As shown in the figure, the in-memory database stores optimized data and logical redundancy derivation rules. The optimized data contains reference information pointing to the logical redundancy derivation rules.

[0067] Figure 5 A storage diagram of a memory database provided in another embodiment of this application is shown.

[0068] As shown in the figure, the memory database stores optimized data and common structure templates, and the optimized data contains reference information pointing to the common structure templates.

[0069] Figure 6 This illustration shows a schematic diagram of the data structure for optimizing data and logical redundancy derivation rules provided in one embodiment of this application.

[0070] As shown in the figure, the data structure of the optimized data includes data metadata and data content information. The data metadata includes the tag attribute, and the data content information includes the non-redundant data `optimized_data` and the reference information `strategy_pointer` pointing to the logical redundancy derivation rule. The `strategy_pointer` includes the address of the logical redundancy derivation rule. The data structure of the logical redundancy derivation rule includes data metadata and data content information. The data metadata of the logical redundancy derivation rule includes the tag attribute, and the data content information includes the unique identifier `strategy_id`, the detailed description `strategy_info`, and the refcount. The detailed description of the logical redundancy derivation rule refers to the specific information describing the rule, which may include the logical derivation calculation formula and related original sub-data or fields. The refcount refers to the number of times the optimized data references the logical redundancy derivation rule; specifically, the refcount automatically increments by one each time an optimized data point references a logical redundancy derivation rule. Multiple optimized data points can reference the same logical redundancy derivation rule. For example, for different business data, each different set of data can be represented by totalPrice = sum(price * quantity).

[0071] In some embodiments, if data needs to be queried from the in-memory database, the type of the currently optimized data can be determined by the tag identifier in the data metadata. If it is ordinary raw data, the data content is returned directly; if it is a logical redundancy derivation type, the corresponding logical redundancy derivation rule is found through the strategy_pointer in the optimized data, the optimized_data is restored to the original data, and the data content is returned.

[0072] The dynamic elimination mechanism for redundant business logic fields provided in this embodiment can solve the space optimization bottleneck problem caused by the lack of business semantic understanding, which leads to logical redundancy and invalid storage. By constructing a computational dependency graph between fields and establishing a dynamic identification and elimination mechanism for derivable fields, precise removal of semantic layer redundancy is achieved. This represents a technological advancement from byte stream compression to deep business logic optimization, breaking through the limitations of traditional compression algorithms at the semantic understanding level, significantly reducing the proportion of invalid data storage, and thus improving memory space utilization efficiency.

[0073] In some embodiments, when the number of times the logical redundancy derivation rule is used to optimize data references is zero, the method further includes: clearing the redundancy generation strategy.

[0074] In this embodiment, if the number of times the logical redundancy derivation rule for optimizing data references is zero, it means that the logical redundancy derivation rule no longer has optimized data references. At this time, the redundancy generation strategy can be cleared to further improve the space optimization efficiency of the memory database.

[0075] Figure 7 A schematic diagram of the data structure of the optimized data and common structure template provided in one embodiment of this application is shown.

[0076] As shown in the figure, the data structure of the optimized data includes data metadata and data content information. The data metadata includes the tag attribute, and the data content information includes the differential data `different_data` and the reference information `common_model_pointer` pointing to the common structure template. The reference information to the common structure template includes the address of the common structure template. The data structure of the common structure template includes data metadata and data content information. The data metadata of the common structure template includes the tag attribute, and the data content information includes the unique identifier `model_id`, the common structure template name `data`, and the refcount. The common structure template can include reusable common data. The refcount refers to the number of times the optimized data references the common structure template; specifically, each time an optimized data references the common structure template, the refcount is automatically incremented by one. Multiple optimized data can reference the same common structure template.

[0077] In some embodiments, directly storing the memory address of the redundancy generation strategy is to save storage space and improve lookup efficiency.

[0078] In some embodiments, the redundancy generation strategy further includes the number of times the optimized data references the common structure template. When the number of times the optimized data references the common structure template is zero, the method further includes clearing the redundancy generation strategy.

[0079] In this embodiment, if the number of times the optimized data references the common structure template is zero, it indicates that no optimized data references the common structure template, and the common structure template can be cleared, thereby further improving the storage optimization efficiency of the in-memory database.

[0080] In some embodiments, when querying data from an in-memory database, the current data type is determined by the tag identifier in the data metadata. If it is ordinary raw data, the data content is returned directly; if it is a common structure extraction type, the corresponding common structure template is found through the common_model_pointer in its data, and the elements in different_data are filled into the template in order according to the placeholder positions (such as %s) defined in the data field content of the template to restore the original data.

[0081] In some embodiments, a common structure template storage mechanism can solve the problem of common redundancy and wasted storage resources caused by the lack of identification of common parts across records. By extracting common features of the dataset and constructing reusable templates for independent storage, data instances only store differentiated attributes and common template reference identifiers. This overcomes the technical bottleneck of traditional compression algorithms that can only identify local repetitive segments at the byte stream level, and can significantly improve storage space utilization, especially in high-similarity dataset scenarios.

[0082] Figure 8 A flowchart illustrating a data optimization method provided in yet another embodiment of this application is shown.

[0083] As shown in the figure, data optimization methods may include the following steps: A11. Obtain data.

[0084] In this embodiment, the structure optimizer retrieves data from an in-memory database.

[0085] A12. Data reporting.

[0086] In this embodiment, the structure optimizer reports the acquired data to the intelligent analysis unit.

[0087] A13. Storage optimization analysis.

[0088] In this embodiment, the intelligent analysis unit performs intelligent analysis, redundancy feature analysis, or storage strategy analysis on the original data to be stored based on the storage strategy analysis model to achieve storage optimization analysis and obtain analysis results. Specifically, the intelligent analysis performed by the intelligent analysis unit may include business redundancy field analysis and data similarity feature analysis. The specific analysis steps include: parsing the data to determine the business to which the data belongs, and matching the data with business-related data optimization rules. Here, the data optimization rules refer to pre-stored redundancy generation strategies or optimization rules. If the data matches a matching data optimization rule, it indicates that the data can be optimized. At this time, a comprehensive decision is made based on a multi-dimensional cost model, including data volatility, rule stability, data reconstruction overhead, and access popularity, to obtain the analysis results of the original data and determine whether the data needs to be optimized. Data volatility refers to whether the data is easy to update or change; rule stability refers to whether the matched data optimization rules change frequently; data reconstruction overhead refers to the cost of computing resources and time for optimizing the data; access popularity refers to the frequency of accessing the data; and the multi-dimensional cost model is a decision-making tool or model that quantifies or evaluates the benefit-cost of data optimization. This multi-dimensional cost model can be a pre-trained convolutional neural network model.

[0089] A14. Determine whether data optimization is needed based on the analysis results.

[0090] In this embodiment, if storage optimization is not required, the analysis result indicates no optimization is needed, and the method terminates directly. If storage optimization is required, the analysis result includes redundancy features, redundancy generation strategies, and optimization types. Optimization types include logical redundancy derivation types and structural commonality extraction types.

[0091] A15, Data Reconstruction.

[0092] In this embodiment, the structure optimizer optimizes or reconstructs the data in the memory database based on the received redundancy features and redundancy generation strategy to obtain optimized data.

[0093] A16. Restore and compare for verification.

[0094] In this embodiment, in order to ensure data consistency, the structure optimizer will also restore the optimized data to the original data and compare the optimized data with the original data to obtain the verification results.

[0095] A17. Based on the verification results, determine whether the optimized data and the original data are consistent.

[0096] If they match, the optimized data is obtained; if they do not match, the optimization of the data is stopped, or the above analysis is repeated.

[0097] Figure 9 A schematic diagram of a data optimization device provided in an embodiment of this application is shown. As shown, the data optimization device 500 may include a redundancy feature analysis module 510, a determination module 520, an optimized data generation module 530, and a data storage module 540.

[0098] The redundancy feature analysis module 510 is used to perform redundancy feature analysis on the original data to be stored based on the storage strategy analysis model in response to the storage event of the original data to be stored, and to determine the redundancy features in the original data to be stored.

[0099] The determination module 520 is used to determine the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy features. The redundancy generation strategy is used to derive the redundant data in the original data to be stored.

[0100] The optimized data generation module 530 is used to generate optimized data based on the relationship between the non-redundant data and the redundancy generation strategy.

[0101] The data storage module 540 is used to store the optimized data in place of the original data to be stored.

[0102] In some embodiments, the redundancy feature is a semantic dependency relationship between multiple original sub-data in the original data to be stored. When determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature, the determining module 520 is specifically used for: Based on the semantic dependency relationship, the non-redundant data in the multiple original sub-data are determined, and a logical redundancy derivation rule for deriving the redundant data is determined, and the logical redundancy derivation rule is used as the redundancy generation strategy.

[0103] In some embodiments, the optimized data includes data content information and tag attributes. When the optimized data generation module 530 generates optimized data based on the relationship between the non-redundant data and the redundancy generation strategy, it is specifically used for: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the logical redundancy derivation type.

[0104] In some embodiments, the redundancy generation strategy further includes the number of times the optimized data references the logical redundancy derivation rule. When the number of times the optimized data references the logical redundancy derivation rule is zero, the data optimization device 500 is further configured to: clear the redundancy generation strategy.

[0105] In some embodiments, the redundancy feature is the structural similarity among multiple original sub-data in the original data to be stored. When determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature, the determining module 520 is specifically used for: Extract the structural commonalities among the multiple original sub-data to generate a common structural template; The difference between the original sub-data and the common structure template is used as the non-redundant data, and the common structure template is determined and used as the redundancy generation strategy.

[0106] In some embodiments, the optimized data includes data content information and tag attributes. When the optimized data generation module 530 generates optimized data based on the relationship between the non-redundant data and the redundancy generation strategy, it is specifically used for: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the structural commonality extraction type.

[0107] In some embodiments, the redundancy generation strategy further includes the number of times the optimized data references the common structure template. When the number of times the optimized data references the common structure template is zero, the data optimization device 500 is further configured to: clear the redundancy generation strategy.

[0108] Figure 10 A schematic diagram of the hardware structure of the data optimization device provided in an embodiment of this application is shown.

[0109] The data optimization device may include a processor 301 and a memory 302 storing computer program instructions.

[0110] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0111] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0112] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0113] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0114] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 2 The data optimization method in the illustrated embodiment.

[0115] In one example, the data optimization device may also include a communication interface 303 and a bus 304. As shown in the figure, the processor 301, memory 302, and communication interface 303 are connected via the bus 304 and communicate with each other.

[0116] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0117] Bus 304 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 304 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0118] This data optimization device can execute the online data traffic billing method described in this application embodiment based on currently blocked spam SMS messages and SMS messages reported by users, thereby achieving a combination of... Figure 2 Describe the data optimization methods.

[0119] Furthermore, in conjunction with the data optimization methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data optimization methods described in the above embodiments.

[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data optimization methods described in the above embodiments.

[0121] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0122] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0123] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0124] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0125] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data optimization method, characterized in that, include: In response to a storage event of the original data to be stored, a redundancy feature analysis is performed on the original data to be stored based on a storage strategy analysis model to determine the redundancy features in the original data to be stored. Based on the redundancy characteristics, a redundancy generation strategy and non-redundant data in the original data to be stored are determined. The redundancy generation strategy is used to derive the redundant data in the original data to be stored. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated. The optimized data is used to replace the original data to be stored.

2. The method according to claim 1, characterized in that, The redundancy feature refers to the semantic dependency relationship between multiple original sub-data in the original data to be stored. The step of determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature includes: Based on the semantic dependencies, non-redundant data is determined among the multiple original sub-data, and logical redundancy derivation rules for deriving the redundant data are determined, and the logical redundancy derivation rules are used as the redundancy generation strategy.

3. The method according to claim 2, characterized in that, The optimized data includes data content information and tag attributes. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated, including: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the logical redundancy derivation type.

4. The method according to claim 3, characterized in that, The redundancy generation strategy also includes the number of times the optimized data references the logical redundancy derivation rule. If the number of times the optimized data references the logical redundancy derivation rule is zero, the method further includes clearing the redundancy generation strategy.

5. The method according to claim 1, characterized in that, The redundancy feature is the structural similarity among multiple original sub-data in the original data to be stored. The step of determining the redundancy generation strategy and the non-redundant data in the original data to be stored based on the redundancy feature includes: Extract the structural commonalities among the multiple original sub-data to generate a common structural template; The difference between the original sub-data and the common structure template is used as the non-redundant data, and the common structure template is determined and used as the redundancy generation strategy.

6. The method according to claim 5, characterized in that, The optimized data includes data content information and tag attributes. Based on the relationship between the non-redundant data and the redundancy generation strategy, optimized data is generated, including: Based on the relationship between the non-redundant data and the redundancy generation strategy, reference information pointing to the redundancy generation strategy corresponding to the non-redundant data is generated; The non-redundant data and the reference information pointing to the redundancy generation strategy are used as the data content information of the optimized data; Set the label attribute of the optimized data to the structural commonality extraction type.

7. The method according to claim 5, characterized in that, The redundancy generation strategy also includes the number of times the optimized data references the common structure template. When the number of times the optimized data references the common structure template is zero, the method further includes clearing the redundancy generation strategy.

8. A data optimization device, characterized in that, The device includes: The redundancy feature analysis module is used to respond to the storage event of the original data to be stored, and perform redundancy feature analysis on the original data to be stored based on the storage strategy analysis model to determine the redundancy features in the original data to be stored. The determination module is used to determine a redundancy generation strategy and non-redundant data in the original data to be stored based on the redundancy features. The redundancy generation strategy is used to derive the redundant data in the original data to be stored. An optimized data generation module is used to generate optimized data based on the relationship between the non-redundant data and the redundancy generation strategy; The data storage module is used to store the optimized data in place of the original data to be stored.

9. A data optimization device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images