Optimization method and device of data writing strategy, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a data writing strategy optimization method and device, equipment and a medium, and the method comprises the steps: carrying out the structural processing of operation data, and obtaining a standard data set; performing feature extraction on the standard data set to obtain a cost feature set; respectively training a cost prediction model and a strategy optimization model based on the cost feature set, and analyzing a target writing strategy generated by the strategy optimization model through the cost prediction model to obtain a mapping relationship between the target writing strategy and the estimated cost; and generating a write-in strategy parameter set according to the mapping relationship, pushing the write-in strategy parameter set to a write-in client to execute a data write-in task, and feeding back operation feedback data generated by an execution result to the cost prediction model and the strategy optimization model to perform strategy optimization. The method and the device can be applied to financial science and technology or medical care service program systems, and can realize automatic optimization of the write-in strategy, reduce the cost and improve the query performance and the system stability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for optimizing data writing strategies. Background Technology

[0002] With the development of big data technology, data lake warehouses have become a core platform for enterprises to store and analyze data. In a data lake warehouse architecture, the data write strategy directly impacts system performance, query efficiency, and storage costs. In related technologies, write strategy configuration primarily relies on the experience of senior engineers, including key parameters such as partitioning strategy, sort key selection, target file size, and write parallelism. However, human experience cannot guarantee optimal performance, and static strategies cannot adapt to dynamic changes in data volume, data distribution, and query patterns, easily leading to performance degradation or increased storage overhead.

[0003] Furthermore, there is a complex trade-off between write performance, query efficiency, and storage costs. It is difficult for humans to accurately assess the long-term impact of different strategies, and localized optimizations may lead to increased maintenance costs such as frequent small file merging. Related technologies lack systematic and intelligent methods for dynamically optimizing write strategies, resulting in high overall ownership costs, significant performance fluctuations, and heavy reliance on manual maintenance for data platforms. These issues limit the scalability and continuous optimization capabilities of data lake warehouses and hinder their ability to support large-scale data analysis and real-time business needs. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for optimizing data writing strategies, in order to solve the technical problems in related technologies that cannot achieve automatic optimization of writing strategies, have high writing costs, and have low query performance and system stability.

[0005] Firstly, an optimization method for a data writing strategy is provided, the method comprising: Operational data is obtained from the data platform and structured to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; Feature extraction is performed on the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; Based on the cost feature set, a cost prediction model and a strategy optimization model are trained respectively. The target writing strategy generated by the strategy optimization model is analyzed through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. A set of writing strategy parameters is generated based on the mapping relationship, and the set of writing strategy parameters is pushed to the writing client to execute the data writing task. The running feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization.

[0006] Secondly, an optimization device for a data writing strategy is provided, comprising: The acquisition module is used to acquire operational data from the data platform and perform structured processing on the operational data to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; The feature extraction module is used to extract features from the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; The analysis module is used to train a cost prediction model and a strategy optimization model based on the cost feature set, and to analyze the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. The optimization module is used to generate a set of writing strategy parameters based on the mapping relationship, push the set of writing strategy parameters to the writing client to execute the data writing task, and feed back the running feedback data generated by the execution result to the cost prediction model and the strategy optimization model for strategy optimization.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data writing strategy optimization method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, it implements the steps of the above-described optimized method for data writing strategy.

[0009] The aforementioned optimization method, apparatus, computer equipment, and storage medium for data writing strategies include the following steps: acquiring operational data from a data platform and performing structured processing on the operational data to obtain a standard data set; wherein the operational data includes write logs, query records, and storage information; extracting features from the standard data set to obtain a cost feature set; wherein the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; training a cost prediction model and a strategy optimization model based on the cost feature set, and analyzing the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain a mapping relationship between the target writing strategy and the estimated cost; generating a writing strategy parameter set based on the mapping relationship, and pushing the writing strategy parameter set to the writing client to execute the data writing task; and feeding back the operational feedback data generated by the execution result to the cost prediction model and the strategy optimization model for strategy optimization. In this invention, by performing structured processing on the operational data and extracting multi-dimensional cost features, the write throughput, query latency features, file data features, and cost occupancy features can be comprehensively reflected, making cost prediction more accurate, thereby balancing performance and resource consumption when generating the writing strategy. By leveraging the collaborative analysis of cost prediction and strategy optimization models, the potential costs of different write strategies can be assessed in real time, establishing a mapping relationship between the target write strategy and the estimated cost, thus achieving scientific and optimized strategy selection. Simultaneously, this application achieves closed-loop optimization by pushing the optimized write strategy parameters to the write client and collecting feedback data. This allows for dynamic adjustment of the strategy based on actual operating results, further reducing system operating costs and resource waste, and improving the reliability and stability of data write tasks. In a big data environment, this method helps improve the overall performance of the data lake warehouse architecture, optimize storage resource utilization, and reduce manual intervention and maintenance costs, making it suitable for industries such as finance and healthcare that are sensitive to data write efficiency and cost. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for an optimization method of data writing strategy in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an optimization method for a data writing strategy in one embodiment of the present invention; Figure 3 yes Figure 2A schematic diagram of a specific implementation method for step S10; Figure 4 This is a schematic diagram of a data writing strategy optimization device in one embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The data writing strategy optimization method provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain operational data from the data platform through the client and perform structured processing on the operational data to obtain a standard data set. This operational data includes write logs, query records, and storage information. Feature extraction is performed on the standard data set to obtain a cost feature set. This cost feature set includes write throughput, query latency features, file data features, and cost occupancy features. A cost prediction model and a strategy optimization model are trained based on the cost feature set. The target write strategy generated by the strategy optimization model is analyzed using the cost prediction model to obtain a mapping relationship between the target write strategy and the estimated cost. A write strategy parameter set is generated based on this mapping relationship and pushed to the write client to execute the data write task. The operational feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization. In this invention, by structuring the operational data and extracting multi-dimensional cost features, write throughput, query latency features, file data features, and cost occupancy features can be comprehensively reflected, making cost prediction more accurate. This allows for a balance between performance and resource consumption when generating write strategies. By leveraging the collaborative analysis of cost prediction and strategy optimization models, the potential costs of different write strategies can be assessed in real time, establishing a mapping relationship between the target write strategy and the estimated cost, thus achieving scientific and optimized strategy selection. Simultaneously, this application achieves closed-loop optimization by pushing the optimized write strategy parameters to the write client and collecting feedback data. This allows for dynamic adjustment of the strategy based on actual operating results, further reducing system operating costs and resource waste, and improving the reliability and stability of data write tasks. In a big data environment, this method helps improve the overall performance of the data lake warehouse architecture, optimize storage resource utilization, and reduce manual intervention and maintenance costs, making it suitable for industries such as finance and healthcare that are sensitive to data write efficiency and cost.

[0014] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a method for optimizing a data writing strategy according to an embodiment of the present invention is shown. The method includes the following steps: S10: Obtain operational data from the data platform and perform structured processing on the operational data to obtain a standard data set.

[0016] The operational data includes written logs, query records, and stored information.

[0017] In this embodiment, the first step is to read operational data from the target data table in the data platform. This operational data includes write logs, query records, and storage information, which guide subsequent cost feature extraction and strategy optimization. For example, write logs reflect the write behavior of the target data table in actual business scenarios. In the financial sector, such as auto insurance, operations like reporting vehicle accidents, uploading damage assessments, and submitting claims generate sudden write flows during peak hours, resulting in fluctuating write batch sizes, write concurrency, and write rates. In the healthcare insurance sector, data synchronization for physical examinations, uploading of claim details, and filing of image metadata also create concentrated write pressure during the daytime, affecting write throughput characteristics. Furthermore, query records reflect data access patterns in downstream analysis or reporting scenarios. For instance, auto insurance pricing model training involves high-frequency scanning of historical accident records, and health insurance risk control analysis involves multi-condition filtering of claim details. These operations lead to differences in query predicate combinations, hotspot column distribution, and scanning range, resulting in query latency characteristics. Storage information is used to describe the file size distribution, storage usage, and access bandwidth in the target execution environment in order to determine file data characteristics and resource usage characteristics.

[0018] For example, the first step is to obtain operational data, including write logs, query records, and storage information, from the data platform of the data lake warehouse architecture for the target data table. Since this operational data may originate from different log components, monitoring systems, or multi-tenant nodes, it may have inconsistent formats, different time granularities, and missing fields. Therefore, it cannot be directly used for model training and requires a structured processing step to construct a unified standard dataset. For example, structured processing may include timestamp alignment, field standardization, noise record filtering, unified indicator encoding, and business event mapping. This is used to organize heterogeneous data into a standardized data structure with a unified field system, unified sampling period, and unified business semantics according to preset rules, thereby ensuring the accuracy of subsequent cost feature extraction.

[0019] For example, in the auto insurance business in the financial sector, the log formats generated by accident reporting, damage assessment uploading, and claim submission in different systems are often inconsistent. Through structured processing, these logs can be unified into a standard event structure, and features such as write batch size, write rate, and concurrency can be extracted based on this standard structure. In the healthcare sector, records of physical examination data synchronization, medical image metadata reporting, and claim details uploading may contain redundant fields and duplicate reports. Through structured processing, these records can be converted into a unified health insurance data event structure, thereby facilitating the extraction of stable and reusable data write rhythms.

[0020] Among them, such as Figure 3As shown, step S10, which involves structuring the running data to obtain a standard data set, includes the following steps: S11: Perform data cleaning operations on the running data to obtain a set of structured fields.

[0021] S12: Based on the structured field set, perform association matching on the write link, query link and storage link to obtain the associated data set.

[0022] The query endpoint feature data includes query scan range, query selectivity, and query latency indicators.

[0023] S13: Construct a primary indicator set based on the associated data set.

[0024] The primary indicator set is used to describe the operational behavior of the data platform.

[0025] S14: Standardize and normalize the primary index set to obtain the standard data set.

[0026] For example, step S11 can be used to clean the running data to address issues such as missing fields, dirty data, duplicate records, and inconsistent formats in the original logs. During data cleaning, operations such as timestamp completion, abnormal record filtering, field standardization, and event splitting can be performed based on preset rules to unify write logs, query records, and storage information from different sources into a structured field set. Subsequently, step S12 can be used to correlate and match the write, query, and storage links based on the structured field set. Through time alignment, primary key mapping, event ID association, and resource usage analysis, the originally scattered link information is integrated into a correlated data set. This ensures that each write or query action can establish a one-to-one logical relationship with its corresponding file generation record, storage resource consumption, and query end characteristics (including query scan range, query selectivity, and query latency indicators), thereby ensuring the completeness and semantic consistency of subsequent indicator construction.

[0027] After obtaining the associated data set, step S13 constructs a primary indicator set based on this set. This set is used to characterize the core behavioral patterns of the data platform during operation, such as write batch size, file generation rate, query filtering efficiency, small file generation frequency, and storage bandwidth usage trends. This allows the system to express the dynamic characteristics of the operational chain in an index-based manner. To ensure that different indicators have a unified dimension and comparability, step S14 performs standardization and normalization processing on the primary indicator set, including range scaling, outlier reduction, distribution transformation, and feature alignment. This transforms the set into a standard data set with a unified value range, stable distribution, and consistent statistical semantics. The resulting standard data set can be directly used as the input basis for subsequent cost feature extraction, model training, and policy inference, thereby ensuring the accuracy and scalability of the overall optimization process.

[0028] S20: Extract features from the standard dataset to obtain a cost feature set.

[0029] The cost feature set includes write throughput, query latency features, file data features, and cost occupancy features.

[0030] In some embodiments, the step of extracting features from the standard dataset to obtain a cost feature set includes: extracting the amount of data written per unit time from the standard dataset to obtain the write throughput; determining the time difference from initiating a request to obtaining a result in a query record based on the standard dataset to obtain the query latency feature; determining the number of files and file distribution information generated by the write task based on the standard dataset to obtain the file data feature; and determining the storage resource consumption cost per unit amount of data based on the storage information in the standard dataset to obtain the cost occupancy feature.

[0031] For example, feature extraction operations can be performed based on standardized datasets to construct a cost feature set that comprehensively reflects the operating costs of the data platform. These standardized datasets have been linked and semantically unified, thus they can be directly used to calculate key operational indicators. On the write side, the write throughput feature can be formed by extracting the amount of data written per unit time based on effective write events within a continuous time window. In the financial sector, large-scale instantaneous write volumes occur during peak periods for services such as auto insurance claims, health insurance claims material uploads, and pension account settlements. Similarly, in healthcare and elderly care scenarios, periodic write pressure arises when physical examination centers synchronize test results in batches, elderly care institutions upload residency records, and hospitals synchronize medical details at the end of the day. These differences can all be quantified using write throughput features. On the query side, query latency features can be obtained by analyzing the request-response time difference in the query chain. For instance, insurance companies need to frequently scan historical loss samples during auto insurance pricing model training, medical risk control performs high-dimensional filtering on medical visit details, and elderly care service platforms need to perform cross-period analysis on residency records spanning long periods. The relevant query characteristics can all be reflected through query latency features. On the file generation side, the number of files and file structure distribution can be extracted based on the file write behavior corresponding to the write task. This is applicable to typical scenarios such as the generation of a large number of intermediate files in the financial sector at the end of the day and the generation of high-frequency small files in the medical and elderly care sectors.

[0032] Furthermore, in terms of resource cost, the storage resource consumption cost per unit of data volume can be extracted from the resource usage information in the standard dataset, including CPU cycle usage, memory curve changes, storage bandwidth utilization, and I / O request density, thus forming cost occupancy characteristics. For example, in the claims center scenario of a large insurance company, batch archiving writes will consume a large amount of storage bandwidth in a short period of time; in medical and health data processing, image metadata and structured medical records often jointly drive I / O peaks; in long-term care and elderly care service scenarios, although the daily record volume is relatively stable, the long-term retention requirements make storage layer compression and merging strategies particularly sensitive to cost performance. By jointly extracting four types of indicators—write throughput, query latency characteristics, file data characteristics, and cost occupancy characteristics—this step can comprehensively characterize the operational characteristics of target data tables in different financial and medical elderly care scenarios from four dimensions: write behavior, query behavior, file form, and resource cost, providing a structured, quantifiable, and reasonable input foundation for subsequent cost prediction model training and dynamic strategy optimization.

[0033] S30: Train a cost prediction model and a strategy optimization model based on the cost feature set, and analyze the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost.

[0034] For example, a cost prediction model and a strategy optimization model are trained separately based on a set of cost features. By using multi-dimensional write throughput features, query latency features, file data features, and cost occupancy features as joint inputs, the cost prediction model can accurately characterize the functional relationship between strategy selection and system resource consumption under different business scenarios. Simultaneously, the strategy optimization model uses the same set of features to search the feasible strategy space and generate candidate target write strategies. After candidate strategies are generated, the cost prediction model estimates the cost of each target write strategy, including resource cost assessments in scenarios such as peak write times for auto insurance in the financial sector, multi-insurance pricing training, and batch settlement of pension funds, as well as the actual load overhead brought about by health data synchronization, medical record queries, and pension file accumulation in healthcare and elderly care scenarios. Finally, by mapping the target write strategies to their corresponding estimated costs, a strategy-cost correlation is formed, providing a quantifiable decision-making basis for the subsequent automatic selection of the optimal strategy.

[0035] In some embodiments, the step of training a cost prediction model and a strategy optimization model based on the cost feature set, and analyzing the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost, includes: using the cost feature set as training samples and inputting it into a preset regression model for training to obtain the cost prediction model; combining the cost feature set with candidate writing strategies and inputting them into a preset reinforcement learning model for training to obtain the strategy optimization model; wherein, the strategy optimization model is used to output the target writing strategy; using the cost prediction model, analyzing the multiple target writing strategies output by the strategy optimization model to obtain the cost prediction result corresponding to each target writing strategy; and constructing the mapping relationship between the target writing strategy and the estimated cost based on each target writing strategy and the corresponding cost prediction result.

[0036] For example, a cost feature set, consisting of write throughput, query latency characteristics, file data characteristics, and cost occupancy characteristics, can be used as training samples and input into a pre-defined regression model for training. This yields a cost prediction model that can characterize the relationship between changes in write strategies and system resource consumption. During training, the cost prediction model learns the mapping patterns between feature patterns and cost performance under different business scenarios. For instance, in the financial sector, peak writes in auto insurance lead to sudden increases in bandwidth usage; high-frequency query latency fluctuations caused by health insurance risk control training; or in the medical and elderly care scenario, the resource load differences brought about by batch synchronization of physical examination data and the growth of residential record storage. Simultaneously, the cost feature set and pre-defined candidate write strategies are input into a reinforcement learning model for training. The model simulates the impact of policy adjustments on system behavior and continuously updates policy parameters based on reward signals, ultimately obtaining a policy optimization model that can proactively output target write strategies, thereby achieving efficient search and dynamic generation of the policy space.

[0037] Furthermore, after training both models, the cost prediction model is used to infer the costs of multiple target write strategies output by the strategy optimization model, obtaining the cost prediction results for each target write strategy. The predictions include not only data distribution and file generation costs along the write path, but also multi-dimensional indicators such as latency consumption during the query process and CPU and storage bandwidth usage trends at the resource level. For example, it can analyze the differences in computational resource costs of different strategies in a large-scale pricing training scenario for insurance companies, and also assess the cumulative impact of long-term data accumulation on storage costs in healthcare and elderly care scenarios. Finally, by associating each target write strategy with its corresponding cost prediction results, a mapping relationship between the target write strategy and the estimated cost is constructed, providing a data-driven decision-making basis for automatically selecting the optimal strategy and achieving a dynamic balance between performance and cost.

[0038] S40: Generate a set of writing strategy parameters based on the mapping relationship, and push the set of writing strategy parameters to the writing client to execute the data writing task. Feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization.

[0039] For example, a corresponding set of write strategy parameters can be automatically generated based on the mapping relationship between the target write strategy and the estimated cost. This set includes key parameters affecting performance and cost, such as partitioning strategy, sorting key, file size threshold, write concurrency, and buffer configuration. This set of strategy parameters is then pushed to the write client for actual data write tasks. During execution, the write client performs operations such as data persistence, file generation, partition organization, and resource scheduling according to the issued strategy, generating operational feedback data including write throughput, query latency changes, file distribution, and resource consumption. This feedback data is fed back in real time to the cost prediction model and the strategy optimization model to calibrate prediction errors, update strategy parameters, and optimize decision-making behavior. This allows the model to continuously adapt to dynamic loads under peak financial business conditions (such as high concurrency in auto insurance claims and batch processing of health insurance claims) and medical and elderly care scenarios (such as centralized synchronization of physical examination data and continuous accumulation of long-term care records), achieving closed-loop self-learning and continuous reinforcement of the write strategy in a real environment.

[0040] In some embodiments, generating a write strategy parameter set based on the mapping relationship and pushing the write strategy parameter set to the write client to execute the data write task includes: determining the target write strategy with the lowest estimated cost as the optimal write strategy based on the mapping relationship, and determining the partition granularity, sorting sequence configuration, file capacity, and parallelism parameters of the optimal write strategy as the write strategy parameter set; performing parameter consistency verification on the write strategy parameter set to obtain an executable strategy parameter set after successful verification; and pushing the executable strategy parameter set to the write client to execute the data write task to obtain the execution result.

[0041] For example, firstly, based on the mapping relationship between the target write strategy and the estimated cost, the cost prediction results of all candidate strategies can be compared, and the target write strategy with the lowest estimated cost can be automatically determined as the optimal write strategy in the current environment. Then, core control parameters such as partition granularity, sorting sequence configuration, file size threshold, and write parallelism are extracted from the optimal strategy to form the proposed write strategy parameter set. To ensure that the strategy can be stably executed in a real business environment, the parameter set can be further validated for consistency. This includes checking whether the partition granularity is compatible with the data's temporal distribution and business primary key model, whether the sorting sequence configuration conforms to the access patterns of query hot columns, whether the file size setting matches storage bandwidth constraints, and whether the parallelism meets the resource isolation requirements common in financial, medical, and elderly care scenarios, to obtain a valid executable strategy parameter set. Finally, the validated executable strategy parameter set is pushed to the write client, enabling the client to execute real data write tasks according to the strategy and generate execution results, providing a reliable operational feedback basis for subsequent strategy optimization.

[0042] In some embodiments, feeding back the operational feedback data generated by the execution results to the cost prediction model and the strategy optimization model for strategy optimization includes: parsing the operational feedback data to obtain a set of writing indicators; wherein the set of writing indicators includes the number of files generated and changes in query response; comparing each writing indicator in the set of writing indicators with a corresponding preset indicator to obtain execution deviation data; analyzing the execution deviation data through a strategy execution evaluation model to obtain an execution evaluation result; and updating the parameters of the cost prediction model and the strategy optimization model based on the execution evaluation result for strategy optimization.

[0043] For example, to achieve continuous iterative optimization of the write strategy, the runtime feedback data generated after the write client executes the task can be structured and parsed to extract a set of write indicators that can reflect the actual execution quality, such as the number of files generated and changes in query response. Each write indicator is then compared with its corresponding preset indicator to obtain execution deviation data used to measure the degree of deviation from the strategy execution.

[0044] Subsequently, the strategy execution evaluation model analyzes the effectiveness of the strategy in real business environments based on execution deviation data. This includes assessing whether batch writes in financial scenarios are excessively fragmented, whether pricing training queries experience additional latency, and whether abnormal small files are generated during physical examination data writing and cross-period jitter occurs during residency record queries in healthcare and elderly care scenarios. This results in a quantitative execution evaluation. Ultimately, the parameters of the cost prediction model and strategy optimization model can be dynamically updated based on these evaluation results. This allows the models to automatically absorb behavioral characteristics from real-world scenarios, such as peak fluctuations in financial business and mixed steady-state and sudden loads of medical and elderly care data, enabling continuous adaptive optimization of the write strategy.

[0045] In some embodiments, the method further includes: acquiring training samples and a preset regression model; wherein the training samples include the cost feature set; labeling the training samples to obtain labeling results, wherein the labeling results include the actual costs corresponding to the cost feature set; and training the preset regression model using the training samples and the labeling results to obtain the cost prediction model.

[0046] For example, to construct a cost prediction model that accurately characterizes the relationship between write strategies and resource consumption, a training sample and a pre-defined regression model are first obtained. The training sample consists of a set of cost features, including multi-dimensional indicators reflecting the system's operating status such as write throughput, query latency features, file data features, and cost occupancy features. Subsequently, the training samples are labeled, supplementing each set of cost features with actual cost values ​​generated during real-world execution. Examples include storage bandwidth consumption during peak writes for auto insurance in the financial sector, computational resource consumption due to health insurance risk control queries, or long-term storage costs caused by physical examination data synchronization and elderly care record accumulating in healthcare and elderly care scenarios. These actual costs serve as supervisory signals constituting the labeled results. Finally, the pre-defined regression model is trained using the training samples and corresponding labeled results, enabling the model to learn the mapping relationship between feature patterns and actual costs. This yields a cost prediction model capable of stably predicting resource consumption under different business loads, providing a reliable basis for subsequent cost evaluation and optimization of write strategies.

[0047] As can be seen, the above scheme, by structuring the operational data and extracting multi-dimensional cost features, can comprehensively reflect write throughput, query latency characteristics, file data characteristics, and resource cost occupancy characteristics, making cost prediction more accurate. This allows for a balance between performance and resource consumption when generating write strategies. By utilizing the collaborative analysis of the cost prediction model and the strategy optimization model, the potential costs of different write strategies can be evaluated in real time, forming a mapping relationship between the target write strategy and the estimated cost, achieving scientific and optimized strategy selection. Simultaneously, this application achieves closed-loop optimization by pushing the optimized write strategy parameters to the write client and collecting feedback data. This allows for dynamic adjustment of the strategy based on actual operational results, further reducing system operating costs and resource waste, and improving the reliability and stability of data write tasks. In a big data environment, this method helps improve the overall performance of the data lake warehouse architecture, optimize storage resource utilization, and reduce manual intervention and maintenance costs. It is suitable for industry scenarios such as finance and healthcare that are sensitive to data write efficiency and cost.

[0048] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0049] In one embodiment, a data writing strategy optimization apparatus is provided, which corresponds one-to-one with the data writing strategy optimization methods in the above embodiments. For example... Figure 4 As shown, the optimization device for this data writing strategy includes an acquisition module 101, a feature extraction module 102, an analysis module 103, and an optimization module 104. Detailed descriptions of each functional module are as follows: The acquisition module 101 is used to acquire operational data from the data platform and perform structured processing on the operational data to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; The feature extraction module 102 is used to extract features from the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; Analysis module 103 is used to train a cost prediction model and a strategy optimization model based on the cost feature set, and to analyze the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. The optimization module 104 is used to generate a set of writing strategy parameters based on the mapping relationship, push the set of writing strategy parameters to the writing client to execute the data writing task, and feed back the running feedback data generated by the execution result to the cost prediction model and the strategy optimization model for strategy optimization.

[0050] The acquisition module 101 is used to perform data cleaning operations on the running data to obtain a structured field set; based on the structured field set, the write link, query link and storage link are matched to obtain a related data set; a primary indicator set is constructed based on the related data set; wherein the primary indicator set is used to describe the running behavior of the data platform; the primary indicator set is standardized and normalized to obtain the standard data set.

[0051] The feature extraction module 102 is used to extract the amount of data written per unit time from the standard data set to obtain the write throughput; and to determine the time difference from initiating a request to obtaining a result in the query record based on the standard data set to obtain the query latency feature; and to determine the number of files and file distribution information generated by the write task based on the standard data set to obtain the file data feature; and to determine the storage resource consumption cost per unit amount of data based on the storage information in the standard data set to obtain the cost occupancy feature.

[0052] Analysis module 103 is used to input the cost feature set as training samples into a preset regression model for training to obtain the cost prediction model; input the cost feature set and candidate writing strategies into a preset reinforcement learning model for training to obtain the strategy optimization model; wherein, the strategy optimization model is used to output the target writing strategy; using the cost prediction model, the multiple target writing strategies output by the strategy optimization model are analyzed to obtain the cost prediction result corresponding to each target writing strategy; based on each target writing strategy and the corresponding cost prediction result, a mapping relationship between the target writing strategy and the estimated cost is constructed.

[0053] The optimization module 104 is used to determine the target write strategy with the lowest estimated cost as the optimal write strategy based on the mapping relationship, and to determine the partition granularity, sorting sequence configuration, file capacity and parallelism parameters of the optimal write strategy as the write strategy parameter set; to perform parameter consistency verification on the write strategy parameter set to obtain an executable strategy parameter set after successful verification; and to push the executable strategy parameter set to the write client to execute the data write task and obtain the execution result.

[0054] The optimization module 104 is used to parse the operation feedback data to obtain a set of writing indicators; wherein, the set of writing indicators includes the number of files generated and changes in query response; each writing indicator in the set of writing indicators is compared with the corresponding preset indicator to obtain execution deviation data; the execution deviation data is analyzed through a strategy execution evaluation model to obtain an execution evaluation result; based on the execution evaluation result, the parameters of the cost prediction model and the strategy optimization model are updated to perform strategy optimization.

[0055] In one embodiment, the acquisition module 101 is configured to: acquire training samples and a preset regression model; wherein the training samples include the cost feature set; label the training samples to obtain labeling results, wherein the labeling results include the actual costs corresponding to the cost feature set; and train the preset regression model using the training samples and the labeling results to obtain the cost prediction model.

[0056] This invention provides a data write strategy optimization device that can perform structured processing on runtime data and extract multi-dimensional cost features. This comprehensively reflects write throughput, query latency characteristics, file data characteristics, and resource consumption characteristics, making cost prediction more accurate. This allows for a balance between performance and resource consumption when generating write strategies. By utilizing the collaborative analysis of a cost prediction model and a strategy optimization model, the potential costs of different write strategies can be evaluated in real time, forming a mapping relationship between the target write strategy and the estimated cost, achieving scientific and optimized strategy selection. Simultaneously, this application achieves closed-loop optimization by pushing the optimized write strategy parameters to the write client and collecting feedback data. This allows for dynamic adjustment of the strategy based on actual operating results, further reducing system operating costs and resource waste, and improving the reliability and stability of data write tasks. In a big data environment, this method helps improve the overall performance of the data lake warehouse architecture, optimize storage resource utilization, and reduce manual intervention and maintenance costs. It is suitable for industries such as finance and healthcare that are sensitive to data write efficiency and cost.

[0057] Specific limitations regarding the optimization device for the data writing strategy can be found in the limitations of the data writing strategy optimization method described above, and will not be repeated here. Each module in the aforementioned data writing strategy optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0058] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements a data writing strategy optimization method, server-side functions or steps.

[0059] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an optimized data writing strategy.

[0060] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Operational data is obtained from the data platform and structured to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; Feature extraction is performed on the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; Based on the cost feature set, a cost prediction model and a strategy optimization model are trained respectively. The target writing strategy generated by the strategy optimization model is analyzed through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. A set of writing strategy parameters is generated based on the mapping relationship, and the set of writing strategy parameters is pushed to the writing client to execute the data writing task. The running feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization.

[0061] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Operational data is obtained from the data platform and structured to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; Feature extraction is performed on the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; Based on the cost feature set, a cost prediction model and a strategy optimization model are trained respectively. The target writing strategy generated by the strategy optimization model is analyzed through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. A set of writing strategy parameters is generated based on the mapping relationship, and the set of writing strategy parameters is pushed to the writing client to execute the data writing task. The running feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization.

[0062] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for optimizing a data writing strategy, characterized in that, The method includes: Operational data is obtained from the data platform and structured to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; Feature extraction is performed on the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; Based on the cost feature set, a cost prediction model and a strategy optimization model are trained respectively. The target writing strategy generated by the strategy optimization model is analyzed through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. A set of writing strategy parameters is generated based on the mapping relationship, and the set of writing strategy parameters is pushed to the writing client to execute the data writing task. The running feedback data generated by the execution result is fed back to the cost prediction model and the strategy optimization model for strategy optimization.

2. The method according to claim 1, characterized in that, The process of structuring the operational data to obtain a standard data set includes: The running data is cleaned to obtain a set of structured fields; Based on the structured field set, the write link, query link, and storage link are matched to obtain the associated data set; A primary indicator set is constructed based on the associated data set; wherein, the primary indicator set is used to describe the operational behavior of the data platform; The primary index set is standardized and normalized to obtain the standard data set.

3. The method according to claim 1, characterized in that, The step of extracting features from the standard dataset to obtain a cost feature set includes: The write throughput is obtained by extracting the amount of data written per unit time from the standard dataset; and, Based on the standard dataset, the time difference between initiating a request and obtaining a result in the query records is determined to obtain the query latency characteristic; and... Based on the standard dataset, the number and distribution information of files generated by the write task are determined, thus obtaining the file data characteristics; and... Based on the storage information in the standard dataset, the storage resource consumption cost per unit of data is determined, thus obtaining the cost occupancy characteristic.

4. The method according to claim 1, characterized in that, The process involves training a cost prediction model and a strategy optimization model based on the cost feature set, and then analyzing the target writing strategy generated by the strategy optimization model using the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost, including: The cost feature set is used as a training sample and input into a preset regression model for training to obtain the cost prediction model; The cost feature set and candidate writing strategies are combined and input into a preset reinforcement learning model for training to obtain the policy optimization model; wherein, the policy optimization model is used to output the target writing strategy; Using the cost prediction model, the multiple target writing strategies output by the strategy optimization model are analyzed to obtain the cost prediction result corresponding to each target writing strategy; Based on each target write strategy and the corresponding cost prediction result, a mapping relationship between the target write strategy and the estimated cost is constructed.

5. The method according to claim 1, characterized in that, The step of generating a set of write strategy parameters based on the mapping relationship and pushing the set of write strategy parameters to the write client to execute the data write task includes: Based on the mapping relationship, the target write strategy with the lowest estimated cost is determined as the optimal write strategy, and the partition granularity, sorting sequence configuration, file capacity and parallelism parameters of the optimal write strategy are determined as the write strategy parameter set; Perform parameter consistency verification on the written strategy parameter set to obtain the executable strategy parameter set after successful verification. The executable strategy parameter set is pushed to the writing client to execute the data writing task and obtain the execution result.

6. The method according to claim 1, characterized in that, The step of feeding back the operational feedback data generated from the execution results to the cost prediction model and the strategy optimization model for strategy optimization includes: The operation feedback data is parsed to obtain a set of writing metrics; wherein, the set of writing metrics includes the number of files generated and changes in query response. Each write indicator in the write indicator set is compared with its corresponding preset indicator to obtain execution deviation data; The execution deviation data is analyzed using a strategy execution evaluation model to obtain the execution evaluation results; Based on the performance evaluation results, the parameters of the cost prediction model and the strategy optimization model are updated to optimize the strategy.

7. The method according to claim 1, characterized in that, The method further includes: Obtain training samples and a preset regression model; wherein, the training samples include the cost feature set; The training samples are labeled to obtain the labeling results, wherein the labeling results include the actual costs corresponding to the cost feature set; The cost prediction model is obtained by training the preset regression model using the training samples and the labeled results.

8. An optimization device for a data writing strategy, characterized in that, include: The acquisition module is used to acquire operational data from the data platform and perform structured processing on the operational data to obtain a standard data set; wherein, the operational data includes written logs, query records and storage information; The feature extraction module is used to extract features from the standard dataset to obtain a cost feature set; wherein, the cost feature set includes write throughput, query latency features, file data features, and cost occupancy features; The analysis module is used to train a cost prediction model and a strategy optimization model based on the cost feature set, and to analyze the target writing strategy generated by the strategy optimization model through the cost prediction model to obtain the mapping relationship between the target writing strategy and the estimated cost. The optimization module is used to generate a set of writing strategy parameters based on the mapping relationship, push the set of writing strategy parameters to the writing client to execute the data writing task, and feed back the running feedback data generated by the execution result to the cost prediction model and the strategy optimization model for strategy optimization.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the optimized method for the data writing strategy as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the optimized method for the data writing strategy as described in any one of claims 1 to 7.

Images