Data processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining table data in a data warehouse, the table data comprising access data, scheduling data and dependency data; inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; a target life cycle is determined according to the life cycle coefficient and an initial life cycle, and the initial life cycle is determined according to table data; and processing the table data according to the target life cycle. The technical problem of storage resource waste caused by the fact that the life cycle of the data cannot be accurately determined in the data management process is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing method, apparatus, electronic device, and readable storage medium. Background Art

[0002] With the accelerating pace of the transformation of the automotive industry towards electrification, intelligence, and networking, the large amount of big data generated in the vehicle networking field is showing an explosive growth, which directly drives the sharp increase in the demand for data warehouses (hereinafter referred to as "data warehouses"). In a data warehouse, there are various data structures such as fact tables, intermediate process tables, and temporary storage tables. If there is a lack of scientific and reasonable data life cycle management and the outdated data is not cleared in time, it will inevitably lead to the disorderly expansion of the data accumulation volume. This phenomenon not only significantly increases the complexity of data warehouse management but also wastes valuable storage resources unnecessarily, having a negative impact on the system performance.

[0003] To address this challenge, we urgently need innovative data life cycle management strategies to achieve effective data cleaning and reasonable resource allocation, ensure the efficient operation of the data warehouse, and at the same time reduce unnecessary storage and computing costs. Among the existing data management methods, for the setting of the life cycle, it is proposed that the life cycle of the corresponding data slice can be considered for the time-slice data that has been accessed, but the specific calculation method of the life cycle is not involved. This method cannot accurately determine the life cycle of the data, and an unreasonable life cycle will lead to waste of data warehouse storage resources.

[0004] Therefore, in the process of data management, how to accurately determine the life cycle of data and avoid waste of storage resources is one of the important technical problems in the related technical field. For the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, apparatus, electronic device, and readable storage medium, aiming to improve the problem of waste of storage resources caused by the inability to accurately determine the life cycle of data in the data management process.

[0006] According to one aspect of the embodiments of the present invention, a data processing method is provided, including: obtaining table data in a data warehouse, where the table data includes access data, scheduling data, and dependency data; inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; determining a target life cycle according to the life cycle coefficient and an initial life cycle, where the initial life cycle is determined based on the table data; and processing the table data according to the target life cycle.

[0007] The above data processing method provided by the embodiments of the present application achieves the following technical effects: By obtaining the access, scheduling, and dependency data of various tables in the data warehouse, it is possible to systematically understand the actual application value and dynamic requirements of each table. Inputting these table data into a pre-trained life cycle coefficient recommendation model, the model, based on data analysis and machine learning techniques, can accurately evaluate the first life cycle coefficient of different tables. This is equivalent to providing a personalized and forward-looking assessment of the storage requirements for each table. Combining the initial life cycle of each table, we can calculate a more accurate target life cycle. This process realizes the dynamic adjustment and optimization of data life cycle management to adapt to the real-time changes of data in the data warehouse and the evolution of business requirements. Processing the table data based on the accurate target life cycle achieves the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solves the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0008] Optionally, the construction of the life cycle coefficient recommendation model includes: obtaining training data; screening out a target table set from the training data according to a preset screening rule; determining the second life cycle coefficient of the target table set according to the business requirements of the target table set; using the second life cycle coefficient and multi-dimensional data as sample data to train a pre-constructed initial model to obtain a life cycle coefficient recommendation model, where the multi-dimensional data is included in the training data, and the multi-dimensional data includes access dimension data, scheduling dimension data, and dependency dimension data.

[0009] The above optional embodiments of the present application can achieve the following technical effects: First, based on a large amount of training data sets, a representative table set closely related to business requirements, that is, the target table set, is selected through a preset screening rule. According to the specific business requirements of the target table set, the second life cycle coefficient is set, which represents the storage cycle optimization target based on business judgment. Using these second life cycle coefficients and the multi-dimensional data of the target table set (including key information such as access frequency, scheduling cycle, and upstream and downstream dependency relationships) as sample data for model training, through iterative training, the model gradually learns and masters the optimization rules of the storage cycle of tables under different business backgrounds. Finally, the trained life cycle coefficient recommendation model can automatically calculate the first life cycle coefficient based on the input table data, providing personalized and forward-looking storage cycle suggestions for each table. The use of this model simplifies the complexity of data life cycle management, reduces the need for professional intervention, and at the same time ensures the scientificity and rationality of data storage strategies.

[0010] Optionally, the determination of the initial life cycle includes: obtaining the level of the table in the data warehouse in the table data; determining the initial life cycle of the table according to the level.

[0011] The above optional embodiments of the present application can achieve the following technical effects: The determination of the initial life cycle depends on the position of the table in the data warehouse hierarchy. By identifying the hierarchy of the table, its role and importance in the overall data analysis process can be determined. Tables at different levels may carry different types of data and business logics. Therefore, their initial life cycle settings should also be different. This hierarchical life cycle setting method ensures that the data warehouse can manage its storage resources strategically.

[0012] Optionally, managing the table data according to the target life cycle includes: obtaining the current retention period of the table in the table data; comparing the current retention period with the target life cycle to obtain a first comparison result; in response to the first comparison result indicating that the current retention period is greater than the target life cycle, deleting the expired data slices in the table, where the expired data slices are data slices whose slice retention period is greater than the target life cycle.

[0013] The above optional embodiments of the present application can achieve the following technical effects: Continuously monitor the current retention period of each table in the data warehouse, that is, the continuous storage time of the data from generation to the current moment. By accurately comparing the current retention period of the table with the target life cycle, the system can generate a first comparison result, which directly reflects whether the actual storage time of the table data exceeds its due life cycle. When the first comparison result indicates that the current retention period of a certain table has exceeded its target life cycle, the system will automatically trigger the data cleaning process to delete the data slices in the table that exceed the retention period. This automatic deletion mechanism based on the target life cycle avoids redundant storage of data, effectively releases storage space, and reduces storage costs. At the same time, it also reduces the workload of data warehouse maintenance personnel and improves the automation level and efficiency of data management.

[0014] Optionally, the data processing method further includes: in response to the current retention period being a preset retention period, deleting the table and taking the scheduling task corresponding to the table offline at the same time.

[0015] The above optional embodiments of the present application can achieve the following technical effects: The strategy of completely deleting the table and taking the relevant scheduling task offline at the same time in response to the current retention period reaching the preset retention period ensures the timely cleaning of useless data in the data warehouse, avoids the storage space being occupied by data that has not been updated for a long time and is no longer needed, thereby significantly improving the utilization rate of storage resources and reducing unnecessary storage costs. At the same time, by taking the scheduling tasks related to the deleted data tables offline, the system can automatically stop further processing and calculation of these tables, effectively saving computing power resources, reducing unnecessary computing overhead, and improving the operating efficiency and economy of the entire system.

[0016] Optionally, managing the table data according to the target life cycle includes: comparing the target life cycle with the expected life cycle to obtain a second comparison result, where the expected life cycle is determined according to a preset cycle determination rule, access data, scheduling data, and dependent data; in response to the second comparison result indicating that the difference between the expected life cycle and the target life cycle is greater than a preset difference, hierarchically process the table data according to the target life cycle and the expected life cycle.

[0017] The above optional embodiment of the present application can achieve the following technical effects: when the system compares the target life cycle with the expected life cycle, that is, the actual storage cycle and the business-expected storage cycle, and the second comparison result shows that the difference between the two exceeds the preset difference threshold, the system will automatically trigger the hierarchical processing of the table data, providing a flexible hierarchical data storage solution for the data warehouse.

[0018] Optionally, hierarchically processing the table data according to the target life cycle and the expected life cycle includes: obtaining the current retention period of the table in the table data; in response to the current retention period being greater than the target life cycle and less than the expected life cycle, storing the table in the cold data layer, where the cold data layer is used to store data with an access frequency lower than a preset frequency threshold; or, in response to the current retention period being less than or equal to the target life cycle, storing the table in the hot data layer, where the hot data layer is used to store data with an access frequency higher than a preset frequency threshold.

[0019] The above optional embodiment of the present application can achieve the following technical effects: the system will automatically obtain the current retention period of each table. When the retention period exceeds the target life cycle but still has not reached the longer life cycle expected by the business, it means that although the data has a low usage frequency at present, it still retains certain business value and access possibilities. Therefore, the system identifies this type of data as "cold data" and migrates it from the original hot storage layer to a dedicated cold data layer for storage. For those tables with a current retention period less than or equal to the target life cycle, the system determines them as "hot data", that is, data with high-frequency access and high business value, and these data will be stored in the hot data layer. Through the above intelligent hierarchical mechanism, the data warehouse can accurately allocate resources according to the life cycle and access characteristics of the data, improve the overall efficiency and economy of data storage, ensure business continuity and data usability at the same time, and provide strong support for data-driven decision-making.

[0020] Optionally, the data processing method further includes: using the target life cycle, access data, scheduling data, and dependent data as training data to update the life cycle coefficient recommendation model to obtain an updated life cycle coefficient recommendation model.

[0021] The above optional embodiments of the present application can achieve the following technical effects: By continuously using the target life cycle, access data, scheduling data, and dependency data as feedback training data to update and optimize the life cycle coefficient recommendation model, the self-evolution of the model and the improvement of the intelligent decision-making ability are realized.

[0022] According to another aspect of the present application, there is also provided a data processing device, including: an acquisition module, configured to acquire table data in a data warehouse, where the table data includes access data, scheduling data, and dependency data; a recommendation module, configured to input the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; a determination module, configured to determine a target life cycle according to the life cycle coefficient and an initial life cycle, where an initial life cycle is corresponding to the table in the table data; and a processing module, configured to manage the table data according to the target life cycle.

[0023] The above data processing device provided by the embodiments of the present application achieves the following technical effects: The acquisition module collects the access data, scheduling data, and dependency data of all tables in the data warehouse. The recommendation module inputs the acquired table data into the pre-trained life cycle coefficient recommendation model, analyzes the characteristics and patterns of the data using machine learning algorithms, and outputs a first life cycle coefficient. The determination module calculates the target life cycle, that is, the optimal storage period that each table should maintain in the current business environment, according to the recommended first life cycle coefficient and the initial life cycle preset for each table. The processing module is responsible for executing the storage management decision based on the target life cycle. Through the close cooperation among the above modules, the target life cycle can be accurately calculated, the automation and intelligence of data life cycle management are realized, the storage efficiency and cost-effectiveness of the data warehouse are improved, and the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process is solved.

[0024] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor and a memory, where the memory is configured to store a computer program; and the processor is configured to execute the computer program stored on the memory to implement the data processing method in any one of the above.

[0025] The above electronic device provided by the embodiments of the present application achieves the following technical effects: The electronic device executes the above data processing method, can process the table data based on the accurate target life cycle, achieves the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solves the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0026] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium storing a computer program therein, and when the computer program is executed by a processor, the data processing method in any one of the above is implemented.

[0027] The above computer-readable storage medium provided by the embodiments of the present application achieves the following technical effects: The computer program stored in the computer-readable storage medium executes the above data processing method, and can process table data based on an accurate target life cycle, achieving the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solving the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process. Description of the Drawings

[0028] Figure 1 is a flowchart of the data processing method provided by an embodiment of the present application;

[0029] Figure 2 is a flow example diagram of the data processing method provided by an embodiment of the present application;

[0030] Figure 3 is a structural diagram of the data processing device provided by the embodiments of the present application;

[0031] Figure 4 is a structural diagram of the electronic device provided by the embodiments of the present application. Detailed Embodiments

[0032] In order to make the technical problems, technical solutions and beneficial effects solved by the present application clearer, the present application will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] According to an embodiment of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0035] A data processing method provided by an embodiment of the present application includes: obtaining table data in a data warehouse, where the table data includes access data, scheduling data, and dependency data; inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; determining a target life cycle according to the life cycle coefficient and the initial life cycle, where the initial life cycle is determined based on the table data; and processing the table data according to the target life cycle.

[0036] The above data processing method provided by an embodiment of the present application achieves the following technical effects: By obtaining the access, scheduling, and dependency data of various tables in the data warehouse, it is possible to systematically understand the actual application value and dynamic requirements of each table. Inputting this table data into a pre-trained life cycle coefficient recommendation model, the model, based on data analysis and machine learning techniques, can accurately evaluate the first life cycle coefficient of different tables, which is equivalent to providing a personalized and forward-looking assessment of the storage requirements of each table. Combining the initial life cycle of each table, we can calculate a more accurate target life cycle. This process realizes the dynamic adjustment and optimization of data life cycle management to adapt to the real-time changes of data in the data warehouse and the evolution of business requirements. Processing the table data based on the accurate target life cycle achieves the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solves the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0037] Embodiment 1

[0038] An embodiment of the present application provides a data processing method. Please refer to Figure 1 , including the following steps:

[0039] Step S110, obtain table data in a data warehouse, where the table data includes access data, scheduling data, and dependency data.

[0040] Exemplarily, the system first actively collects detailed information of all data tables in the data warehouse, including access records of the tables (such as access frequency, last access time, etc.), scheduling information (such as periodicity of scheduling tasks, execution time consumption, resource occupancy, etc.), and data dependency situations (i.e., the association between tables, reflecting the dependency relationships in the data processing flow). These information jointly constitute a comprehensive view of the table data and provide basic data for subsequent life cycle analysis.

[0041] Optionally, in some embodiments of the present invention, the table data includes basic information of the table (including table name, belonging data warehouse level, creation time, belonging business, number of table fields, granularity, whether life cycle is set, number of existing date partitions, etc., last modification time), access information (last access time, accessed fields, accessed partitions, accessed data volume, last accessor, access times in the recent week (month, half year, year), etc., partitions accessed in the recent week (month, half year, year)), scheduling and upstream and downstream dependency information (whether there is a scheduling task, scheduling period, scheduling online time, scheduling modification time, scheduling execution time consumption, resources occupied by scheduling execution, whether there is an upstream and downstream dependent scheduling, data tables corresponding to the dependent scheduling), etc.

[0042] Step S120, input the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient.

[0043] Exemplarily, after collecting the table data, the system takes this data as input and feeds it into a pre-trained life cycle coefficient recommendation model. Optionally, the model uses machine learning algorithms to analyze the characteristics of the table data, predict and output the first life cycle coefficient, which reflects the proportion of the actual duration that the data table should be retained relative to its initial life cycle, and provides a quantitative indicator to evaluate the storage value of the data.

[0044] Step S130, determine the target life cycle according to the life cycle coefficient and the initial life cycle, where the initial life cycle is determined based on the table data.

[0045] Exemplarily, the system will then calculate a target life cycle according to the first life cycle coefficient obtained in step S120, in combination with the initial life cycle of each table, that is, the optimal storage period of each table under the current business requirements. This process takes into account both the dynamic value of the data and the preset storage strategy of the data warehouse, ensuring the scientificity and rationality of the storage decision.

[0046] Step S140, process the table data according to the target life cycle.

[0047] Exemplarily, after determining the target lifecycle, storage management decisions are executed. Automatically identify and clean up data whose storage period has exceeded the target lifecycle. At the same time, for some data that has not reached the upper limit of the lifecycle but has extremely low usage frequency, implement hot and cold tiered storage to reduce storage costs and improve data access efficiency. This series of processes ensures that the storage resources in the data warehouse can be more reasonably allocated, while improving the automation level of data management and the overall operation efficiency.

[0048] Based on the above content, it can be seen that by automatically collecting access data, scheduling data, and dependency data of all tables, the system can form a detailed data usage and association relationship map, which provides rich and accurate data support for subsequent lifecycle analysis, ensuring the basis and scientificity of decision-making.

[0049] The table data is input into a pre-trained lifecycle coefficient recommendation model. Utilizing the powerful analysis ability of machine learning, the model can identify the dynamic change law of data value, and thus output the first lifecycle coefficient. The lifecycle coefficient intuitively reflects the actual duration ratio that the data table should be retained, and is the key basis for subsequent lifecycle optimization decisions.

[0050] Combining the first lifecycle coefficient and the initial lifecycle of each table, the system can calculate the target lifecycle, that is, the optimal storage duration of each table in the current business environment and data characteristics. It not only considers the dynamic value of the data but also takes into account the preset storage strategy, realizing the formulation of personalized storage strategies and improving the accuracy of resource allocation.

[0051] Based on the target lifecycle, the system automatically manages the table data, including but not limited to cleaning data slices, hot and cold data tiered storage, etc. This automated processing flow not only reduces the waste of storage resources but also optimizes the access efficiency through data tiering, reduces storage costs, and at the same time ensures that the storage strategy of the data warehouse can flexibly adapt to changes in business requirements.

[0052] In summary, the data processing method provided in the above embodiments includes: obtaining table data in the data warehouse, where the table data includes access data, scheduling data, and dependency data; inputting the table data into a pre-trained lifecycle coefficient recommendation model to obtain the first lifecycle coefficient; determining the target lifecycle according to the lifecycle coefficient and the initial lifecycle, where the initial lifecycle is determined based on the table data; and processing the table data according to the target lifecycle. By adopting the above method, it is possible to process the table data based on an accurate target lifecycle, achieving the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solving the technical problem of waste of storage resources caused by the inability to accurately determine the lifecycle of data during the data management process.

[0053] Optionally, in some embodiments of the present invention, the construction of the life cycle coefficient recommendation model includes the following steps:

[0054] Step S121, obtain training data.

[0055] During the execution of this step, the system collects a large amount of training data related to data storage and life cycle management, including but not limited to access records of data tables, scheduling information, dependency relationships, and business requirements, etc. The comprehensiveness and accuracy of the training data are the keys to the success of model training, ensuring that the model can learn the dynamic changes of data value from multiple perspectives.

[0056] Step S122, according to the preset screening rules, screen out the target table set from the training data.

[0057] During the execution of this step, the system uses the preset screening rules to identify a set of target table sets from the obtained training data.

[0058] Optionally, the preset screening rules are based on characteristics such as the usage frequency, business importance, and data type of the data tables. The purpose is to focus on the most representative data tables or those with a greater impact on the storage strategy, so as to improve the efficiency and pertinence of model training.

[0059] Step S123, according to the business requirements of the target table set, determine the second life cycle coefficient of the target table set.

[0060] In this step, the system further analyzes the business requirements of the target table set and determines a second life cycle coefficient for these table sets according to the business requirements. The life cycle coefficient reflects the proportion of the duration that the data table should be retained from the business perspective. It is a key reference index in the model optimization and verification process, reflecting the direct impact of business requirements on the decision-making of the data storage period.

[0061] Exemplarily, determining the second life cycle coefficient according to the business requirements is implemented based on a relationship comparison table, and different business requirements can correspond to different second life cycle coefficients.

[0062] Step S124, use the second life cycle coefficient and multi-dimensional data as sample data to train the pre-constructed initial model to obtain a life cycle coefficient recommendation model, where the multi-dimensional data is included in the training data, and the multi-dimensional data includes access dimension data, scheduling dimension data, and dependency dimension data.

[0063] In this step, the second life cycle coefficient and the multi-dimensional data (including access dimension data, scheduling dimension data, and dependency dimension data) extracted from the training data are used as sample data for model training. Through these sample data, the system continuously iterates and optimizes the initial model using machine learning algorithms until a recommendation model that can accurately predict the life cycle coefficient of the data table is obtained.

[0064] It can be understood that the life cycle coefficient recommendation model can intelligently recommend the storage period of each data table based on the usage status of the data and business requirements, providing a scientific decision-making tool for data life cycle management and improving the operation efficiency and economy of the data warehouse.

[0065] Optionally, in some embodiments of the present invention, the determination of the initial life cycle includes the following steps:

[0066] Step S131, obtain the level of the table in the data warehouse in the table data.

[0067] In this step, the system needs to obtain the position or level of each data table in the data warehouse structure. The data warehouse is usually hierarchically organized according to business domains, processing stages, or data types, such as the fact layer, dimension layer, summary layer, source data layer, etc. Data tables at different levels may have different business uses and data characteristics. For example, the fact layer may store the most granular business data, while the summary layer may contain analyzed and summarized data results. Identifying these levels is crucial for understanding the business value and storage requirements of the data.

[0068] Step S132, determine the initial life cycle of the table according to the level.

[0069] Exemplarily, after clarifying the level of the data table, the system can set an initial life cycle according to the general storage strategy of the data warehouse and the characteristics of the data at this level. This setting takes into account factors such as the business importance of the data, access frequency, data volume size, and processing cost. For example, for fact layer data tables that are frequently accessed and have high business value, their initial life cycle may be set longer to ensure data availability; while for summary layer or intermediate layer data tables that are less accessed and generated for specific tasks, their initial life cycle may be shorter to reduce the occupation of storage resources. The setting of the initial life cycle provides a starting point for subsequent life cycle coefficient recommendation and personalized adjustment, ensuring the systematicness and comprehensiveness of data life cycle management.

[0070] Optionally, in some embodiments of the present invention, in step S104, managing the table data according to the target life cycle includes the following steps:

[0071] Step S1041, obtain the current retention period of the table in the table data.

[0072] During the execution of this step, first obtain the actual storage duration of each table in the data warehouse, that is, the time span from the time point when the data table was created or last updated to the current time. This information reflects the actual existence time of the data in storage and is crucial for determining whether the data has exceeded its target life cycle.

[0073] Step S1042: Compare the current storage period with the target life cycle to obtain a first comparison result.

[0074] During the execution of this step, compare the current storage period with the target life cycle to obtain a first comparison result; then, the system compares and analyzes the current storage period of each table with the previously calculated target life cycle to generate a first comparison result. If the current storage period is less than or equal to the target life cycle, it indicates that the storage duration of the data is within a reasonable range; conversely, if the current storage period exceeds the target life cycle, the first comparison result will appear in a state indicating expiration, prompting the system to perform corresponding life cycle management operations.

[0075] Step S1043: In response to the first comparison result indicating that the current storage period is greater than the target life cycle, delete the expired data slices in the table, where the expired data slices are data slices whose slice storage period is greater than the target life cycle.

[0076] During the execution of this step, when the first comparison result confirms that the current storage period of the table exceeds its target life cycle, the system will automatically perform a data cleaning operation to delete the data slices in the table that exceed the storage period (target life cycle). It should be noted that different data slices in the table have their own current storage periods.

[0077] Optionally, if the overall data of a table exceeds its target life cycle and after evaluation it is confirmed that the table no longer has business value or the storage cost is higher than its value, the system will perform a delete operation on the entire table. This step ensures that the data stored in the data warehouse conforms to the latest storage policy, avoids unnecessary resource occupation, and at the same time improves the data management efficiency and the overall performance of the data warehouse.

[0078] Optionally, in some embodiments of the present invention, the data processing method further includes: in response to the current storage period being a preset storage period, delete the table and at the same time take offline the scheduling task corresponding to the table.

[0079] Specifically, when the system detects that the current retention period of a certain data table is equal to or exceeds the preset retention period (which is determined based on the access pattern, scheduling requirements, and business value analysis of the data), it will trigger an automatic deletion mechanism. This means that for those data tables without storage value, the system will no longer retain their data to free up storage resources.

[0080] Optionally, in some embodiments of the present invention, the preset retention period is zero.

[0081] Meanwhile, the system will further check the scheduling tasks related to this table. In a data warehouse and big data processing environment, data tables are often associated with scheduling mechanisms such as timed ETL (Extract, Transform, Load) jobs, data update or cleaning tasks, etc. These tasks run continuously during the life cycle of the data table to maintain the accuracy and timeliness of the data. Once the data table is marked as exceeding the preset retention period and it is evaluated and confirmed that the table is no longer needed, the system will take offline the scheduling tasks directly associated with this table, that is, stop their execution, thus avoiding the continuous generation and processing of useless data and further saving computing resources.

[0082] Optionally, in some embodiments of the present invention, step S104, managing the table data according to the target life cycle includes the following steps:

[0083] Step S1044, comparing the target life cycle with the expected life cycle to obtain a second comparison result, where the expected life cycle is determined according to a preset cycle determination rule, accessed data, scheduled data, and dependent data.

[0084] In this step, the system calculates the expected life cycle of each data table based on multi-dimensional information such as a preset cycle determination rule, accessed data, scheduled data, and dependent data. This cycle reflects the optimal time length that a data table should theoretically be retained under the current business requirements, data access pattern, and processing efficiency. Then, the system compares and analyzes the calculated expected life cycle with the previously determined target life cycle to obtain a second comparison result. If the difference between the two is not significant, it indicates that the current target life cycle setting is relatively reasonable and can be maintained; otherwise, if the difference exceeds the preset difference, this usually means that there is a large difference between the target life cycle and the actual requirements, and further optimization and adjustment are needed.

[0085] Step S1045, in response to the second comparison result indicating that the difference between the expected life cycle and the target life cycle is greater than the preset difference, stratify the table data according to the target life cycle and the expected life cycle.

[0086] Exemplarily, when the second comparison result confirms that the difference between the two exceeds the preset range, the system will perform a hierarchical processing on the data table according to the difference between the target life cycle and the expected life cycle, that is, implement a hot and cold data separation strategy. This strategy divides the data into "hot data" (frequently accessed, high business value) and "cold data" (less accessed, lower business value but may have historical or potential value). For those data that should be deleted earlier according to the recommended target life cycle, but still need to be retained based on business requirements and expected life cycle judgment, the system will identify these data as cold data. The cold data is migrated to a storage layer with lower cost and relatively slower access speed, while the hot data is retained in a storage layer with higher cost but faster access speed. This hierarchical strategy not only optimizes the storage cost, but also improves the data access efficiency and processing speed, ensuring the high performance and economy of the data warehouse.

[0087] Optionally, in some embodiments of the present invention, step S1045 of hierarchically processing the table data according to the target life cycle and the expected life cycle includes the following steps:

[0088] Step S1045a, obtaining the current retention period of the table in the table data;

[0089] Step S1045b, in response to the current retention period being greater than the target life cycle and less than the expected life cycle, storing the table in the cold data layer, where the cold data layer is used to store data with an access frequency lower than a preset frequency threshold; or, in response to the current retention period being less than or equal to the target life cycle, storing the table in the hot data layer, where the hot data layer is used to store data with an access frequency higher than a preset frequency threshold.

[0090] In the above steps, the system first determines the actual existence time of the data table since the last update or creation, that is, the current retention period. This information is crucial for judging the storage status of the data and subsequent hierarchical storage decisions.

[0091] If the current retention period is greater than the target life cycle but less than the expected life cycle: This means that although the retention duration of the data has exceeded the target life cycle calculated based on its access pattern and scheduling tasks, according to business requirements and potential value analysis of the data, the data still needs to be retained, although its access frequency may be low. In this case, the system migrates the data table to the cold data layer for storage. The cold data layer usually uses storage devices with lower cost (such as tapes or low-cost hard disks), and has low requirements for the data reading speed, which is suitable for storing data with an access frequency lower than the preset frequency threshold, so as to maximize the storage cost benefit on the premise of ensuring data security and accessibility.

[0092] If the current retention period is less than or equal to the target lifecycle: This indicates that the retention duration of the data meets or is lower than its optimal storage period, and the data still has high business value and access requirements. The system will store such data tables in the hot data layer, which usually uses high-performance storage devices (such as SSD solid-state drives) to provide fast data read and write capabilities, ensuring the efficient processing of frequently accessed and high-business-value data. In this way, hot data can be accessed more quickly and utilized more effectively, further improving the response speed and processing capacity of the data warehouse.

[0093] Optionally, the data processing method further includes: using the target lifecycle, access data, scheduling data, and dependent data as training data to update the lifecycle coefficient recommendation model, resulting in an updated lifecycle coefficient recommendation model.

[0094] In this step, the system uses these four types of information, namely the target lifecycle, access data, scheduling data, and dependent data, as training data for updating and optimizing the lifecycle coefficient recommendation model. The target lifecycle refers to the period that the data table should theoretically be retained, calculated by the system based on factors such as business requirements and data characteristics; access data includes information such as the access frequency, access time, and visitors of the data table, reflecting the actual usage of the data; scheduling data refers to the scheduling task information of the data table, including the scheduling period, scheduling time, and status of the scheduling task; dependent data is the upstream and downstream dependency relationships between the data table and other data tables or systems, which reveals the role and value of the data in a broader business process.

[0095] Taking these multi-dimensional data as input, the system will train and update the existing lifecycle coefficient recommendation model through machine learning or other data analysis techniques. The training process of the model can use various machine learning algorithms, such as regression analysis, neural networks, decision trees, etc., to identify and learn the complex relationships between the data lifecycle and access patterns, scheduling tasks, and business requirements. Through the continuous learning and optimization of the model, the accuracy and adaptability of the data lifecycle prediction can be improved, making the lifecycle coefficient output by the model closer to the actual business requirements, thereby enhancing the efficiency and effectiveness of data management.

[0096] Obtaining the updated lifecycle coefficient recommendation model: After training and optimization, the model will be able to more accurately predict and recommend the lifecycle coefficient of the data table. This coefficient is a key parameter used to adjust the actual retention period of the data table, which changes dynamically according to the access, scheduling, and dependency information of the data, and can guide the system to manage the storage period of the data more intelligently, avoid resource waste, and ensure the long-term accessibility of important data.

[0097] Optionally, referring to Figure 2, in some embodiments of the present invention, the data processing method provided by the present invention is exemplarily implemented as follows:

[0098] Step 1-1: Set the initial life cycle t1 of each table according to the level of the table in the data warehouse. Optionally, tables at each level can have different life cycles, or the same life cycle can be set for all.

[0099] Step 1-2: Select some commonly used and important tables, as well as some historical intermediate tables from all the tables. Set the corresponding life cycle coefficient r according to business requirements, which is between [0-1]. It means that according to the initial life cycle t1, the actual life cycle that these tables should retain is between t1*1 and t1*0.

[0100] Step 2-1: Collect all information of the data warehouse tables in the system, including basic information of the table (including table name, data warehouse level to which it belongs, creation time, business to which it belongs, number of table fields, granularity, whether the life cycle is set, number of existing date partitions, etc., last modification time), access information (last access time, accessed fields, accessed partitions, access data volume, last access person, access times in the last week (month, half year, year), etc., partitions accessed in the last week (month, half year, year), etc.), scheduling and upstream and downstream dependency information (whether there is a scheduling task, scheduling period, scheduling online time, scheduling modification time, scheduling execution time consumption, scheduling execution resource occupancy, whether there is an upstream and downstream dependent scheduling, data tables corresponding to the dependent scheduling), etc.

[0101] Step 2-2: Combine the life cycle coefficient confirmed in Step 1-2 and the dimension information of all tables in Step 2-1 to build a model to simulate the recommended values of the life cycle coefficient for different situation tables.

[0102] Step 2-3: Multiply the recommended life cycle coefficient by the model by the initial life cycle t1 to obtain the recommended life cycle of all remaining tables except the tables in Step 1-2.

[0103] Step 3-1: Synchronize the recommended results of the life cycle of all tables and the input dimension information to data warehouse professionals and business personnel to finally confirm the life cycle value t2 of each table.

[0104] Step 3-2: Manage the tables according to the finally confirmed life cycle value. Since the coefficient is a value between 0 and 1, the life cycle value will generally be less than or equal to the initially set value. When the retention period of the table exceeds the recommended value, the data slices in the table that exceed the recommended value can be deleted to complete the recovery of storage resources.

[0105] Step 3-3: For some tables with a life cycle value of 0, execute table deletion, and at the same time, if there is a corresponding scheduling task, take the task offline to recover the corresponding computing power resources.

[0106] Step 4-1: Divide the confirmed life cycle values of professionals and business personnel by the initial life cycle values to inversely calculate the corresponding life cycle coefficients. Use the coefficients and dimension values as the model inputs for calculating the life cycle coefficients of the next calculation table. Make full use of the business knowledge of professionals to train the model to make the next calculation result closer to the business reality.

[0107] Step 4-2: For the data in Step 3-1 whose recommended storage period by the model is much less than the life cycle determined by professionals, use the cycle recommended by the model as the boundary for cold and hot data hierarchical storage. The data that exceeds the recommended life cycle and needs to be saved as identified by professionals is designated as cold data, and based on this, cold and hot data hierarchical storage is carried out to further reduce the storage cost.

[0108] Embodiment II

[0109] The embodiment of the present application also provides a vehicle body collision safety design device 30. Please refer to Figure 3 , including an acquisition module 310 for acquiring table data in a data warehouse, where the table data includes access data, scheduling data, and dependency data; a recommendation module 320 for inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; a determination module 330 for determining a target life cycle according to the life cycle coefficient and the initial life cycle, where the table in the table data corresponds to an initial life cycle; and a processing module 340 for managing the table data according to the target life cycle.

[0110] The above data processing device provided by the embodiment of the present application achieves the following technical effects: The acquisition module collects the access data, scheduling data, and dependency data of all tables in the data warehouse. The recommendation module inputs the acquired table data into the pre-trained life cycle coefficient recommendation model, analyzes the characteristics and patterns of the data using machine learning algorithms, and outputs the first life cycle coefficient. The determination module calculates the target life cycle, that is, the optimal storage period that each table should maintain in the current business environment, according to the recommended first life cycle coefficient and the initially set initial life cycle of each table. The processing module is responsible for executing the storage management decision based on the target life cycle. Through the close cooperation among the above modules, the target life cycle can be accurately calculated, realizing the automation and intelligence of data life cycle management, improving the storage efficiency and cost-effectiveness of the data warehouse, and solving the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0111] It should be noted that the optional implementation manners of this embodiment can refer to the relevant descriptions in Embodiment I, and will not be elaborated here.

[0112] Embodiment III

[0113] An embodiment of the present application further provides an electronic device 40. Please refer to Figure 4 , which includes a memory 410 and a processor 420. Among them, the memory 410 is used to store a computer program; the processor 420 is used to execute the computer program stored on the memory to implement the data processing method introduced in any embodiment of the present application.

[0114] The above-mentioned electronic device provided by the embodiment of the present application achieves the following technical effects: By executing the above data processing method, the electronic device can process table data based on an accurate target life cycle, achieving the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solving the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0115] Those of ordinary skill in the art can understand that Figure 4 the structure shown is only for illustration. The electronic device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a Mobile Internet Device (MID for short). Figure 4 This does not limit the structure of the above-mentioned electronic device. For example, the electronic device 40 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 4 , or have a different configuration from that shown in Figure 4 .

[0116] Embodiment 4

[0117] An embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the data processing method introduced in any embodiment of the present application.

[0118] Optionally, in this embodiment, the above storage medium may include, but is not limited to: various media that can store computer programs such as a USB flash drive, a Read-Only Memory (ROM for short), a Random Access Memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc.

[0119] The computer-readable storage medium provided by the embodiments of the present application achieves the following technical effects: The computer program stored in the computer-readable storage medium executes the above data processing method, and can process table data based on an accurate target life cycle, achieving the technical effects of effectively releasing storage resources, reducing unnecessary storage costs, and avoiding waste of storage resources, and solving the technical problem of waste of storage resources caused by the inability to accurately determine the life cycle of data during the data management process.

[0120] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0121] In the above embodiments of the present application, the descriptions of the various embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0122] In the present application, "a plurality of" means two or more.

[0123] In the present application, unless otherwise clearly defined, the terms "installed", "connected", and "coupled" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific situations.

[0124] The terms "first", "second", "third", "fourth", etc. (if any) in the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence.

[0125] The term "and / or" in the present application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects before and after.

[0126] If there is no special description, all steps of the present application can be carried out in sequence or randomly. For example, the method includes steps A and B, indicating that the method may include steps A and B carried out in sequence, or may include steps B and A carried out in sequence. For example, it is mentioned that the method may further include step C, indicating that step C can be added to the method in any order. For example, the method may include steps A, B, and C, or may include steps A, C, and B, or may include steps C, A, and B, etc.

[0127] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire table data in a data warehouse, wherein the table data includes access data, scheduling data, and dependency data; Inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; Determine a target life cycle according to the life cycle coefficient and the initial life cycle, wherein the initial life cycle is determined according to the table data; The table data is processed according to the target life cycle.

2. The method according to claim 1, characterized in that The construction of the life cycle coefficient recommendation model includes: Get training data; According to preset screening rules, a target table set is screened out from the training data; Determining a second life cycle coefficient of the target table set according to the business requirements of the target table set; The second life cycle coefficient and multi-dimensional data are used as sample data to train the pre-built initial model to obtain the life cycle coefficient recommendation model, wherein the multi-dimensional data is contained in the training data, and the multi-dimensional data includes access dimension data, scheduling dimension data and dependency dimension data.

3. The method according to claim 1, characterized in that The determination of the initial life cycle includes: Obtaining the level of the table in the table data in the data warehouse; The initial life cycle of the table is determined according to the level.

4. The method according to claim 1, characterized in that: The managing the table data according to the target life cycle includes: Obtaining the current retention period of the table in the table data; Comparing the current preservation period with the target life cycle to obtain a first comparison result; In response to the first comparison result indicating that the current retention period is greater than the target life cycle, overdue data slices in the table are deleted, wherein the overdue data slices are data slices whose retention period is greater than the target life cycle.

5. The method according to claim 4, characterized in that Also includes: In response to the current retention period being the preset retention period, the table is deleted, and the scheduling task corresponding to the table is taken offline.

6. The method according to claim 1, characterized in that The managing the table data according to the target life cycle includes: Comparing the target lifecycle with the expected lifecycle to obtain a second comparison result, wherein the expected lifecycle is determined according to a preset cycle rule, the access data, the scheduling data, and the dependency data; In response to the second comparison result indicating that the difference between the expected life cycle and the target life cycle is greater than a preset difference, the table data is stratified according to the target life cycle and the expected life cycle.

7. The method according to claim 6, characterized in that The stratifying the table data according to the target life cycle and the expected life cycle includes: Obtaining the current retention period of the table in the table data; In response to the current retention period being greater than the target life cycle and less than the expected life cycle, storing the table in a cold data layer, wherein the cold data layer is used to store data with an access frequency lower than a preset frequency threshold; or, In response to the current retention period being less than or equal to the target life cycle, the table is stored in a hot data layer, wherein the hot data layer is used to store data with an access frequency higher than the preset frequency threshold.

8. The method according to claim 1, characterized in that Also includes: The target lifecycle, the access data, the scheduling data and the dependency data are used as training data to update the lifecycle coefficient recommendation model to obtain an updated lifecycle coefficient recommendation model.

9. A data processing device, characterized in that: include: An acquisition module, used to acquire table data in the data warehouse, wherein the table data includes access data, scheduling data and dependency data; A recommendation module, used for inputting the table data into a pre-trained life cycle coefficient recommendation model to obtain a first life cycle coefficient; A determination module, configured to determine a target life cycle according to the life cycle coefficient and an initial life cycle, wherein a table in the table data corresponds to the initial life cycle; A processing module is used to manage the table data according to the target life cycle.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein Memory, used to store computer programs; A processor, used to execute a program stored in a memory to implement the method described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images