Data storage optimization method and device, electronic equipment and storage medium
By acquiring multi-dimensional information from stored data, using model clusters to perform inference calculations and popularity scoring, and dynamically adjusting the storage layer, the "cold start" latency problem in cloud storage technology is solved, achieving efficient data access and management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-04-07
AI Technical Summary
Existing cloud storage technologies suffer from a "cold start" delay in predicting data access demands, failing to meet enterprises' needs for timely and flexible data access. Current solutions are reactive and cannot predict future access demands.
By acquiring multi-dimensional information from stored data, using model clusters for inference, calculating a comprehensive popularity score, dynamically adjusting the storage layer, and migrating data to the appropriate level in advance, we can avoid "cold start" delays.
It achieves a fundamental reduction in data access latency, meets the high-performance access requirements of real-time business, and enables more accurate storage management decisions.
Smart Images
Figure CN121807221A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data storage optimization method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the widespread adoption of cloud computing, enterprise data volumes are growing explosively. Managing the entire data lifecycle economically and efficiently, and achieving a balance between storage costs and access performance, has become a key challenge for enterprise digital transformation.
[0003] The mainstream solution offered by existing cloud service providers is automated tiered storage technology based on access frequency. This technology divides storage resources into different tiers such as hot, warm, and cold. It determines the popularity of data based on the historical access frequency of the data and automatically migrates high-frequency access data to the high-performance hot storage tier and low-frequency access data to the low-cost cold storage tier.
[0004] However, this type of technology is essentially a "passive reactive" optimization, which has obvious drawbacks. One of the core problems is the "cold start" latency issue: it relies solely on historical access data to determine value and cannot predict future access needs. When "cold data" that has not been accessed for a long time needs to be retrieved, restoring data from the cold storage layer or archive layer will result in a significant delay of several minutes to several hours, which seriously affects the performance of real-time or near real-time business operations and makes it difficult to meet the needs of enterprises for the timeliness and flexibility of data access. A more intelligent storage optimization solution is urgently needed. Summary of the Invention
[0005] This application aims to at least partially address one of the technical problems in the related art.
[0006] Therefore, one objective of this application is to propose a data storage optimization method, which involves obtaining multi-dimensional information corresponding to target stored data located in a storage database; based on the multi-dimensional information, inferring from candidate models in a model cluster to obtain multiple scores corresponding to the target stored data, including access probability score, content value score, and event influence score; obtaining the weights corresponding to each of the multiple scores, and performing a weighted summation based on the weights to obtain a comprehensive popularity score for the target stored data; determining the target storage layer corresponding to the target stored data from the hot / warm / cold / archive storage layers in the storage database according to the comprehensive popularity score; determining the current storage layer of the target stored data; and migrating the target stored data from the current storage layer to the target storage layer in response to inconsistencies between the target storage layer and the current storage layer.
[0007] The second objective of this application is to propose a data storage optimization device.
[0008] The third objective of this application is to propose an electronic device.
[0009] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.
[0010] The fifth objective of this application is to provide a computer program product.
[0011] To achieve the above objectives, the first aspect of this application proposes a data storage optimization method, which involves obtaining multi-dimensional information corresponding to target stored data located in a storage database; based on the multi-dimensional information, inferring from candidate models in a model cluster to obtain multiple scores corresponding to the target stored data, including access probability score, content value score, and event influence score; obtaining the weights corresponding to each of the multiple scores, and performing a weighted summation based on the weights to obtain a comprehensive popularity score for the target stored data; determining the target storage layer corresponding to the target stored data from the hot / warm / cold / archived storage layers in the storage database according to the comprehensive popularity score; determining the current storage layer of the target stored data; and migrating the target stored data from the current storage layer to the target storage layer in response to the inconsistency between the target storage layer and the current storage layer.
[0012] The data storage optimization method described in this application does not aim to move data after it becomes "cold," but rather to pre-deploy it to a suitable storage layer (different storage layers correspond to different resources) before it becomes "hot." This fundamentally eliminates or significantly reduces access latency caused by data layer migration, especially the "cold start" problem, and meets the stringent requirements of real-time business for high-performance access. This solution can truly "understand" the business significance and potential value of data, thereby making storage management decisions that are far more accurate and better suited to business needs than existing technologies.
[0013] According to one embodiment of this application, the multi-dimensional information includes time-series related information, text elements, and a knowledge graph. Based on the multi-dimensional information, candidate models in the model cluster are invoked for inference to obtain multiple scores corresponding to the target stored data. This includes: inputting time-series related information into a pre-trained access probability score prediction model to obtain the access probability score of the target stored data in the target time period, wherein the time-series related information includes the access time series, access frequency information, and access latency information of the target stored data in historical time periods; inputting text elements into a pre-trained content value assessment model to obtain the content value score corresponding to the target stored data, wherein the text elements include at least one of the keyword information, entity information, and full text of the target stored data; and inputting the knowledge graph into a pre-trained event impact analysis model to obtain the event impact score corresponding to the target stored data, wherein the knowledge graph is used to store the relationships between entities, data, and business events.
[0014] According to one embodiment of this application, the data storage optimization method further includes: after the cost target and performance target corresponding to the storage database change, dynamically adjusting the weights corresponding to multiple scores according to the changed cost target and performance target.
[0015] According to one embodiment of this application, the data storage optimization method further includes: obtaining actual access information of each storage layer in the storage database; determining the storage performance evaluation index of the storage database based on the actual access information; and fine-tuning or retraining the models in the model cluster when the storage performance evaluation index does not meet expectations.
[0016] According to one embodiment of this application, the data storage optimization method further includes: receiving a data access request sent by a client based on a data access proxy, wherein the data access request is a read request or a write request for target access data; forwarding the data access request to the storage database for data access through the data access proxy, and returning the data access result to the client; and generating an access log corresponding to the data access request.
[0017] According to one embodiment of this application, after generating the access log corresponding to the data access request, the method further includes: From the access logs, the latest metadata information of the target access data is extracted, and the corresponding metadata information of the target access data in the metadata database is updated based on the latest metadata information. The metadata information contains multiple metadata fields, and the metadata database is used to store the correspondence between data and metadata information. From the access logs, the access time of the target access data is extracted, and the corresponding time series information of the target access data in the time series information database is updated based on the access time. The time series information database is used to store the correspondence between data and time series information. Content analysis is performed on the target access data to establish the relationship between entities, data, and business events, and the relevant knowledge graph in the graph database is updated based on the relationship.
[0018] According to one embodiment of this application, a method for obtaining multi-dimensional information includes: querying a time-series related information database to obtain time-series related information corresponding to target stored data; extracting at least one of keyword information, entity information, and full text of the target stored data to obtain text elements corresponding to the target stored data; querying a graph database to obtain a knowledge graph corresponding to the target stored data; and constructing multi-dimensional information corresponding to the target stored data based on the time-series related information, text elements, and knowledge graph.
[0019] According to one embodiment of this application, the data storage optimization method further includes: in response to receiving a first data query request, parsing to obtain the metadata field carried by the first data query request; and filtering from the metadata field carried by the first data query request to obtain a first data list corresponding to the first data query request from the metadata database.
[0020] According to one embodiment of this application, the data storage optimization method further includes: in response to receiving a second data query request, parsing to obtain the target entity carried in the second data query request; querying a target knowledge graph containing the target entity from a graph database; and filtering to obtain a second data list corresponding to the second data query request from the graph database based on the association relationships in the target knowledge graph.
[0021] To achieve the above objectives, a second aspect of this application proposes a data storage optimization apparatus, comprising: an information acquisition module for acquiring multi-dimensional information corresponding to target stored data located in a storage database; a model inference module for inferring multiple scores corresponding to the target stored data based on the multi-dimensional information, including access probability score, content value score, and event influence score; a weighted summation module for acquiring the weights corresponding to the multiple scores respectively, and performing a weighted summation of the multiple scores based on the weights to obtain a comprehensive popularity score for the target stored data; a storage layer determination module for determining the target storage layer corresponding to the target stored data from the hot / warm / cold / archived storage layers in the storage database according to the comprehensive popularity score; and a data migration module for determining the current storage layer of the target stored data, and migrating the target stored data from the current storage layer to the target storage layer in response to the inconsistency between the target storage layer and the current storage layer.
[0022] According to one embodiment of this application, the multi-dimensional information includes time-series related information, text elements, and a knowledge graph. The model inference module is further configured to: input the time-series related information into a pre-trained access probability score prediction model to obtain the access probability score of the target stored data in the target time period, wherein the time-series related information includes the access time series, access frequency information, and access latency information of the target stored data in the historical time period; input the text elements into a pre-trained content value assessment model to obtain the content value score corresponding to the target stored data, wherein the text elements include at least one of the keyword information, entity information, and full text of the target stored data; and input the knowledge graph into a pre-trained event impact analysis model to obtain the event impact score corresponding to the target stored data, wherein the knowledge graph is used to store the relationships between entities, data, and business events.
[0023] According to one embodiment of this application, the data storage optimization device further includes a weight adjustment module, which is used to: dynamically adjust the weights corresponding to multiple scores according to the changed cost target and performance target after the cost target and performance target corresponding to the storage database change.
[0024] According to one embodiment of this application, the data storage optimization device further includes a model performance evaluation module, which is used to: obtain actual access information of each storage layer in the storage database; determine the storage performance evaluation index of the storage database based on the actual access information; and fine-tune or retrain the models in the model cluster when the storage performance evaluation index does not meet expectations.
[0025] According to one embodiment of this application, the data storage optimization device further includes a data access proxy module, configured to: receive a data access request sent by a client based on the data access proxy, wherein the data access request is a read request or a write request for target access data; forward the data access request to the storage database for data access through the data access proxy, and return the data access result to the client; and generate an access log corresponding to the data access request.
[0026] According to one embodiment of this application, the data storage optimization device further includes a knowledge base module, configured to: extract the latest metadata information of the target access data from the access log, and update the metadata information corresponding to the target access data in the metadata database based on the latest metadata information, wherein the metadata information includes multiple metadata fields, and the metadata database is used to store the correspondence between data and metadata information; extract the access time of the target access data from the access log, and update the time series related information corresponding to the target access data in the time series related information database based on the access time, wherein the time series related information database is used to store the correspondence between data and time series related information; perform content analysis on the target access data, establish the association relationship between entities, data, and business events, and update the relevant knowledge graph in the graph database based on the association relationship.
[0027] According to one embodiment of this application, the information acquisition module is further configured to: query time-series related information corresponding to the target stored data from a time-series related information database; extract at least one of keyword information, entity information, and full text of the target stored data to obtain text elements corresponding to the target stored data; query a knowledge graph corresponding to the target stored data from a graph database; and construct multi-dimensional information corresponding to the target stored data based on the time-series related information, text elements, and knowledge graph.
[0028] According to one embodiment of this application, the data storage optimization device further includes a data query module, configured to, in response to receiving a first data query request, parse the metadata field carried by the first data query request to obtain the first data list corresponding to the first data query request from the metadata field carried by the first data query request.
[0029] According to one embodiment of this application, the data query module is further configured to: in response to receiving a second data query request, parse the target entity carried in the second data query request; query a target knowledge graph containing the target entity from the graph database; and filter the second data list corresponding to the second data query request from the graph database based on the association relationships in the target knowledge graph.
[0030] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to implement the data storage optimization method as described in the first aspect of this application.
[0031] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the data storage optimization method as described in the first aspect of this application.
[0032] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data storage optimization method as described in the first aspect of this application. Attached Figure Description
[0033] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method according to one embodiment of this application.
[0034] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method according to one embodiment of this application.
[0035] Figure 3 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method according to one embodiment of this application.
[0036] Figure 4 This is an exemplary schematic diagram of a data storage optimization apparatus shown in one embodiment of this application.
[0037] Figure 5 This is a schematic diagram of an electronic device according to one embodiment of this application. Detailed Implementation
[0038] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0039] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0040] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method shown in this application, such as... Figure 1 As shown, this data storage optimization method includes the following steps: S101, obtain multi-dimensional information corresponding to the target stored data located in the storage database.
[0041] In this application, the specific process for optimizing the storage of target data in the storage database can be triggered on a scheduled basis or manually by business personnel.
[0042] The storage database is used to store data and includes a hot storage layer, a warm storage layer, a cold storage layer, and an archive storage layer.
[0043] In this application, the storage database can be a regular storage database or a cloud storage database.
[0044] The storage database contains a large amount of data. The target storage data described in this application embodiment can be part of the data in the storage database or all of the data in the storage database.
[0045] In this application, after determining the target stored data located in the storage database, multi-dimensional information corresponding to each piece of target stored data is obtained. This multi-dimensional information may include time-series related information, text elements, and knowledge graphs corresponding to the target stored data.
[0046] S102, based on multi-dimensional information, calls the candidate models in the model cluster to perform inference and obtain multiple scores corresponding to the target stored data. The multiple scores include access probability score, content value score and event influence score.
[0047] In this application, the model cluster may include three candidate models, which are used to predict the access probability score, content value score and event influence score corresponding to the target stored data, respectively.
[0048] For any target stored data, after determining the multi-dimensional information corresponding to the target stored data as described above, the multi-dimensional information can be input into its corresponding candidate model for reasoning to obtain the access probability score, content value score, and event influence score corresponding to the target stored data.
[0049] The access probability score refers to the probability that the target stored data will be accessed within a target time period in the future. For example, the target time period can be set to the next week; that is, the access probability score refers to the probability score of the target stored data being accessed within the next week. The higher the probability of access, the higher the access probability score. For example, if a contract is renewed annually, the probability of the contract being accessed increases as the contract renewal date approaches.
[0050] The content value score refers to the value of the content stored within the target data. For example, if a target data is a confidential internal contract of a company, its content value score is relatively high; if a target data is a notice of the company's holiday arrangements for the Spring Festival last year, its content value score is relatively low.
[0051] The event impact score refers to the degree of impact on the company if the target stored data were lost. For example, if the target stored data is the company's original experimental data on core technologies, its loss could lead to technology leakage, a decline in core competitiveness, or even huge economic losses, then the event impact score corresponding to the target stored data is relatively high. Conversely, if the target stored data is the company's annual meeting attendance sheet from 10 years ago, its loss would have little impact on the company's current situation, then the event impact score corresponding to the target stored data is relatively low.
[0052] The access probability score, content value score, and event influence score can all be expressed as percentages.
[0053] S103: Obtain the weights corresponding to multiple scores, and sum the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data.
[0054] The formula for calculating the overall popularity score is as follows:
[0055] In the above formula, Represents the probability score of access; The weights representing the access probability scores; Represents the value score of the content; The weight representing the content value score; The score represents the impact of the event; The weight of the score representing the impact of an event.
[0056] S104. Based on the comprehensive heat score, determine the target storage layer corresponding to the target storage data from the hot / warm / cold / archive storage layers in the storage database.
[0057] The system can set the comprehensive popularity score ranges corresponding to hot / warm / cold / archive storage layers respectively. After determining the comprehensive popularity score of the target storage data, the target storage layer corresponding to the target storage data is determined according to the range that the comprehensive popularity score of the target storage data falls within.
[0058] For example, the comprehensive popularity score ranges corresponding to the hot / warm / cold / archive storage layers can be set as follows: , , , .
[0059] S105, determine the current storage layer of the target storage data, and in response to the inconsistency between the target storage layer and the current storage layer, migrate the target storage data from the current storage layer to the target storage layer.
[0060] For example, if a target data item is currently stored in the warm storage layer of the storage database, but its corresponding target storage layer is determined to be a hot storage layer based on the comprehensive heat score, then it will be migrated to the hot storage layer.
[0061] In this application, the migration of target storage data from the current storage layer to the target storage layer can be performed based on a preset storage execution engine.
[0062] Specifically, the storage execution engine can call the storage application programming interface (API) provided by the cloud service provider to perform data migration, replication, or deletion operations. The storage execution engine generates specific cloud API call tasks based on the migration task, and then places these tasks into a priority queue. During execution, multiple executors process them concurrently, sequentially retrieving cloud API call tasks from the priority queue, calling the cloud service provider's API, executing them, and obtaining the execution results. If execution is successful, the latest storage location is recorded and the source data is updated. If execution fails, a failure log is recorded, and an exponential backoff algorithm is used for retrying. For tasks that fail after multiple retries, an alert is issued, and administrator intervention is notified.
[0063] The cloud API adaptation layer uses an abstract interface to design a unified storage operation interface and implements a specific adapter for each supported cloud platform, encapsulating their respective SDK and API call details to improve system portability.
[0064] This application proposes a data storage optimization method, including: obtaining multi-dimensional information corresponding to target stored data located in a storage database; based on the multi-dimensional information, calling candidate models in a model cluster for inference to obtain multiple scores corresponding to the target stored data, including access probability score, content value score, and event influence score; obtaining the weights corresponding to the multiple scores respectively, and performing a weighted summation of the multiple scores based on the weights to obtain a comprehensive popularity score for the target stored data; determining the target storage layer corresponding to the target stored data from the hot / warm / cold / archive storage layers in the storage database according to the comprehensive popularity score; determining the current storage layer of the target stored data, and migrating the target stored data from the current storage layer to the target storage layer in response to the inconsistency between the target storage layer and the current storage layer. The data storage optimization method described in this application does not aim to move data after it becomes "cold," but rather to pre-deploy it to a suitable storage layer (different storage layers correspond to different resources) before it becomes "hot." This fundamentally eliminates or significantly reduces access latency caused by data layer migration, especially the "cold start" problem, and meets the stringent requirements of real-time business for high-performance access. This solution can truly "understand" the business significance and potential value of data, thereby making storage management decisions that are far more accurate and better suited to business needs than existing technologies.
[0065] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method shown in this application, such as... Figure 2 As shown, this data storage optimization method includes the following steps: S201, Obtain multi-dimensional information corresponding to the target stored data located in the storage database.
[0066] S202, based on multi-dimensional information, calls the candidate models in the model cluster to perform inference and obtain multiple scores corresponding to the target stored data. These multiple scores include access probability score, content value score, and event influence score.
[0067] The multi-dimensional information includes time-series related information, textual elements, and knowledge graphs.
[0068] The candidate models in the model cluster include an access probability score prediction model, a content value assessment model, and an event impact analysis model. All three models are pre-trained using their respective sample training sets and real score labels.
[0069] Specifically, time-series related information from multi-dimensional information is input into a pre-trained access probability score prediction model to obtain the access probability score of the target stored data in the target time period. The time-series related information includes the access time series, access frequency information, and access latency information of the target stored data in historical time periods (e.g., the past 30 days).
[0070] Textual elements from multi-dimensional information are input into a pre-trained content value assessment model to obtain the content value score corresponding to the target stored data. The textual elements include at least one of the following: keyword information, entity information, and the full text of the target stored data. The content value assessment model can be fine-tuned based on a pre-trained natural language processing model.
[0071] By inputting the knowledge graph from multi-dimensional information into a pre-trained event impact analysis model, the event impact score corresponding to the target stored data is obtained. The knowledge graph is used to store the relationships between entities, data, and business events.
[0072] S203: Obtain the weights corresponding to multiple scores, and sum the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data.
[0073] The formula for calculating the overall popularity score is as follows:
[0074] In the above formula, Represents the probability score of access; The weights representing the access probability scores; Represents the value score of the content; The weight representing the content value score; The score represents the impact of the event; The weight of the score representing the impact of an event.
[0075] In this application, the weights corresponding to the multiple scores are related to the cost and performance objectives of the storage database. It is easy to understand that the emphasis on cost and performance objectives may differ at different times; therefore, the weights corresponding to the multiple scores will be dynamically adjusted based on the changed cost and performance objectives of the storage database.
[0076] In some embodiments, multiple sets of weights corresponding to cost targets and performance targets with different emphases can be preset ( , , After the cost and performance targets corresponding to the storage database change, the appropriate weight parameter is selected from a set of pre-set weights based on the changed cost and performance targets.
[0077] S204. Based on the comprehensive heat score, determine the target storage layer corresponding to the target storage data from the hot / warm / cold / archive storage layers in the storage database.
[0078] S205, determine the current storage layer of the target storage data, and in response to the inconsistency between the target storage layer and the current storage layer, migrate the target storage data from the current storage layer to the target storage layer.
[0079] For details on the specific implementation of steps S204 to S205, please refer to the relevant parts of the above embodiments, which will not be repeated here.
[0080] S206, Obtain the actual access information of each storage layer in the storage database.
[0081] S207, Determine the storage performance evaluation indicators of the storage database based on the actual access information.
[0082] In this application, the data actually accessed is used as the real hot data.
[0083] Storage performance evaluation metrics include accuracy and recall.
[0084] Accuracy refers to the probability that data currently stored in the hot storage layer should not actually be classified into that hot storage layer. That is, accuracy = the number of real hot data in the hot storage layer / the total number of all data in the hot storage layer.
[0085] Recall refers to the proportion of real-time data that is correctly assigned to the hot storage layer. Recall = Number of real-time data correctly assigned to the hot storage layer / Total number of real-time data.
[0086] S208 performs model fine-tuning or retraining on the models in the model cluster when the storage performance evaluation metrics do not meet expectations.
[0087] In this application, when the storage performance evaluation metrics do not meet expectations (accuracy and / or recall are lower than preset thresholds), the models in the model cluster can be fine-tuned or retrained.
[0088] In this application, when the storage performance evaluation indicators meet expectations, the model parameters in the current model cluster can be fixed or the model parameters can be enhanced.
[0089] The data storage optimization method described in this application does not aim to move data after it becomes "cold," but rather to pre-deploy it to a suitable storage layer (different storage layers correspond to different resources) before it becomes "hot," thereby fundamentally eliminating or significantly reducing access latency caused by data layer migration, especially the "cold start" problem, and meeting the stringent requirements of real-time business for high-performance access. This application introduces an adaptive feedback loop by fine-tuning or retraining the models in the model cluster when storage performance evaluation indicators do not meet expectations. This allows the model to continuously learn and adapt to changes in business models, ensuring long-term effectiveness and reducing operational complexity.
[0090] Figure 3 This is a schematic diagram illustrating an exemplary implementation of a data storage optimization method shown in this application, such as... Figure 3 As shown, this data storage optimization method includes the following steps: S301, based on the data access proxy, receives data access requests sent by the client. The data access request is a read request or a write request for the target data.
[0091] The data access proxy is a component that serves as the unified entry point and traffic sentinel for all data access requests. It is used to intercept all data access requests to the storage database (and deny unauthorized access during this process).
[0092] S302 forwards data access requests to the storage database through the data access proxy, returns the data access results to the client, and generates access logs corresponding to the data access requests.
[0093] The data access proxy parses data access requests. If the data access request is a read request, it forwards the data access request to the storage database for data access and returns the data access result of the read request to the client.
[0094] The data access proxy parses data access requests. If the data access request is a write request, it forwards the data access request to the storage database for data access and returns the data access result of the write request to the client.
[0095] The access logs should include at least the access time, accessing user ID, access type, and the latest metadata information of the target access data.
[0096] The data access proxy can be deployed using either the Sidecar proxy mode or the API gateway mode. For object storage (such as S3), a gateway compatible with the S3 API can be developed; for block storage, a lightweight Sidecar container can be deployed on the host.
[0097] The data access proxy needs to transparently intercept all requests and forward them to the underlying storage database service through efficient network input / output (such as non-blocking input / output) to ensure low latency.
[0098] The data access proxy itself must have authentication and authorization capabilities to ensure that only legitimate requests can pass through.
[0099] In this application, a knowledge base is pre-defined, which includes a metadata database, a time series-related information database, and a graph database.
[0100] After the data access agent generates the access log corresponding to the data access request, the access log can be sent to the knowledge base in an asynchronous, non-blocking manner (for example, a message queue can be used or it can be directly written to a high-performance cache and then persisted to the knowledge base by the backend service), so that the following steps can be performed based on the access log to update the knowledge base.
[0101] S303: Extract the latest metadata information of the target access data from the access log, and update the corresponding metadata information of the target access data in the metadata database based on the latest metadata information.
[0102] The metadata information includes multiple metadata fields, such as data size, data type, data creation time, current data storage location, data identifier (unique identifier), etc.
[0103] The metadata database is used to store the correspondence between data and metadata information. In some implementations, considering that each data corresponds to a unique data identifier, the metadata database can store data according to the correspondence between data identifier and metadata information.
[0104] In some implementations, the metadata database can also store the corresponding index keywords for the data, which can be added after the metadata information. For example, if the data is a contract document in the economic field, its index keyword could be "economic".
[0105] S304, in response to receiving the first data query request, the metadata field carried by the first data query request is parsed to obtain the first data list corresponding to the first data query request by filtering from the metadata database based on the metadata field carried by the first data query request.
[0106] Based on the metadata database described above, if a first data query request is received, the metadata fields carried by the first data query request are parsed to obtain the first data list corresponding to the first data query request by filtering from the metadata database based on the metadata fields carried by the first data query request.
[0107] In some implementations, the first data query request may also carry only index keywords (for example, the index keyword may be "economy"), and the first data list corresponding to the first data query request can be obtained from the metadata database based on the index keywords carried by the first data query request.
[0108] In some implementations, the first data query request can carry both metadata fields and index keywords. The first data list corresponding to the first data query request can be obtained by filtering from the metadata database based on the metadata fields and index keywords carried by the first data query request.
[0109] The first data list may contain metadata information corresponding to multiple data points filtered from the metadata database. Users can access the data access interface by clicking on the metadata information.
[0110] S305, extract the access time of the target access data from the access log, and update the corresponding time series information of the target access data in the time series information database based on the access time.
[0111] Among them, the time series-related information database is used to store the correspondence between data and time series-related information.
[0112] In some implementations, considering that each data point corresponds to a unique data identifier, time series-related information databases can be stored according to the correspondence between data identifiers and time series-related information.
[0113] S306 performs content analysis on the target access data, establishes relationships between entities, data, and business events, and updates the relevant knowledge graph in the graph database based on these relationships.
[0114] In this application, natural language processing technology can be used to perform content analysis on the target access data.
[0115] The knowledge graph's node types include data files, users, customers, projects, etc., and the edge relationships include those that are visited, belong to the project, or are mentioned in the data.
[0116] In some implementations, considering that each piece of data corresponds to a unique data identifier, graph databases can store data according to the correspondence between data identifiers and knowledge graphs.
[0117] Considering that multiple data points may share common entities, a single knowledge graph can be built upon these multiple data points; that is, multiple data points may correspond to one knowledge graph. In other words, a knowledge graph can simultaneously reflect the relationships between data points.
[0118] S307, in response to receiving the second data query request, the system parses the target entity carried in the second data query request; queries the target knowledge graph containing the target entity from the graph database; and filters the second data list corresponding to the second data query request from the graph database based on the relationships in the target knowledge graph.
[0119] The second data list may contain data identifiers corresponding to multiple data points selected from the graph database. Furthermore, the corresponding metadata information may be extracted from the metadata database based on the data identifiers and displayed in the second data list.
[0120] S308, after timed or manual triggering of storage optimization in the storage database, obtains multi-dimensional information corresponding to the target storage data located in the storage database.
[0121] Specifically, the time-series information corresponding to the target stored data is retrieved from a time-series related information database. It's easy to understand that each data point corresponds to a unique data identifier; here, the time-series information corresponding to the target stored data is retrieved from the time-series related information database based on the data identifier.
[0122] Specifically, at least one of the keyword information, entity information, and full text of the target stored data is extracted to obtain the text elements corresponding to the target stored data.
[0123] Specifically, the knowledge graph corresponding to the target stored data is retrieved from the graph database. It's easy to understand that each data point corresponds to a unique data identifier; here, the knowledge graph corresponding to the target stored data can be retrieved from the graph database based on the data identifier.
[0124] Based on the time-series related information, text elements, and knowledge graph obtained above, construct multi-dimensional information corresponding to the target stored data in the storage database.
[0125] S309, based on multi-dimensional information, calls candidate models in the model cluster to perform inference and obtains multiple scores corresponding to the target stored data. These multiple scores include access probability score, content value score, and event influence score.
[0126] S310: Obtain the weights corresponding to multiple scores, and sum the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data.
[0127] S311, Based on the comprehensive heat score, determine the target storage layer corresponding to the target storage data from the hot / warm / cold / archive storage layers in the storage database.
[0128] S312, determine the current storage layer of the target storage data, and in response to the inconsistency between the target storage layer and the current storage layer, migrate the target storage data from the current storage layer to the target storage layer.
[0129] For details on the specific implementation of steps S309 to S312, please refer to the relevant parts of the above embodiments, which will not be repeated here.
[0130] Furthermore, referring to the above description, obtain the actual access information of each storage layer in the storage database; determine the storage performance evaluation index of the storage database based on the actual access information; if the storage performance evaluation index does not meet expectations, perform model fine-tuning or model retraining in the model cluster.
[0131] This application's embodiments achieve storage resource optimization through a full-link design of "data access agent + multi-dimensional knowledge base + intelligent scoring migration." This application eliminates the need for manual storage tier division, automatically assessing data popularity through a model to achieve "on-demand allocation" of storage resources, reducing ineffective storage expansion and improving storage media utilization. The goal of this application is not to move data after it becomes "cold," but to pre-deploy it to a suitable storage tier (different storage tiers correspond to different resources) before it becomes "hot," thereby fundamentally eliminating or significantly reducing access latency caused by data tier migration, especially the "cold start" problem, and meeting the stringent requirements of real-time business for high-performance access.
[0132] Figure 4 This is an exemplary schematic diagram of a data storage optimization device shown in this application, such as... Figure 4 As shown, the data storage optimization device 400 includes an information acquisition module 401, a model inference module 402, a weighted summation module 403, a storage layer determination module 404, and a data migration module 405, wherein: The information acquisition module 401 is used to acquire multi-dimensional information corresponding to the target stored data located in the storage database.
[0133] The model inference module 402 is used to infer based on multi-dimensional information by calling candidate models in the model cluster to obtain multiple scores corresponding to the target stored data. These multiple scores include access probability score, content value score, and event influence score.
[0134] The weighted summation module 403 is used to obtain the weights corresponding to multiple scores, and to perform a weighted summation on the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data.
[0135] The storage layer determination module 404 is used to determine the target storage layer corresponding to the target storage data from the hot / warm / cold / archive storage layers in the storage database based on the comprehensive heat score.
[0136] The data migration module 405 is used to determine the current storage layer of the target storage data, and in response to the inconsistency between the target storage layer and the current storage layer, migrates the target storage data from the current storage layer to the target storage layer.
[0137] The goal of this device is not to move data away after it becomes "cold," but to pre-deploy it to a suitable storage tier (different storage tiers correspond to different resources) before it becomes "hot," thereby fundamentally eliminating or significantly reducing access latency caused by data tier migration, especially the "cold start" problem, and meeting the stringent requirements of real-time business for high-performance access. This solution can truly "understand" the business significance and potential value of data, thus making storage management decisions that are far more accurate and better suited to business needs than existing technologies.
[0138] Furthermore, the multi-dimensional information includes time-series related information, text elements, and knowledge graphs. The model inference module 402 is also used to: input time-series related information into a pre-trained access probability score prediction model to obtain the access probability score of the target stored data in the target time period, wherein the time-series related information includes the access time series, access frequency information, and access latency information of the target stored data in the historical time period; input text elements into a pre-trained content value assessment model to obtain the content value score corresponding to the target stored data, wherein the text elements include at least one of the keyword information, entity information, and full text of the target stored data; and input the knowledge graph into a pre-trained event impact analysis model to obtain the event impact score corresponding to the target stored data, wherein the knowledge graph is used to store the relationship between entities, data, and business events.
[0139] Furthermore, the data storage optimization device 400 also includes a weight adjustment module, which is used to dynamically adjust the weights corresponding to multiple scores according to the changed cost and performance targets after the cost and performance targets corresponding to the storage database change.
[0140] Furthermore, the data storage optimization device 400 also includes a model performance evaluation module, used to: obtain actual access information of each storage layer in the storage database; determine the storage performance evaluation index of the storage database based on the actual access information; and fine-tune or retrain the models in the model cluster if the storage performance evaluation index does not meet expectations.
[0141] Furthermore, the data storage optimization device 400 also includes a data access proxy module, which is used to: receive data access requests sent by clients based on the data access proxy, wherein the data access requests are read requests or write requests for target access data; forward the data access requests to the storage database for data access through the data access proxy, and return the data access results to the client; and generate access logs corresponding to the data access requests.
[0142] Furthermore, the data storage optimization device 400 also includes a knowledge base module, used for: extracting the latest metadata information of the target access data from the access log, and updating the corresponding metadata information of the target access data in the metadata database based on the latest metadata information. The metadata information contains multiple metadata fields, and the metadata database is used to store the correspondence between data and metadata information; extracting the access time of the target access data from the access log, and updating the corresponding time series related information of the target access data in the time series related information database based on the access time. The time series related information database is used to store the correspondence between data and time series related information; performing content analysis on the target access data, establishing the relationship between entities, data, and business events, and updating the relevant knowledge graph in the graph database based on the relationship.
[0143] Furthermore, the information acquisition module 401 is also used to: query time-series related information corresponding to the target stored data from a time-series related information database; extract at least one of the keyword information, entity information, and full text of the target stored data to obtain the text elements corresponding to the target stored data; query the knowledge graph corresponding to the target stored data from a graph database; and construct multi-dimensional information corresponding to the target stored data based on the time-series related information, text elements, and knowledge graph.
[0144] Furthermore, the data storage optimization device 400 also includes a data query module, which, in response to receiving a first data query request, parses the metadata fields carried by the first data query request and filters the first data list corresponding to the first data query request from the metadata database.
[0145] Furthermore, the data query module is also used to: in response to receiving a second data query request, parse the target entity carried in the second data query request; query the target knowledge graph containing the target entity from the graph database; and, based on the relationships in the target knowledge graph, filter the second data list corresponding to the second data query request from the graph database.
[0146] To implement the above embodiments, this application also proposes an electronic device 500, such as... Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502 communicatively connected to the processor. The memory 502 stores instructions that can be executed by at least one processor. The instructions are executed by at least one processor 501 to implement the data storage optimization method as shown in the above embodiment.
[0147] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to implement the data storage optimization method shown in the above embodiments.
[0148] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the data storage optimization method as shown in the above embodiments.
[0149] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0150] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0151] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A data storage optimization method, characterized in that, include: Retrieve multi-dimensional information corresponding to the target stored data located in the storage database; Based on the multi-dimensional information, candidate models in the model cluster are invoked for inference to obtain multiple scores corresponding to the target stored data. The multiple scores include access probability score, content value score, and event influence score. Obtain the weights corresponding to the multiple scores respectively, and perform a weighted summation of the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data; Based on the comprehensive popularity score, the target storage layer corresponding to the target storage data is determined from the hot / warm / cold / archive storage layers in the storage database; The current storage layer of the target storage data is determined, and in response to the inconsistency between the target storage layer and the current storage layer, the target storage data is migrated from the current storage layer to the target storage layer.
2. The method according to claim 1, characterized in that, The multi-dimensional information includes time-series related information, text elements, and knowledge graphs. Based on the multi-dimensional information, candidate models in the model cluster are invoked for inference to obtain multiple scores corresponding to the target stored data, including: The time series-related information is input into a pre-trained access probability score prediction model to obtain the access probability score of the target stored data in the target time period. The time series-related information includes the access time series, access frequency information and access latency information of the target stored data in the historical time period. The text elements are input into a pre-trained content value assessment model to obtain the content value score corresponding to the target stored data. The text elements include at least one of the keyword information, entity information and full text of the target stored data. The knowledge graph is input into a pre-trained event impact analysis model to obtain the event impact score corresponding to the target stored data. The knowledge graph is used to store the relationships between entities, data, and business events.
3. The method according to claim 1 or 2, characterized in that, The method further includes: After the cost and performance targets corresponding to the storage database change, the weights corresponding to the multiple scores are dynamically adjusted according to the changed cost and performance targets.
4. The method according to claim 3, characterized in that, The method further includes: Obtain the actual access information of each storage layer in the storage database; The storage performance evaluation indicators of the storage database are determined based on the actual access information. If the storage performance evaluation metrics do not meet expectations, the models in the model cluster will be fine-tuned or retrained.
5. The method according to claim 2, characterized in that, The method further includes: The data access proxy receives data access requests sent by the client, where the data access request is a read request or a write request for the target data to be accessed. The data access proxy forwards the data access request to the storage database for data access and returns the data access result to the client. Generate access logs corresponding to the data access requests.
6. The method according to claim 5, characterized in that, After generating the access log corresponding to the data access request, the method further includes: The latest metadata information of the target access data is extracted from the access log, and the metadata information corresponding to the target access data in the metadata database is updated based on the latest metadata information. The metadata information contains multiple metadata fields, and the metadata database is used to store the correspondence between data and metadata information. From the access log, the access time of the target access data is extracted, and the time series related information corresponding to the target access data in the time series related information database is updated based on the access time. The time series related information database is used to store the correspondence between data and time series related information. Content analysis is performed on the target access data to establish relationships between entities, data, and business events, and the relevant knowledge graph in the graph database is updated based on these relationships.
7. The method according to claim 6, characterized in that, The method for obtaining the multi-dimensional information includes: The time series-related information corresponding to the target stored data is obtained by querying the time series-related information database. Extract at least one of the keyword information, entity information, and full text of the target stored data to obtain the text elements corresponding to the target stored data; The knowledge graph corresponding to the target stored data is obtained by querying the graph database. Multi-dimensional information corresponding to the target stored data is constructed based on the time series information, the text elements, and the knowledge graph.
8. The method according to claim 6, characterized in that, The method further includes: In response to receiving a first data query request, the metadata fields carried by the first data query request are parsed to obtain the metadata fields. The first data list corresponding to the first data query request is obtained by filtering from the metadata fields carried in the first data query request from the metadata database.
9. The method according to claim 6, characterized in that, The method further includes: In response to receiving a second data query request, the target entity carried in the second data query request is parsed and obtained. Query the target knowledge graph containing the target entity from the graph database; Based on the relationships in the target knowledge graph, a second data list corresponding to the second data query request is obtained by filtering from the graph database.
10. A data storage optimization device, characterized in that, include: The information acquisition module is used to acquire multi-dimensional information corresponding to the target stored data located in the storage database; The model inference module is used to infer multiple scores corresponding to the target stored data based on the multi-dimensional information, including access probability score, content value score and event influence score. The weighted summation module is used to obtain the weights corresponding to the multiple scores respectively, and to perform a weighted summation on the multiple scores based on the weights to obtain the comprehensive popularity score of the target stored data; The storage layer determination module is used to determine the target storage layer corresponding to the target storage data from the hot / warm / cold / archive storage layers in the storage database based on the comprehensive heat score. The data migration module is used to determine the current storage layer of the target storage data, and in response to the inconsistency between the target storage layer and the current storage layer, migrate the target storage data from the current storage layer to the target storage layer.
11. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.
13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-9.