A data processing method, apparatus, device, medium, and program product

By designing a three-layer architecture of memory, buffer, and hard disk storage areas and adopting an asynchronous eviction strategy, the problem of high storage costs for large language models is solved, achieving efficient memory utilization and optimized storage costs.

CN123068412APending Publication Date: 2026-11-10GUANGZHOU TENCENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510596225.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

With the development of deep learning and artificial intelligence technologies, the size of models has exploded. Traditional parameter storage solutions can no longer economically manage the hundreds of billions of parameters in large language models, resulting in excessively high storage costs.

Method used

The design incorporates a three-tiered storage architecture consisting of a memory storage area, a buffer storage area, and a hard disk storage area. An asynchronous eviction strategy is employed to enable data flow between different storage tiers, thereby optimizing memory utilization and reducing storage costs.

Benefits of technology

Through a three-tier storage architecture and asynchronous eviction strategy, the system can manage data exceeding memory capacity, optimize memory utilization, and reduce storage costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068412A_ABST
    Figure CN123068412A_ABST
Patent Text Reader

Abstract

This application discloses a data processing method, apparatus, device, medium, and program product for reducing storage costs during data storage. It can be applied to fields such as computers and artificial intelligence. The method includes: acquiring first data and a first index, wherein the first index is the index of the first data when it is written to a memory storage area; writing the first data to the memory storage area based on the first index; when the first data conforms to an asynchronous eviction policy, allocating a first identifier and a second index to the first data, and adding the first data to a write queue in a cache storage area, wherein the first identifier indicates that the first data will be written to the cache storage area, and the second index is the index of the first data when it is written to the cache storage area; writing the first data to the cache storage area based on the write queue and the second index; allocating a third index to the first data, wherein the third index is the index of the first data when it is written to a hard disk storage area; and writing the first data to the hard disk storage area based on the third index.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of deep learning and artificial intelligence technologies, model size has exploded. From early models with millions of parameters to today's models with hundreds of billions or even trillions of parameters, unprecedented challenges have been brought to parameter storage and access systems. As a core component of large-scale machine learning systems, parameter servers are responsible for storing, managing, and providing efficient access to model parameters. Their performance and scalability directly affect the efficiency of the entire model training and inference system.

[0003] Traditional parameter storage solutions typically rely on distributed key-value stores or dedicated in-memory parameter servers. Early systems used a parameter server-based architecture, distributing model parameters across multiple server nodes. Subsequent deep learning frameworks also adopted similar distributed parameter management schemes, coordinating parameter storage and updates through network communication. However, with the emergence of large-scale language models, the number of model parameters has exceeded hundreds of billions, making it no longer economically feasible to rely solely on in-memory parameter storage.

[0004] Therefore, there is an urgent need for a solution that can reduce the storage cost of parameter storage. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, medium, and program product for reducing storage costs when storing data.

[0006] In view of this, this application provides a data processing method, comprising: acquiring first data and a first index table, the first index table including a first index, the first index being the index of the first data when written to a memory storage area; writing the first data to the memory storage area based on the first index; when the first data conforms to an asynchronous eviction policy, allocating a first identifier and a second index to the first data, and adding the first data to a write queue in a cache storage area, the first identifier being used to indicate that the first data will be written to the cache storage area, the second index being the index of the first data when written to the cache storage area; writing the first data to the cache storage area based on the write queue and the second index, and updating the first index table to the second index table, the second index table including the first index and the second index; allocating a third index to the first data, the third index being the index of the first data when written to a hard disk storage area; writing the first data to the hard disk storage area based on the third index, and updating the second index table to the third index table, the third index table including the first index, the second index, and the third index.

[0007] Another aspect of this application provides a data processing apparatus, including: an acquisition module, configured to acquire first data and a first index table, the first index table including a first index, the first index being the index of the first data when it is written into a memory storage area;

[0008] The processing module is configured to: write the first data to the memory storage area based on the first index; when the first data meets the asynchronous eviction policy, allocate a first identifier and a second index to the first data, and add the first data to the write queue of the cache storage area, wherein the first identifier is used to indicate that the first data will be written to the cache storage area, and the second index is the index when the first data is written to the cache storage area; write the first data to the cache storage area based on the write queue and the second index, and update the first index table to the second index table, wherein the second index table includes the first index and the second index; allocate a third index to the first data, wherein the third index is the index when the first data is written to the hard disk storage area; write the first data to the hard disk storage area based on the third index, and update the second index table to the third index table, wherein the third index table includes the first index, the second index, and the third index.

[0009] In one possible design, in another implementation of another aspect of the embodiments of this application, the memory storage area is managed based on a least recently used mechanism, or based on a least-used number of times mechanism, or based on an adaptive replacement cache, or based on a clock algorithm.

[0010] This memory storage area is shared memory.

[0011] In one possible design, in another implementation of another aspect of the embodiments of this application, when the memory storage area is managed based on the least recently used mechanism, the processing module is used to check the most recent access time of the first data and the memory usage information of the memory storage area; when the memory usage information is greater than a first threshold and the most recent access time of the first data meets a first condition, it is determined that the first data conforms to the asynchronous eviction policy, wherein the first condition is that in the sorting based on the most recent access time from the nearest to the furthest, the most recent access time of the first data is ranked in the last N positions, where N is a positive integer.

[0012] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to write the first data into the memory storage area based on the first index and the write-time comparison mechanism.

[0013] In one possible design, in another implementation of another aspect of the embodiments of this application, when the first data is the updated data of the target parameter, the processing module is used to perform a write-time comparison on the first data to obtain a comparison result;

[0014] When the comparison result indicates a successful comparison, the second data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The second data is the data of the target parameter that has been written to the memory storage area.

[0015] When the comparison result indicates a failure, the update time of the first data is compared with that of the third data of the target parameter, where the third data is the data of the target parameter that has been written into the memory storage area.

[0016] When the update time of the first data is later than the update time of the third data, the first data is written to the memory storage area based on the first index;

[0017] If the update time of the first data is earlier than the update time of the third data, the writing of the first data is abandoned.

[0018] In one possible design, in another implementation of another aspect of the embodiments of this application, after performing a write-time comparison on the first data to obtain a comparison result, the processing module is configured to retry the write-time comparison of the first data when the comparison result indicates that the first memory region indicated by the first index has been written with the fourth data, until the fourth data is written completely, and then allocate a second memory region for the first data, the second memory region being a memory region that follows the first memory region and is available, and the fourth data being different data with different keys but the same index as the first data;

[0019] The first data is written to the second memory region based on the first index, and the first index is updated.

[0020] In one possible design, in another implementation of another aspect of the embodiments of this application, before assigning a first identifier and a second index to the first data and adding the first data to the write queue of the cache storage area, the processing module is used to traverse multiple write buffers of the cache storage area to obtain traversal results. The traversal results are used to indicate whether there is duplicate data of the first data or expired data of the target parameter corresponding to the first data in the cache storage area.

[0021] When the traversal result indicates the existence of duplicate data of the first data or expired data of the target parameter corresponding to the first data, delete the duplicate data of the first data or the expired data of the target parameter corresponding to the first data.

[0022] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to traverse the number of valid keys and the number of invalid keys that have stored data in multiple write buffers of the buffer storage area.

[0023] When the ratio of the number of valid keys to the number of invalid keys is greater than a first threshold, multiple write buffers of the buffer storage area are traversed to obtain the traversal result.

[0024] When the ratio of the number of valid keys to the number of invalid keys is less than or equal to the first threshold, multiple write buffers of the buffer storage area are periodically triggered to obtain the traversal result.

[0025] In one possible design, in another implementation of another aspect of the embodiments of this application, after the first data is written to the hard disk storage area based on the third index and the second index table is updated to the third index table, the acquisition module is used to acquire a read request, the read request indicating that the first data is read;

[0026] The processing module is used to obtain the third index table of the first data based on the read request. The third index table includes the first index, the second index, and the third index.

[0027] Based on the first index, a query is performed in the memory storage area to obtain the first query result;

[0028] When the first query result indicates that the first data exists in the memory storage area, the first data is read from the memory storage area;

[0029] When the first query result indicates that the first data does not exist in the memory storage area, a query is performed in the buffer storage area based on the second index to obtain the second query result;

[0030] When the second query result indicates that the first data exists in the buffer storage area, the first data is read from the buffer storage area;

[0031] When the second query result indicates that the first data does not exist in the buffer storage area, a query is performed in the hard disk storage area based on the third index to read the first data from the hard disk storage area.

[0032] In one possible design, in another implementation of another aspect of the embodiments of this application, after querying the hard disk storage area based on the third index and reading the first data from the hard disk storage area, the processing module is used to write the first data to the memory storage area based on the first index and the write comparison mechanism.

[0033] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform a write-time comparison on the first data to obtain a second comparison result;

[0034] When the second comparison result indicates a successful comparison, the fifth data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The fifth data is the data of the target parameter that has been written to the memory storage area.

[0035] If the second comparison result indicates a failure, the writing of the first data is abandoned.

[0036] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module is used to acquire a snapshot creation request, the snapshot creation request being used to instruct the creation of a target snapshot including the first data;

[0037] The processing module is used to acquire a global write lock in response to the snapshot creation request, generate a snapshot identifier for the target snapshot, and initialize snapshot metadata, which includes snapshot creation time, snapshot version information, and snapshot storage area identifier.

[0038] Based on the snapshot identifier and the initial snapshot metadata, the hot data of the memory storage area and the cached data of the buffer storage area are dumped, and the cold data of the hard disk storage area is dumped through hard links to generate an intermediate snapshot;

[0039] Based on the first index table, the second index table, and the third index table, the intermediate snapshot is deduplicated to obtain the target snapshot, and the global write lock is released.

[0040] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to obtain the start position, end position and space size of the target snapshot based on the snapshot storage area identifier;

[0041] The snapshot copy space of the target snapshot is created based on the space size and the end position, wherein the end position serves as the start position of the snapshot copy space, and the elements stored in the snapshot copy space are used to indicate the element positions of the target snapshot.

[0042] The starting position is reassigned based on the end position of the snapshot copy space;

[0043] After the global write lock is released, the backlogged data accumulated during the global write lock period is written based on the allocation start position.

[0044] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is configured to determine the end position of memory garbage collection based on the start position of the target snapshot during the dumping process of the target snapshot; and perform memory garbage collection based on the end position.

[0045] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform modulo processing on the allocation head with the maximum memory address when the allocation head exceeds the maximum memory address of the memory storage area, so as to reset the allocation head to the minimum memory address, wherein the allocation head is the storage tail of the backlog data.

[0046] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0047] The memory is used to store programs;

[0048] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0049] Bus systems are used to connect memory and processor to enable communication between them.

[0050] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0051] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0052] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: a three-layer storage architecture of memory storage area, buffer storage area and hard disk storage area is designed for the same data, and during the storage process, the asynchronous eviction strategy is used to realize the flow storage of data between different storage layers. This enables the system to manage data exceeding the memory capacity, thereby optimizing memory utilization and reducing memory storage costs. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of an application system architecture for data processing in an embodiment of this application;

[0054] Figure 2This is a schematic diagram of a data processing flow of the application system in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of one embodiment of data processing in this application.

[0056] Figure 4 This is a schematic diagram of a storage area design for data storage in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of one embodiment of data reading in this application.

[0058] Figure 6 This is a flowchart illustrating the data writing and data reading processes in an embodiment of this application.

[0059] Figure 7 This is a schematic diagram of an embodiment generated from a snapshot in this application.

[0060] Figure 8 This is a schematic diagram of the snapshot generation process in an embodiment of this application;

[0061] Figure 9 This is a schematic diagram of a snapshot memory design in an embodiment of this application;

[0062] Figure 10 This is a schematic diagram of one embodiment of the data processing device in this application;

[0063] Figure 11 This is a schematic diagram of one embodiment of the server in this application;

[0064] Figure 12 This is a schematic diagram of one embodiment of the terminal in this application. Detailed Implementation

[0065] This application provides a data processing method, apparatus, device, medium, and program product for reducing storage costs when storing data.

[0066] It should be noted that the terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Unless the context clearly indicates otherwise, the singular forms "a," "one," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. The quantities of "a plurality" or "multiple copies," etc., mentioned in the embodiments of this application all refer to a quantity of "at least two," for example, "a plurality" means "at least two," and "multiple copies" means "at least two copies." It should be understood that such data used can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "corresponding to," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0067] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0068] With the rapid development of deep learning and artificial intelligence technologies, model size has exploded. From early models with millions of parameters to today's models with hundreds of billions or even trillions of parameters, unprecedented challenges have been brought to parameter storage and access systems. As a core component of large-scale machine learning systems, parameter servers are responsible for storing, managing, and providing efficient access to model parameters; their performance and scalability directly impact the efficiency of the entire model training and inference system. Traditional parameter storage solutions typically rely on distributed key-value storage systems or dedicated in-memory parameter servers. Early systems used a parameter server-based architecture, distributing model parameters across multiple server nodes. Subsequent deep learning frameworks also adopted similar distributed parameter management schemes, coordinating parameter storage and updates through network communication. However, with the emergence of large language models, the scale of model parameters has exceeded hundreds of billions, making in-memory storage no longer economically feasible. Therefore, a solution that can reduce the storage cost of parameter storage is urgently needed.

[0069] To address the aforementioned problems, this application provides the following technical solution: Obtaining first data and a first index table, the first index table including a first index, which is the index of the first data when written to a memory storage area; writing the first data to the memory storage area based on the first index; when the first data conforms to an asynchronous eviction policy, allocating a first identifier and a second index to the first data, and adding the first data to a write queue in a cache storage area, the first identifier indicating that the first data will be written to the cache storage area, and the second index being the index of the first data when written to the cache storage area; writing the first data to the cache storage area based on the write queue and the second index, and updating the first index table to the second index table, the second index table including the first index and the second index; allocating a third index to the first data, the third index being the index of the first data when written to a hard disk storage area; writing the first data to the hard disk storage area based on the third index, and updating the second index table to the third index table, the third index table including the first index, the second index, and the third index. This three-tiered storage architecture, consisting of a memory storage area, a buffer storage area, and a hard disk storage area, is designed for the same data. During the storage process, an asynchronous eviction strategy is used to enable data to flow between different storage layers. This allows the system to manage data exceeding the memory capacity, thereby optimizing memory utilization and reducing memory storage costs.

[0070] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown in the diagram includes an offline computing component 101, a training parameter server 102, a streaming data processing component 103, an inference parameter server 104, an online inference server 105, and a streaming sample processing component 106. The offline computing component 101 trains the target model. The training parameter server 102 acquires the model structure, dense parameters, and sparse parameters obtained after training and sends these parameters to the streaming data processing component 103 for storage. The streaming data processing component 103 sends the model structure and dense parameters to the online inference server 105 for storage and the sparse parameters to the inference parameter server 104 for storage. Thus, when the online inference server 105 runs the target model, it can obtain the corresponding sparse parameters from the inference parameter server 104 and read the model structure and dense parameters from its own storage database. Based on the read sparse parameters, model structure, and dense parameters, the target model can then output the corresponding model inference results. The inference parameter server 104 can use the data processing method provided in this application when storing the sparse parameters, thereby enabling the inference parameter server 104 to manage data exceeding the memory capacity, thereby optimizing memory utilization and reducing memory storage costs.

[0071] The offline computing component 101, streaming data processing component 103, and streaming sample processing component 106 can be various terminal devices or servers. When they are terminal devices, they can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The training parameter server 102, inference parameter server 104, online inference server 105, etc., can all be implemented using independent servers or server clusters composed of multiple servers. Servers can be cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution platforms, and big data and artificial intelligence platforms, or they can be physical servers.

[0072] This application relates to artificial intelligence (AI) technologies, such as storing model parameters of machine learning models. Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0073] This application also relates to machine learning (ML) in artificial intelligence technology. For example, this application requires training a machine learning model to obtain model parameters. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0074] In one embodiment, a data processing system is provided, including a training parameter server, a streaming data processing component, an inference parameter server, and an online inference server. The training parameter server is used to acquire sparse and dense parameters of a target model generated through model update training, and send the sparse and dense parameters to the streaming data processing server. The target model is a machine learning model whose update frequency of sparse and dense parameters satisfies a high-frequency condition. When it is determined that the sparse and dense parameters cached by the streaming data processing component meet the parameter transfer condition, the sparse parameters in the streaming data processing component are loaded into the memory of the inference parameter server through multi-threaded parallel processing, while the dense parameters are loaded into the online inference server. The online inference server is used to read the corresponding sparse parameters from the memory of the inference parameter server based on the input query key value when using the target model, so as to output the corresponding model inference result based on the read sparse parameters by the target model.

[0075] The following sections elaborate on the specific components used together to implement the data processing method in this application: the training parameter server, the streaming data processing component, the inference parameter server, and the online inference server.

[0076] For the training parameter server, reference Figure 2 The training parameter server deploys a model read / write module and a model transfer module. The model read / write module retrieves model parameters (i.e., sparse and dense parameters, as well as dense and sparse parameter updates) obtained through model training from the offline computing server. The model transfer module sends the retrieved incremental model parameters to the streaming data processing component. The model parameters retrieved by the model read / write module can be either the initial version or an updated version. For example, the model parameters obtained during the initial training of the target model are the initial version, while the parameters obtained during updated training of the target model are the updated version. To make the model inference results output by the target model more accurate, the target model can be continuously trained and updated. The target model is a machine learning model whose parameter update frequency meets the high-frequency condition; for example, it could be an information recommendation model, such as a music recommendation model or a news recommendation model.

[0077] The streaming data processing component deploys a streaming data processing platform, which includes a parameter storage module. When the streaming data processing component receives model parameters, the parameter storage module stores these parameters. Model parameters refer to the parameters of the target model obtained through model training. Model parameters include sparse parameters and dense parameters. Sparse parameters refer to the parameters required as input to the machine learning model, while dense parameters refer to the node information of the machine learning model. For example, when the machine learning model is an information recommendation model, sparse data can be word embeddings of features of the candidate information to be recommended, word embeddings of features of the target object receiving the recommendation information, or word embeddings of the cross-features of the candidate information to be recommended and the target object, etc. In this application, the parameter storage module can be constructed in the form of dense parameter publish / subscribe and sparse parameter publish / subscribe.

[0078] When the streaming data processing component determines that the sparse and dense parameters meet the parameter transfer conditions, it reads the sparse parameters from the streaming data processing component in parallel using multiple threads and loads them into the inference parameter server. Simultaneously, it reads the dense parameters from the streaming data processing component in parallel using multiple threads and loads them into the online inference server. The parameter transfer conditions can be freely set according to requirements. For example, the parameter transfer conditions can be determined when it is determined that there are stacked model parameters in the streaming data processing component, that is, when the number of model parameters to be transferred in the streaming data processing component is greater than or equal to a preset threshold.

[0079] Furthermore, the online inference server deploys an online inference service. When it's necessary to output the corresponding model inference result from the target model—for example, when it's necessary to output recommendation information from the target model—the online inference service can determine the query key. The query key refers to the key value of the sparse parameters to be queried; for example, the query key value could be the key value of the target object's features or information features. The online inference service reads the corresponding sparse parameters from the memory of the inference parameter server based on the input query key value. Since the sparse parameters can be the parameters required by the model's input, the online inference service can trigger the target model to output the model inference result based on the read sparse parameters, such as outputting recommendation information. Figure 2 A schematic diagram of a data processing system in one embodiment is shown.

[0080] It should be understood that since the target model is continuously trained and updated, the sparse parameters stored in the inference parameter server also need to be continuously updated. Therefore, the data processing system described above can continuously store the latest version of the sparse parameters obtained from the updated target model training in the inference parameter server. This allows the target model in the online inference server to output more accurate model inference results based on the latest version of the sparse parameters. The number of sparse parameters stored in the inference parameter server is quite large, for example, in the tens of millions. Therefore, the inference parameter server can be used for distributed storage of these tens of millions of sparse parameters.

[0081] In the aforementioned data processing system, by setting up a streaming data processing component, model parameters can be cached. The streaming data processing component then reads the model parameters in parallel using multiple threads, and loads the sparse parameters from these parameters onto the inference parameter server. By loading the sparse parameters onto the inference parameter server, when the online inference server uses the target model, it can retrieve the corresponding sparse parameters from the inference parameter server based on the query key value. This allows the target model to output more accurate model inference results based on the retrieved sparse parameters. In this application, the inference parameter server uses a three-tier storage architecture—memory storage, buffer storage, and hard disk storage—to store the sparse parameters. During storage, an asynchronous eviction strategy is used to enable data flow between different storage layers. This allows the system to manage data exceeding memory capacity, thereby optimizing memory utilization and reducing memory storage costs.

[0082] The data processing system also includes an offline computing component and a streaming sample processing component. The streaming sample processing component is used for training data and performs processes such as sample cleaning, sample aggregation, and feature concatenation to obtain training samples. This streaming sample processing component sends the training samples to the offline computing component. The offline computing component is used to update and train the target model based on the training samples, obtaining the model structure, sparse parameters, and dense parameters. Sparse parameters refer to the parameters required for the target model's input; dense parameters refer to the parameters representing the node information of the target model. The training parameter server is used to acquire the sparse parameters, dense parameters, and model structure, and send them to the streaming data processing component, and also to the online inference server. The online inference server, when using the target model, reads the dense parameters and model structure from itself, and reads the corresponding sparse parameters from the memory of the inference parameter server cluster based on the input query key value, so that the target model can output model inference results based on the read dense parameters, model structure, and model parameters.

[0083] Specifically, refer to Figure 2 The data processing system also includes an offline computing component. The offline computing server deploys an offline computing platform, which includes a sample processing module, a distributed model training module, and a gradient update module. The sample processing module processes training samples to obtain the input to the target model. The distributed model training module trains the model to obtain model data. The gradient update module provides gradient update strategies for the distributed model training. The model data includes sparse parameters, model structure, and dense parameters. Dense parameters refer to parameters that characterize the node information of the target model.

[0084] Furthermore, the training parameter server acquires sparse parameters, dense parameters, and model structure, and sends these to the streaming data processing component. The streaming data processing component then transfers the received sparse parameters to the inference parameter server and the received dense parameters and model structure to the online inference server. When the online inference server performs online inference, it can read the dense parameters and model structure from itself and retrieve the corresponding sparse parameters from the inference parameter server based on the query key. The target model then outputs the model inference results based on the read dense parameters, model structure, and sparse parameters. Figure 2 A schematic diagram of a data processing system in one embodiment is shown. It should be understood that the online inference server can also store the dense parameters and model structure through a distributed storage system, thereby reducing the storage cost of the online inference server.

[0085] In this embodiment, the online inference server can read the latest version of the model data from the inference parameter server, and then output more accurate model inference results based on the latest version of the model data.

[0086] It is understood that in the specific implementation of this application, data related to model structure and model parameters are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0087] Based on the above introduction, the data processing method in this application will be described below, taking the server as the execution entity. Please refer to [link / reference needed]. Figure 3 One embodiment of the data processing method in this application includes:

[0088] 301. Obtain first data and a first index table, wherein the first index table includes a first index, which is the index of the first data when it is written to the memory storage area.

[0089] In this embodiment, the server receives a write request for the first data; the server obtains the query key value of the first data through the write request, and then obtains the corresponding hash value based on the query key value using a hash algorithm; the server calculates the corresponding first index in the first index table based on the hash value.

[0090] It should be understood that during the writing process, if the first data is the data being written for the first time, there will be no index corresponding to the first data in the first index table. In this case, the server will create an index for it in the first index table to indicate the write position of the first data in the memory storage area. If the first data is updated data, there will be a historical index corresponding to the first data in the first index table, that is, the historical index will be used as the first index.

[0091] In this embodiment, the first index table can be a lock-free data structure, such as a lock-free hash table or a lock-free skip list. The design of these lock-free data structures relies on atomic operations (Compare-And-Swap, CAS) and clever marking mechanisms to achieve thread-safe high-concurrency operations while avoiding performance bottlenecks caused by locks. These data structures are widely used in high-throughput, multi-threaded scenarios (such as real-time recommendation systems, cache management, and event processing).

[0092] The following explanation uses a lock-free hash table as an example to illustrate the routing process of this index table. The index table of a lock-free hash table is a data structure implemented using an array or linked list to store hash buckets. Each hash bucket may contain a linked list or other data structure (such as a tree) to store conflicting data (i.e., multiple keys mapping to the same hash value). Based on this, the lock-free index table can be composed as follows: a fixed-size array, where each element points to a hash bucket. The size of the index table is typically a power of 2 (e.g., 16, 32, etc.) to optimize hash calculations. Hash Bucket: Each element of the index table corresponds to a hash bucket, which can be a linked list, a dynamic array, or a binary tree. Therefore, in a lock-free hash table, the first step in writing data is to calculate the hash value of the key using a hash function and then locate the specific hash bucket based on the size of the index table. Assuming the first index table is "Index Table = [Bucket0, Bucket1, Bucket2, Bucket3, Bucket4, Bucket5, Bucket6, Bucket7]", the key-value pair corresponding to the first data can be represented as "(key1, value1)" when written. During the write process, hash(key) is calculated, assuming the result is 12345. The index is calculated as: index = hash(key1) % 8 = 12345 % 8 = 1. The index is then located at the first position in the index table (Bucket1).

[0093] In this embodiment, to enable data to flow between different storage areas, the memory storage area needs to be designed with a cache eviction mechanism. This mechanism determines how to remove certain cache entries to free up space for new data when the cache is full. This is particularly important in systems with limited storage resources (such as memory caches and disk caches), as it helps to find a balance between performance and storage utilization. The cache eviction mechanism can include several possible implementations: data management based on a Least Recently Used (LRU) mechanism, a Least Frequently Used (LRU) mechanism, an adaptive cache replacement mechanism, or a clock algorithm.

[0094] The principle behind Least Recently Used (LRU) is to evict the least recently used data, prioritizing the removal of entries that haven't been accessed for a long time. This can be implemented using doubly linked lists or hash tables. Doubly linked lists record the access order of data, with the most recently accessed entry moving to the head of the list. Hash tables, on the other hand, enable fast data lookup.

[0095] The principle of Least Frequently Used (LFU) is to eliminate the least frequently accessed data, that is, to remove the entries with the fewest accesses first. In the implementation, a hash table is used to record the access frequency of each piece of data. A min-heap or bucketing structure is used to quickly find the least frequently accessed data.

[0096] The principle behind this adaptive replacement cache is to dynamically adjust the weights of Least Recently Used (LRU) and Least Frequently Used (LFU) data, combining the advantages of both. During implementation, two queues are maintained: one for recently accessed data (LRU) and the other for frequently accessed data (LFU). The sizes of the two queues are dynamically adjusted based on access patterns.

[0097] The principle behind this clocking algorithm is to set a timeout for each cached entry, which is then automatically removed upon expiration. In its implementation, a priority queue or a time wheel timer is used to efficiently manage expiration times.

[0098] To ensure maximum data integrity and rapid recovery in the event of container restarts or other abnormal situations, this memory storage area can be designed as shared memory. Shared memory is a memory region provided by the operating system, which multiple processes can map to share data. It's a direct memory access communication method that avoids data copying between processes. At system startup, a process creates a shared memory segment through the operating system's shared memory interface, returning a shared memory identifier. Other processes map this shared memory segment into their own address spaces using the shared memory identifier, thus achieving cross-process data sharing. This allows other processes to directly read and write data in the shared memory. When a process no longer needs the shared memory, it can unmap it. Once all processes have unmapped it, the shared memory segment can be deleted. For example, if process 1 and process 2 are mapped to the same shared memory, both processes can read or modify the data in the shared memory. If process 1 crashes and the operating system restarts process 1, the restarted process 1 can still directly access the data in the shared memory, preventing data loss and ensuring service stability.

[0099] 302. Write the first data into the memory storage area based on the first index.

[0100] In this embodiment, after the server determines the first index of the first data, it writes the first data into the memory storage area based on the first index and the write comparison mechanism.

[0101] It should be understood that this write comparison mechanism can be a possible implementation of atomic operations (CAS). The specific process can be as follows: Perform a write-time comparison on the first data to obtain a comparison result; if the comparison result indicates success, add the second data of the target parameter to the delayed deletion queue, and write the first data to the memory storage area based on the first index; the second data represents the data of the target parameter currently written to the memory storage area; if the comparison result indicates failure, compare the update time of the first data with the update time of the third data of the target parameter; the third data represents the data of the target parameter currently written to the memory storage area; if the update time of the first data is later than the update time of the third data, write the first data to the memory storage area based on the first index; if the update time of the first data is earlier than the update time of the third data, abandon the writing of the first data.

[0102] In one exemplary scheme, it is assumed that the first data is written in the form of "key1, value1" during the writing process. At this time, the first index of the first data is determined by calculating the hash value of "key1", and then the write position of the first data in the memory storage area is determined based on the first index; and the historical data at the write position is read; finally, the historical data is compared with the first data.

[0103] If the historical data at the write location is different from the first data, a write operation is performed. This involves adding the historical data to a delayed deletion queue for delayed deletion, followed by memory resource release and reclamation. Then, the first data is written to the write location, and the data in the first index table is updated. This ensures thread safety in multi-threaded environments if other threads read the old value at this time. Furthermore, after adding the historical data to the delayed deletion queue, mechanisms such as reference counting or memory barriers can be used to determine the deletion time of the historical data (i.e., dynamically determining the delay time of the historical data in the delayed deletion queue). For example, reference counting can be used to determine whether the historical data has been referenced by all threads, and the historical data can only be deleted after all references have been completed. Suppose threads 1, 2, and 3 reference the historical data, and after the historical data is marked as deleted, they have not yet completed their corresponding operations; in this case, the reference count of the historical data can be marked as 3, and it can be added to the delayed deletion queue. After thread 1 finishes referencing the historical data, the reference count of the historical data is modified to 2. This process is repeated until the reference count of the historical data reaches 0, at which point the historical data can be deleted.

[0104] If the historical data at the write position is the same as the first data, the comparison fails. In this case, the update times of the historical data and the first data can be compared. If the update time of the first data is later than the update time of the historical data (i.e., the first data is updated), then the write operation is executed. The historical data is added to a delayed deletion queue for delayed deletion, and memory resources are released and reclaimed after deletion. Then, the first data is written to the write position, and the data in the first index table is updated. This ensures thread safety in a multi-threaded environment if other threads read the old value at this time.

[0105] If the update time of the first data is earlier than the update time of the historical data (i.e., the first data is historical data), then the write operation of the first data is abandoned.

[0106] It should be understood that if the first data is data that floats from the hard disk storage area, then during the comparison process at the write time, if the historical data at the write location is the same as the first data, then the comparison is determined to have failed. At this time, it can also be assumed that the memory already has the latest value or that the data stored in the memory is also the first data, that is, the writing of the first data is abandoned.

[0107] It should be understood that if a conflict occurs in the memory storage location indicated by the first index during the routing phase of the first data, the server can retry the write-time comparison until the conflicting data has been processed. Then, a new memory storage location is allocated for the first data, and the first data is written to that memory storage location based on the first index, and the index table is updated. The specific process can be as follows: When the comparison result indicates a failure and the first memory region indicated by the first index has already been used to write the fourth data, the write-time comparison of the first data is retried until the fourth data is written. Then, a second memory region is allocated for the first data. This second memory region is the available memory region after the first memory region. The fourth data is different data with a different key but the same index as the first data. The first data is then written to the second memory region based on the first index. It should be understood that, to avoid deadlocks during this process, a spinlock or a mechanism to limit the number of retry comparisons can be set when retrying write-time comparisons.

[0108] 303. When the first data meets the asynchronous eviction policy, a first identifier and a second index are assigned to the first data, and the first data is added to the write queue of the cache storage area. The first identifier is used to indicate that the first data will be written to the cache storage area, and the second index is the index when the first data is written to the cache storage area.

[0109] In this embodiment, after the first data is written to the memory storage area, the memory storage area can manage the eviction of the first data based on a cache eviction mechanism. After the first data meets the asynchronous eviction policy, a first identifier can be added to the first data indicating that it has been written to the cache storage area, and a second index can be allocated to the first data for writing to the cache storage area.

[0110] It should be understood that this asynchronous eviction policy is related to the cache eviction mechanism of the memory storage area. In an exemplary scenario, when the cache eviction mechanism of the memory storage area is the Least Recently Used (LRU) mechanism, the asynchronous eviction policy can be as follows:

[0111] The server checks the most recent access time of the first data and the memory usage information of the memory storage area; when the memory usage information is greater than a first threshold and the most recent access time of the first data meets a first condition, it determines that the first data conforms to the asynchronous eviction policy, wherein the first condition is that in the sorting based on the most recent access time from the nearest to the furthest, the most recent access time of the first data is ranked in the last N positions, where N is a positive integer.

[0112] or,

[0113] The server checks the most recent access time of the first data; when the most recent access time of the first data meets the first condition, it determines that the first data conforms to the asynchronous eviction policy, wherein the first condition is that the most recent access time of the first data ranks in the last N positions in the sorting based on the most recent access time from the nearest to the furthest, and N is a positive integer.

[0114] In this embodiment, the first identifier

[0115] For example, suppose the capacity of the memory storage region is 3, and the access sequence is A→B→C→D→A→C→E. Then the asynchronous eviction policy of this LRU can be expressed as follows:

[0116] Initial cache state: [empty].

[0117] Access A: Cache status: [A].

[0118] Accessing B: Cache state: [B, A].

[0119] Accessing C: Cache state: [C, B, A].

[0120] Accessing D: Cache is full, evict the least used A, cache status: [D, C, B].

[0121] Accessing A: Cache is full, the least used B is evicted, cache status: [A, D, C].

[0122] Accessing C: Cache hit, update order, cache state: [C, A, D].

[0123] Accessing E: Cache is full, the least used D is evicted, cache status: [E, C, A].

[0124] It should be understood that the above process is only an exemplary solution. In practical applications, the capacity of the memory storage area is much greater than 3, and the amount of data in the access sequence is also very large. Therefore, the corresponding parameters of the eviction mechanism can be set according to the actual application requirements. For example, when the cache occupancy of the memory storage area exceeds 80%, the 100 data items with the lowest access times can be asynchronously evicted.

[0125] Meanwhile, when allocating the second index to the first data, the storage location of the first data in the buffer storage area can be determined based on the key value of the first data; finally, the second index of the first data in the buffer storage area is determined based on the key value and the storage location.

[0126] In practical applications, to save storage space and reduce the number of writes to the hard drive storage area, thereby effectively reducing hard drive wear, data compression is required during the data writing process in the buffer storage area. In an exemplary scheme, taking the first data as an example, the data compression process can be as follows: Traverse multiple write buffers of the buffer storage area to obtain traversal results. These results indicate whether there is duplicate data of the first data or expired data of the target parameter corresponding to the first data within the buffer storage area. When the traversal results indicate the existence of duplicate data of the first data or expired data of the target parameter corresponding to the first data, delete the duplicate data of the first data or the expired data of the target parameter corresponding to the first data.

[0127] In practical applications, this server can also reduce fragmentation by reorganizing data and storing valid data in a continuous manner.

[0128] After data compression, the server also needs to update the index so that the index points to the new data location.

[0129] Meanwhile, since data compression is performed asynchronously, the data compression speed needs to be dynamically adjusted based on memory limitations and the available percentage of the buffer storage area to avoid affecting normal write and memory eviction processes. An exemplary solution could be as follows: The server iterates through the multiple write buffers in the buffer storage area, counting the number of valid keys and invalid keys. When the ratio of valid keys to invalid keys is greater than a first threshold, it triggers a traversal of the multiple write buffers in the buffer storage area to obtain the traversal result. When the ratio of valid keys to invalid keys is less than or equal to the first threshold, it periodically triggers a traversal of the multiple write buffers in the buffer storage area to obtain the traversal result.

[0130] In another exemplary scheme, the server can monitor the occupancy of the available storage area in the buffer storage area in real time. When the available storage area in the buffer storage area is less than a preset threshold, data compression can be accelerated to free up more available space. When the available storage area in the buffer storage area is greater than the preset threshold, data compression can be slowed down to reduce the resources occupied by asynchronous threads.

[0131] 304. Based on the write queue and the second index, write the first data to the buffer storage area and update the first index table to the second index table, the second index table including the first index and the second index.

[0132] In this embodiment, when the server determines to add the first data to the write queue and allocate the second index, the first data is written to the buffer storage area, and the first index table is updated to the second index table.

[0133] It should be understood that when the server writes the first data to the buffer storage area, it can use the method described above for writing the first data to the memory storage area. Please refer to the above description for details, which will not be repeated here.

[0134] It should also be understood that the first index table and the second index table can be different states of the same hash table, or they can be different hash tables; the specifics are not limited here. For example, the first state of the hash table (i.e., the first index table) can be as shown in Table 1.

[0135] Table 1

[0136] key Location Timestamp Key1 Memory_Block_1 2025-03-26 10:00:00 Key2 Memory_Block_3 2025-03-26 10:01:00 Key3 Memory_Block_4 2025-03-26 10:02:00

[0137] After the data is written to the buffer storage area, the second state of the hash table (i.e., the second index table) can be as shown in Table 2.

[0138] Table 2

[0139]

[0140] 305. Allocate a third index for the first data, which is the index used when the first data is written to the hard disk storage area.

[0141] In this embodiment, after the server writes the first data into the buffer storage area, it can call the I / O interface to write the first data already stored in the buffer storage area into the hard disk storage area. During this process, the server needs to first allocate a corresponding index to the first data, which is then used to indicate the storage location of the first data in the hard disk storage area.

[0142] In this embodiment, the hard disk storage area can be a solid-state drive or a mechanical hard disk; there is no specific limitation here.

[0143] 306. Write the first data to the hard disk storage area based on the third index, and update the second index table to the third index table, wherein the third index table includes the first index, the second index and the third index.

[0144] In this embodiment, the server writes the first data to the hard disk storage area based on the third index. The specific writing method can be referred to the above method of writing the first data to the memory storage area, which will not be repeated here.

[0145] It should be understood that after the data is successfully written, the server also needs to update the second index table to the third index table. In an exemplary scenario, the third index table can be represented as shown in Table 3.

[0146] Table 3

[0147]

[0148] Based on the above description, in an exemplary solution, the memory design within the server can be as follows: Figure 4 As shown. The server's memory is divided into memory storage areas (i.e. Figure 4 Shared memory in the middle), buffer storage area (i.e. Figure 4 The diagram shows write buffer 1 and write buffer 2, as well as the hard disk storage area. This memory storage area stores the data in the hash table through byte regions, and the hash table itself is also stored in this memory storage area. For example... Figure 4 As shown, "key1, value1" is hot data and can therefore be directly stored in this memory storage area. "key2, value2" is in the process of being pushed down to the hard disk storage area and is currently being written to this buffer storage area. "keyn, valuen" is cold data and has already been stored in the hard disk storage area. Meanwhile, when the data in the hash table changes (e.g., ...), ... Figure 4As shown in “key4, value4”, the server needs to perform a write-time comparison. If the comparison is successful, the old value is added to the delayed deletion queue and the new value is written.

[0149] Based on the above stored procedure, the following description uses the server as the execution entity to illustrate the process of reading the first data. Please refer to [link / reference needed]. Figure 5 One embodiment of data reading in this application includes:

[0150] 501. Obtain a read request that indicates the reading of the first data.

[0151] The server receives a read request indicating the need to read the first data through a transceiver interface. This read request includes the key value of the first data (e.g., primary key, filename), the logical address of the data (e.g., offset or block number), and the size and range to be read (e.g., 4KB or a specific field). The server parses the content of the read request and then obtains the key value of the first data (e.g., primary key, filename), the logical address of the data (e.g., offset or block number), and the size and range to be read (e.g., 4KB or a specific field).

[0152] 502. Based on the read request, obtain the third index table of the first data, the third index table including the first index, the second index and the third index.

[0153] After the server parses the key value of the first data, it obtains the final third index table corresponding to the first data based on the key value, and retrieves the first index, the second index, and the third index from the third index table.

[0154] 503. Based on the first index, perform a query in the memory storage area to obtain the first query result.

[0155] During the reading process, the server first performs a query in the memory storage area based on the first index to obtain the first query result of the first data.

[0156] In this embodiment, after the server queries the storage location corresponding to the first index, it reads the data at the storage location and verifies the data. If the verification passes, it is determined that the data at the storage location indicated by the first query result is the first data; if the verification fails, it is determined that the first query result indicates that the first data does not exist in the memory storage area.

[0157] 504. When the first query result indicates that the first data exists in the memory storage area, the first data is read from the memory storage area.

[0158] In this embodiment, when the first query result indicates that the first data exists in the memory storage area, the first data can be directly read from the memory storage area and returned.

[0159] 505. When the first query result indicates that the first data does not exist in the memory storage area, a query is performed in the buffer storage area based on the second index to obtain the second query result.

[0160] In this embodiment, when the first query result indicates that the first data does not exist in the memory storage area, the server performs a query in the buffer storage area based on the second index to obtain the second query result.

[0161] In this embodiment, after the server queries the storage location corresponding to the second index, it reads the data at the storage location and verifies the data. If the verification passes, it is determined that the data at the storage location indicated by the second query result is the first data; if the verification fails, it is determined that the first data does not exist in the buffer storage area.

[0162] 506. When the second query result indicates that the first data exists in the buffer storage area, the first data is read from the buffer storage area.

[0163] In this embodiment, when the second query result indicates that the first data exists in the buffer storage area, the first data can be read directly from the buffer storage area and returned.

[0164] 507. When the second query result indicates that the first data does not exist in the buffer storage area, a query is performed in the hard disk storage area based on the third index, and the first data is read from the hard disk storage area.

[0165] In this embodiment, when the second query result indicates that the first data does not exist in the buffer storage area, the server performs a query in the hard disk storage area based on the third index to obtain the third query result.

[0166] In this embodiment, after the server queries the storage location corresponding to the third index, it reads the data at the storage location and verifies the data. If the verification passes, it determines that the data at the storage location indicated by the third query result is the first data; if the verification fails, it determines that the first data does not exist in the hard disk storage area, and returns the result that the first data does not exist and the reading failed.

[0167] In combination with the above Figure 3 and Figure 5 The scheme shown below will be used as an example. Figure 6 The flowchart shown illustrates the data writing and data reading processes in this application.

[0168] like Figure 6 As shown, the data writing process can be as follows:

[0169] After routing to index table 2, the server calculates the corresponding index based on the key value of the written data. Then, based on this index, it writes the data to the memory storage area. Simultaneously, it performs cache eviction management in the memory storage area using an asynchronous thread. When the written data meets the asynchronous eviction policy, it writes the data to the buffer storage area (i.e., the Write Pipeline). Then, based on the asynchronous thread, it compresses the data in the buffer storage area according to the compression policy, thereby reorganizing the written data and adding it to the write queue of the disk storage area. Finally, it calls the I / O interface to write the data to the disk storage area based on the write queue, and finally persists the data.

[0170] like Figure 6 As shown, the data reading process can be as follows:

[0171] Based on the read request, the server routes to the corresponding index table 1. First, it queries the memory storage area according to the first index; if the memory storage area is matched, the data is returned; if not, it queries the buffer storage area according to the second index; if the buffer storage area is matched, the data is returned; if not, it queries the disk storage area according to the third index; if the disk storage area is matched, the data is returned; if not, a read failure result is returned.

[0172] Based on the above solution, this application can also provide a data snapshot storage solution; for details, please refer to [link / reference needed]. Figure 7 As shown.

[0173] 701. Obtain a snapshot creation request, which instructs the creation of a target snapshot that includes the first data.

[0174] In practical applications, to ensure data recoverability, the server can trigger the snapshot creation request periodically; it can also trigger the snapshot creation request according to progress; it can also trigger the snapshot creation request when the model performance index reaches a threshold; or it can be triggered by a third party. In this embodiment, the snapshot creation request may include the first data.

[0175] 702. In response to the snapshot creation request, acquire a global write lock, generate a snapshot identifier for the target snapshot and initialize snapshot metadata, which includes snapshot creation time, snapshot version information and snapshot storage area identifier.

[0176] After receiving the snapshot creation request, the server can acquire a global write lock in response to the request to ensure the consistency of the snapshot data. Simultaneously, the server acquires the snapshot identifier and initializes the snapshot metadata for the target snapshot. The snapshot identifier can be the directory structure of the target snapshot. The initialized snapshot metadata can include the snapshot creation time, snapshot version information (e.g., the parameter range for snapshot storage), and the snapshot storage region identifier (i.e., the location where the snapshot is stored).

[0177] 703. Based on the snapshot identifier and the initial snapshot metadata, dump the hot data of the memory storage area and the cached data of the buffer storage area, and dump the cold data of the hard disk storage area through hard links to generate an intermediate snapshot.

[0178] The server dumps the data in the memory storage area and the buffer storage area based on the snapshot identifier and the initial snapshot metadata; at the same time, it dumps the cold data in the hard disk storage area based on hard links; and generates a snapshot file as an intermediate snapshot at the current time point, and stores the intermediate snapshot.

[0179] In this embodiment, hard links are used for the data in the hard disk storage area to avoid repeatedly writing the same cold data. The specific process is as follows: For cold data already existing in the hard disk storage area (potentially belonging to an old snapshot), hard links are created directly. That is, the new hard links point to the same physical file, rather than copying the data. If the data does not change in subsequent snapshots, the newly created snapshot file points to the old file content through hard links. For example, in the first snapshot, files A, B, and C exist on the SSD. When snapshot 1 is created, hard links pointing to these files are generated: Snapshot 1: A->File A; B->File B; C->File C. In the second snapshot (with some data updates): File B is updated, and file D contains new data. When snapshot 2 is created: For the unchanged files A and C, hard links are created directly: Snapshot 2: A->File A (reused); C->File C (reused); For the updated file B and the newly added file D, new files are created: B->File B' (new version); D->File D (new file). Therefore, the final stored snapshot file can be as follows:

[0180] Snapshot 1: A->File A; B->File B; C->File C.

[0181] Snapshot 2: A->File A; B->File B'; C->File C; D->File D.

[0182] It should be understood that, to ensure the stability of snapshot files, if the stored data in the disk storage area changes during the generation of two snapshots (i.e., incremental data is generated), the incremental data can be written to a newly allocated storage area, and the corresponding index can be updated simultaneously. For example, in snapshot 1: A -> file A; B -> file B; C -> file C, and before snapshot 2 is generated, file B changes and generates incremental data, file E; then, file E will not be directly written to the storage area B of file B. Instead, a new storage area E will be allocated for file E, and file E will be written to that storage area E. At the same time, the index of file B will be updated, changing the storage area B indicated by its index to storage area E.

[0183] 704. Based on the first index table, the second index table, and the third index table, perform deduplication on the intermediate snapshot to obtain the target snapshot, and release the global write lock.

[0184] In this embodiment, the server performs deduplication on the intermediate snapshot based on the first index table, the second index table, and the third index table to obtain the target snapshot and release the global write lock.

[0185] This means the server uses index fields (such as timestamps or version numbers) to deduplicate data in the snapshot. Hard links are used to retain only the references to the latest version of the data, avoiding duplicate storage.

[0186] Based on the above description, the process by which the server generates this snapshot can be as follows: Figure 8 As shown: First, the hot data in the memory storage area and the data in the buffer storage area are dumped, and the dumped data is stored in the snapshot buffer storage area; then, the data in the hard disk storage area is dumped through hard links; then, the dumped data in the snapshot buffer storage area is used to generate a dump value log; the intermediate snapshot is generated based on the value logs 1 to n of the hard links and the dump value log; the intermediate snapshot is scanned from newest to oldest based on the index field to detect duplicates and delete duplicates to obtain the target snapshot.

[0187] To maintain data consistency and real-time performance during snapshot generation while avoiding blocking normal write operations, this embodiment also provides a snapshot replica space technical solution. The specific solution is as follows: Obtain the start position, end position, and space size of the target snapshot based on the snapshot storage area identifier; create the snapshot replica space of the target snapshot based on the space size and the end position, wherein the end position serves as the start position of the snapshot replica space, and the elements stored in the snapshot replica space are used to indicate the element positions of the target snapshot; redirect the allocation of the start position based on the end position of the snapshot replica space; after the global write lock is released, write the backlogged data accumulated during the global write lock period based on the allocated start position.

[0188] To achieve efficient reuse of memory resources while maintaining system state, this embodiment also provides a memory reclamation scheme. The specific scheme is as follows: during the dumping process of the target snapshot, the end position of memory garbage collection is determined based on the start position of the target snapshot; memory garbage collection is then performed based on this end position.

[0189] Meanwhile, in order to avoid memory waste and improve resource utilization efficiency, this embodiment also provides a circular buffer design for the memory storage area. The specific scheme is as follows: when the allocation head exceeds the maximum memory address of the memory storage area, the allocation head is modulo-processed with the maximum memory address to reset the allocation head to the minimum memory address, wherein the allocation head is the storage tail of the backlog data.

[0190] Based on the above scheme, an exemplary scheme is that the snapshot function memory design can be as follows: Figure 9 As shown. A single thread performs garbage collection from low to high memory addresses. However, the end position of memory reclamation must not exceed the start position of the snapshot before the data dump in the snapshot area is complete. If the allocation header exceeds the maximum memory address, the allocation header is modulo the maximum memory address, and the allocation header is reset, reallocating memory space starting from the minimum memory address. Simultaneously, a snapshot copy is created with the end position of the snapshot as the start position and the size of the snapshot as the size of the snapshot's space; the elements of this fast copy are used to indicate the element positions of the snapshot; the allocation start position is redirected based on the end position of the snapshot copy space; after the global write lock is released, the backlogged data during the global write lock period is written based on the allocation start position, thus completing the incremental data writing.

[0191] The data processing apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the data processing apparatus in this application. The data processing apparatus 20 includes:

[0192] The acquisition module 201 is used to acquire first data and a first index table, the first index table including a first index, which is the index of the first data when it is written to the memory storage area;

[0193] Processing module 202 is configured to write the first data to the memory storage area based on the first index; when the first data meets the asynchronous eviction policy, allocate a first identifier and a second index to the first data, and add the first data to the write queue of the cache storage area, wherein the first identifier is used to indicate that the first data will be written to the cache storage area, and the second index is the index when the first data is written to the cache storage area; based on the write queue and the second index, write the first data to the cache storage area, and update the first index table to the second index table, wherein the second index table includes the first index and the second index; allocate a third index to the first data, wherein the third index is the index when the first data is written to the hard disk storage area; write the first data to the hard disk storage area based on the third index, and update the second index table to the third index table, wherein the third index table includes the first index, the second index, and the third index.

[0194] This application provides a data processing apparatus. Using this apparatus, a three-tiered storage architecture—memory storage area, buffer storage area, and hard disk storage area—is designed for the same data. During storage, an asynchronous eviction strategy is used to enable data to flow between different storage layers. This allows the system to manage data exceeding the memory capacity, thereby optimizing memory utilization and reducing memory storage costs.

[0195] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0196] The memory storage area is managed based on the least recently used mechanism, the least frequent use mechanism, the adaptive replacement cache, or the clock algorithm.

[0197] This memory storage area is shared memory.

[0198] This application provides a data processing apparatus. Using this apparatus, multiple cache eviction mechanisms are employed to manage data in the memory storage area. This allows for the automatic selection or combination of optimal strategies based on different workload characteristics, thereby optimizing memory utilization. Simultaneously, the memory storage area is divided into shared memory, which maximizes data integrity protection and supports rapid recovery in abnormal situations such as container restarts.

[0199] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, when the memory storage area is managed based on the least recently used mechanism, the processing module 202 is used to check the recent access time of the first data and the memory occupancy information of the memory storage area; when the memory occupancy information is greater than a first threshold and the recent access time of the first data meets a first condition, it is determined that the first data conforms to the asynchronous eviction policy, wherein the first condition is that in the sorting based on the recent access time from nearest to farthest, the recent access time of the first data is ranked in the last N positions, where N is a positive integer.

[0200] This application provides a data processing apparatus. Using this apparatus, multiple cache eviction mechanisms are employed to manage data in the memory storage area. This allows for the automatic selection or combination of optimal strategies based on different workload characteristics, thereby optimizing memory utilization.

[0201] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0202] The processing module 202 is used to write the first data into the memory storage area based on the first index and the write-time comparison mechanism.

[0203] This application provides a data processing apparatus. Using this apparatus, an index structure is established and precise data comparison is performed before write operations, achieving intelligent selective updates, significantly improving system performance, optimizing resource utilization, and enhancing data consistency assurance.

[0204] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0205] When the first data is the updated data of the target parameter, the processing module 202 is used to perform a write-time comparison on the first data to obtain the comparison result;

[0206] When the comparison result indicates a successful comparison, the second data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The second data is the data of the target parameter that has been written to the memory storage area.

[0207] When the comparison result indicates a failure, the update time of the first data is compared with that of the third data of the target parameter, where the third data is the data of the target parameter that has been written into the memory storage area.

[0208] When the update time of the first data is later than the update time of the third data, the first data is written to the memory storage area based on the first index;

[0209] If the update time of the first data is earlier than the update time of the third data, the writing of the first data is abandoned.

[0210] In this application embodiment, a data processing apparatus is provided. Using this apparatus, a write-time comparison mechanism is employed to identify and filter write requests that do not represent substantial changes, writing only the substantially changed data, thereby reducing write traffic.

[0211] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, after performing a write-time comparison on the first data to obtain a comparison result, the processing module 202 is used to retry the write-time comparison of the first data when the comparison result indicates a comparison failure and the first memory region indicated by the first index has been written with the fourth data, until the fourth data is written, and then allocate a second memory region for the first data. The second memory region is a memory region that is available after the first memory region, and the fourth data is different data with different keywords but the same index as the first data.

[0212] The first data is written to the second memory region based on the first index.

[0213] This application provides a data processing apparatus. Using this apparatus, a write-time comparison mechanism is employed to identify and filter write requests that do not represent substantial changes, writing only the substantially changed data, thereby reducing write traffic. Furthermore, in cases where different data share the same index, causing write conflicts, the write operation is retried, thus ensuring data storage consistency.

[0214] Optionally, in the above Figure 10 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application embodiment, before assigning a first identifier and a second index to the first data and adding the first data to the write queue of the cache storage area, the processing module 202 is used to traverse multiple write buffers of the cache storage area to obtain traversal results. The traversal results are used to indicate whether there is duplicate data of the first data or expired data of the target parameter corresponding to the first data in the cache storage area.

[0215] When the traversal result indicates the existence of duplicate data of the first data or expired data of the target parameter corresponding to the first data, delete the duplicate data of the first data or the expired data of the target parameter corresponding to the first data.

[0216] In this embodiment of the application, a data processing apparatus is provided. Using this apparatus, during the data writing process to the buffer storage area, the data is compressed, thereby significantly saving storage space and reducing the number of write operations to the hard disk storage area, thus effectively reducing hard disk wear.

[0217] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0218] The processing module 202 is used to traverse the number of valid keys and the number of invalid keys that have stored data in multiple write buffers of the buffer storage area;

[0219] When the ratio of the number of valid keys to the number of invalid keys is greater than a first threshold, multiple write buffers of the buffer storage area are traversed to obtain the traversal result.

[0220] When the ratio of the number of valid keys to the number of invalid keys is less than or equal to the first threshold, multiple write buffers of the buffer storage area are periodically triggered to obtain the traversal result.

[0221] In this embodiment of the application, a data processing apparatus is provided. Using this apparatus, the compression strategy is adjusted based on the number of valid keys and the number of invalid keys, thereby adapting to changes in parameter distribution and effectively controlling the memory data eviction process.

[0222] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0223] After writing the first data to the hard disk storage area based on the third index and updating the second index table to the third index table, the acquisition module 201 is used to acquire a read request, which indicates reading the first data;

[0224] The processing module 202 is used to obtain the third index table of the first data based on the read request. The third index table includes the first index, the second index, and the third index.

[0225] Based on the first index, a query is performed in the memory storage area to obtain the first query result;

[0226] When the first query result indicates that the first data exists in the memory storage area, the first data is read from the memory storage area;

[0227] When the first query result indicates that the first data does not exist in the memory storage area, a query is performed in the buffer storage area based on the second index to obtain the second query result;

[0228] When the second query result indicates that the first data exists in the buffer storage area, the first data is read from the buffer storage area;

[0229] When the second query result indicates that the first data does not exist in the buffer storage area, a query is performed in the hard disk storage area based on the third index to read the first data from the hard disk storage area.

[0230] This application provides a data processing apparatus. Using this apparatus, a three-tiered storage architecture—memory storage area, buffer storage area, and hard disk storage area—is designed for the same data. During storage, an asynchronous eviction strategy is used to enable data to flow between different storage layers. This allows the system to manage data exceeding the memory capacity, thereby optimizing memory utilization and reducing memory storage costs.

[0231] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0232] After querying the hard disk storage area based on the third index and reading the first data from the hard disk storage area, the processing module 202 is used to write the first data into the memory storage area based on the first index and the write comparison mechanism.

[0233] This application provides a data processing apparatus. Using this apparatus, data is processed by "floating" and "sinking" based on its access frequency, thereby enabling data to flow and be stored between different storage layers. This allows the system to manage data exceeding its memory capacity, thus optimizing memory utilization and reducing memory storage costs.

[0234] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0235] The processing module 202 is used to perform a write-time comparison on the first data to obtain a second comparison result;

[0236] When the second comparison result indicates a successful comparison, the fifth data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The fifth data is the data of the target parameter that has been written to the memory storage area.

[0237] If the second comparison result indicates a failure, the writing of the first data is abandoned.

[0238] This application provides a data processing apparatus. Using this apparatus, during the floating write process, only data that has substantially changed or is in the latest updated state is written. This effectively reduces data write operations, achieves intelligent selective updates, significantly improves system performance, optimizes resource utilization, and enhances data consistency assurance.

[0239] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0240] The acquisition module 201 is used to acquire a snapshot creation request, which is used to instruct the creation of a target snapshot including the first data;

[0241] The processing module 202 is used to acquire a global write lock in response to the snapshot creation request, generate a snapshot identifier for the target snapshot, and initialize snapshot metadata, which includes snapshot creation time, snapshot version information, and snapshot storage area identifier.

[0242] Based on the snapshot identifier and the initial snapshot metadata, the hot data of the memory storage area and the cached data of the buffer storage area are dumped, and the cold data of the hard disk storage area is dumped through hard links to generate an intermediate snapshot;

[0243] Based on the first index table, the second index table, and the third index table, the intermediate snapshot is deduplicated to obtain the target snapshot, and the global write lock is released.

[0244] This application provides a data processing apparatus. Using this apparatus, data in the memory storage area and buffer storage area is dumped using a fixed snapshot identifier and initial snapshot metadata, while data in the hard disk storage area is dumped using hard links. This way, snapshots only store the necessary metadata and indexes of the hard disk storage area, sharing the original hard disk data, thus greatly saving storage space. Simultaneously, snapshot data is deduplicated based on the index, which significantly reduces memory consumption and system load.

[0245] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, the processing module 202 is used to obtain the start position, end position and space size of the target snapshot based on the snapshot storage area identifier;

[0246] The snapshot copy space of the target snapshot is created based on the space size and the end position, wherein the end position serves as the start position of the snapshot copy space, and the elements stored in the snapshot copy space are used to indicate the element positions of the target snapshot.

[0247] The starting position is reassigned based on the end position of the snapshot copy space;

[0248] After the global write lock is released, the backlogged data accumulated during the global write lock period is written based on the allocation start position.

[0249] In this application embodiment, a data processing apparatus is provided. Using this apparatus, a snapshot copy space is created, and the memory allocation pointer is redirected to the end of the snapshot copy space. Incremental data is then written based on the allocation start position, thereby maintaining snapshot consistency with the original region data. In this way, the copy space only stores incremental data after the snapshot generation time, avoiding duplicate storage and saving memory.

[0250] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, the processing module 202 is used to determine the end position of memory garbage collection based on the start position of the target snapshot during the dumping process of the target snapshot; and to perform memory garbage collection based on the end position.

[0251] This application provides a data processing apparatus. Using this apparatus, while storing data, invalid data, fragmented space, and expired versions are identified and reclaimed, thereby achieving efficient reuse of memory resources while maintaining system state.

[0252] Optionally, in the above Figure 10 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to perform modulo processing on the allocation head with the maximum memory address when the allocation head exceeds the maximum memory address of the memory storage area, so as to reset the allocation head to the minimum memory address, wherein the allocation head is the storage tail of the backlog data.

[0253] In this application embodiment, a data processing apparatus is provided. Using this apparatus, when the allocation head exceeds the maximum memory address, a modulo operation is performed on the maximum memory address to achieve cyclical use of memory. This design avoids memory waste and improves resource utilization efficiency.

[0254] The data processing apparatus provided in this application can be used on a server; please refer to [link / reference]. Figure 11 , Figure 11This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0255] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM MacOSX TM Unix TM Linux TM FreeBSD TM etc.

[0256] The steps performed by the server in the above embodiments can be based on this Figure 11 The server structure shown.

[0257] The data processing apparatus provided in this application can be used in terminal devices; please refer to [link / reference]. Figure 12 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0258] Figure 12 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 12 The smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that... Figure 12The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0259] The following is combined with Figure 12 A detailed introduction to the various components of a smartphone:

[0260] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0261] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0262] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 480, and can also receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0263] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include a display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 12 In this embodiment, the touch panel 431 and the display panel 441 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 431 and the display panel 441 can be integrated to realize the input and output functions of the smartphone.

[0264] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0265] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0266] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (470), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 12 WiFi module 470 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0267] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0268] The smartphone also includes a power supply 490 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0269] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0270] The steps performed by the terminal device in the above embodiments can be based on this Figure 12 The terminal device structure is shown.

[0271] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0272] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0273] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0274] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0275] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0277] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0278] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain first data and a first index table, wherein the first index table includes a first index, which is the index of the first data when it is written to the memory storage area; The first data is written to the memory storage area based on the first index; When the first data meets the asynchronous eviction policy, a first identifier and a second index are assigned to the first data, and the first data is added to the write queue of the cache storage area. The first identifier is used to indicate that the first data will be written to the cache storage area, and the second index is the index when the first data is written to the cache storage area. Based on the write queue and the second index, the first data is written to the buffer storage area, and the first index table is updated to the second index table, the second index table including the first index and the second index; A third index is allocated to the first data, and the third index is the index used when the first data is written to the hard disk storage area; The first data is written to the hard disk storage area based on the third index, and the second index table is updated to the third index table, wherein the third index table includes the first index, the second index, and the third index.

2. The method according to claim 1, characterized in that, The memory storage area is managed based on a least recently used mechanism, a least-used number of times mechanism, an adaptive replacement cache, or a clock algorithm. The memory storage area is shared memory.

3. The method according to claim 2, characterized in that, When the memory storage area is managed based on the least recently used mechanism, before allocating a first identifier and a second index to the first data and adding the first data to the write queue of the cache storage area, the method further includes: Check the most recent access time of the first data and the memory usage information of the memory storage area; When the memory usage information is greater than a first threshold and the most recent access time of the first data meets a first condition, it is determined that the first data conforms to the asynchronous eviction policy, wherein the first condition is that in the sorting based on the most recent access time from the nearest to the furthest, the most recent access time of the first data is ranked in the last N positions, where N is a positive integer.

4. The method according to claim 1, characterized in that, The step of writing the first data to the memory storage area based on the first index includes: The first data is written to the memory storage area based on the first index and the write-time comparison mechanism.

5. The method according to claim 4, characterized in that, When the first data is updated data for the target parameter, writing the first data into the memory storage area based on the first index and the write-time comparison mechanism includes: Perform a write-time comparison on the first data to obtain the comparison result; When the comparison result indicates a successful comparison, the second data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The second data is the data of the target parameter that has been written to the memory storage area. When the comparison result indicates a comparison failure, the update time of the first data is compared with that of the third data of the target parameter, wherein the third data is the data of the target parameter that has been written into the memory storage area. When the update time of the first data is later than the update time of the third data, the first data is written into the memory storage area based on the first index; If the update time of the first data is earlier than the update time of the third data, the writing of the first data is abandoned.

6. The method according to claim 4, characterized in that, After performing a write-time comparison on the first data to obtain a comparison result, the method further includes: When the comparison result indicates that the first memory region indicated by the first index has been written with the fourth data, the write comparison of the first data is retried until the fourth data is written. Then, a second memory region is allocated for the first data. The second memory region is the memory region that is available after the first memory region. The fourth data is different data with different keys but the same index as the first data. The first data is written to the second memory region based on the first index, and the first index is updated.

7. The method according to any one of claims 1 to 6, characterized in that, Before assigning a first identifier and a second index to the first data and adding the first data to the write queue of the cache storage area, the method further includes: The multiple write buffers of the buffer storage area are traversed to obtain the traversal result, which is used to indicate whether there is duplicate data of the first data or expired data of the target parameter corresponding to the first data in the buffer storage area. When the traversal result indicates the existence of duplicate data of the first data or expired data of the target parameter corresponding to the first data, delete the duplicate data of the first data or the expired data of the target parameter corresponding to the first data.

8. The method according to claim 7, characterized in that, Before traversing multiple write buffers of the buffer storage area to obtain the traversal result, the method further includes: Iterate through the number of valid keys and invalid keys that have been stored in the multiple write buffers of the buffer storage area; When the ratio of the number of valid keys to the number of invalid keys is greater than a first threshold, multiple write buffers of the buffer storage area are traversed to obtain the traversal result. When the ratio of the number of valid keys to the number of invalid keys is less than or equal to the first threshold, multiple write buffers of the buffer storage area are periodically triggered to obtain the traversal result.

9. The method according to any one of claims 1 to 6 and 8, characterized in that, After writing the first data to the hard disk storage area based on the third index and updating the second index table to the third index table, the method further includes: Obtain a read request, the read request indicating that the first data should be read; The third index table of the first data is obtained based on the read request, and the third index table includes the first index, the second index and the third index; A query is performed in the memory storage area based on the first index to obtain a first query result; When the first query result indicates that the first data exists in the memory storage area, the first data is read from the memory storage area; When the first query result indicates that the first data does not exist in the memory storage area, a query is performed in the buffer storage area based on the second index to obtain the second query result; When the second query result indicates that the first data exists in the buffer storage area, the first data is read from the buffer storage area; When the second query result indicates that the first data does not exist in the buffer storage area, a query is performed in the hard disk storage area based on the third index to read the first data from the hard disk storage area.

10. The method according to claim 9, characterized in that, After querying the hard disk storage area based on the third index and reading the first data from the hard disk storage area, the method further includes: The first data is written to the memory storage area based on the first index and the write comparison mechanism.

11. The method according to claim 9, characterized in that, The step of writing the first data into the memory storage area based on the first index and the write comparison mechanism includes: Perform a write-time comparison on the first data to obtain a second comparison result; When the second comparison result indicates a successful comparison, the fifth data of the target parameter is added to the delayed deletion queue, and the first data is written to the memory storage area based on the first index. The fifth data is the data of the target parameter that has been written to the memory storage area. If the second comparison result indicates that the comparison failed, the writing of the first data is abandoned.

12. The method according to any one of claims 1 to 6 and 8, characterized in that, The method further includes: Obtain a snapshot creation request, the snapshot creation request being used to instruct the creation of a target snapshot including the first data; In response to the snapshot creation request, a global write lock is acquired, a snapshot identifier for the target snapshot is generated, and snapshot metadata is initialized. The snapshot metadata includes snapshot creation time, snapshot version information, and snapshot storage area identifier. Based on the snapshot identifier and the initial snapshot metadata, the hot data of the memory storage area and the cached data of the buffer storage area are dumped, and the cold data of the hard disk storage area is dumped through hard links to generate an intermediate snapshot; The intermediate snapshot is deduplicated based on the first index table, the second index table, and the third index table to obtain the target snapshot, and the global write lock is released.

13. The method according to claim 12, characterized in that, The method further includes: The start position, end position, and size of the target snapshot are obtained based on the snapshot storage area identifier. The snapshot copy space of the target snapshot is created based on the space size and the end position, wherein the end position serves as the start position of the snapshot copy space, and the elements stored in the snapshot copy space are used to indicate the element positions of the target snapshot; The starting position is reassigned based on the end position of the snapshot copy space; After the global write lock is released, the backlogged data accumulated during the global write lock period is written based on the allocation start position.

14. The method according to claim 13, characterized in that, The method further includes: During the dumping process of the target snapshot, the end position of memory garbage collection is determined based on the start position of the target snapshot; Memory garbage collection is performed based on the stated end position.

15. The method according to claim 13, characterized in that, The method further includes: When the allocation header exceeds the maximum memory address of the memory storage area, the allocation header is modulo-processed with the maximum memory address to reset the allocation header to the minimum memory address, wherein the allocation header is the storage tail of the backlog data.

16. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire first data and a first index table, wherein the first index table includes a first index, which is the index of the first data when it is written to the memory storage area. The processing module is configured to write the first data into the memory storage area based on the first index; When the first data conforms to the asynchronous eviction policy, a first identifier and a second index are allocated to the first data, and the first data is added to the write queue of the cache storage area. The first identifier is used to indicate that the first data will be written to the cache storage area, and the second index is the index when the first data is written to the cache storage area. Based on the write queue and the second index, the first data is written to the cache storage area, and the first index table is updated to the second index table, which includes the first index and the second index. A third index is allocated to the first data, which is the index when the first data is written to the hard disk storage area. Based on the third index, the first data is written to the hard disk storage area, and the second index table is updated to the third index table, which includes the first index, the second index, and the third index.

17. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 15 according to instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

18. A computer-readable storage medium comprising instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the method as described in any one of claims 1 to 15.