Data processing method and device, equipment, storage medium and program product
By adopting a storage-compute separation system architecture, the challenges of resource expansion and stability in storage-compute integrated architectures are solved, achieving reasonable separation and efficient access of computing and storage resources, and improving the system performance and scalability of big data applications.
Patent Information
- Application Number
- CN202511253030.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-03
- Publication Date
- 2025-12-16
AI Technical Summary
Existing in-memory computing architectures face challenges in terms of resource scalability and system stability. In particular, they are prone to forming fault domains in large-scale clusters, and lack resource isolation and have a high risk of fault propagation, making it difficult to support large-scale, highly stable big data application scenarios.
By adopting a storage-compute separation system architecture, separate computing and storage modules are set up, data access is carried out using predefined access paths, and metadata management by the management module is combined to achieve reasonable allocation and efficient location of resources, optimize access strategies, reduce latency, and improve response speed and system scalability.
It achieves a reasonable separation of computing and storage resources, improves data access efficiency and system responsiveness, enhances system scalability and management flexibility, reduces latency, and improves overall system performance and stability.
Smart Images
Figure CN121144352A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of database, and in particular to a data processing method and device, equipment, storage medium and program product. BACKGROUND
[0002] The big data technology system is originally derived from the scene of batch processing of massive data, and includes management nodes and computing nodes in the technical architecture. Due to the limitation of network technology in the early stage of technical development, in order to achieve better performance, the computing nodes will select the machine type with high computing power + local disk, and in the allocation algorithm of the computing task, the processing will be completed as much as possible in the computing node where the data is stored, so as to reduce the transmission amount of data on the network. This architecture of local computing realized by local disk is called storage-computing integrated architecture.
[0003] However, with the continuous expansion of business volume and cluster size, the existing storage-computing integrated architecture faces significant challenges in resource expansion and system stability. On the one hand, the storage-computing integrated architecture has the problem of limited subsequent cluster expansion; on the other hand, the single large-scale cluster under the storage-computing integrated architecture is easy to form a fault domain, which may affect the stability of the entire business system when a fault occurs, lacks resource isolation for different businesses, and has high risk of resource contention and fault diffusion. In summary, it is urgent to optimize the expansion capability, fault isolation mechanism and resource elasticity management of the storage-computing integrated architecture to support large-scale and high-stability big data application scenarios. SUMMARY
[0004] The embodiments of the present application provide a data processing method, device, equipment, storage medium and program product, which can support system elastic expansion of computing or storage resources and dynamically adjust resource configuration according to real-time load.
[0005] In a first aspect, the embodiments of the present application provide a data processing method applied to a computing module in a storage-computing separated system, wherein the storage-computing separated system further includes a storage module, and the computing module and the storage module are separately arranged.
[0006] The method comprises:
[0007] receiving an access instruction of a target object;
[0008] based on the access instruction, reading target data corresponding to the access instruction from the storage module through an access path of the target object;
[0009] responding to the access instruction by using the target data, and storing the response result into the storage module.
[0010] In an implementation, the storage-computing separation system further comprises a management module, and metadata of at least one object is registered in the management module;
[0011] The target data corresponding to the access instruction is read from the storage module through the access path of the target object based on the access instruction.
[0012] The access permission corresponding to the target object is determined based on the access instruction and the metadata stored in the management module.
[0013] When the access permission corresponding to the target object indicates that the target object can access the storage module, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0014] In an implementation, after the access permission corresponding to the target object is determined, the method further comprises:
[0015] When the access permission corresponding to the target object indicates that the target object cannot access the storage module, an alarm prompt is generated.
[0016] In an implementation, the target data corresponding to the access instruction is read from the storage module through the access path of the target object, and the method further comprises:
[0017] It is judged whether the target data corresponding to the access instruction exists in the cache space; if yes, the target data corresponding to the access instruction is read from the cache space.
[0018] The target data corresponding to the access instruction is read from the storage module through the access path of the target object, and the method further comprises:
[0019] When the target data corresponding to the access instruction does not exist in the cache space, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0020] In an implementation, after the response result is stored in the storage module, the method further comprises:
[0021] Based on the target data corresponding to the access instruction, the data stored in the cache space is updated based on the least recently used principle.
[0022] In an implementation, before the access instruction of the target object is received, the method further comprises:
[0023] In response to a migration instruction, data in a target system is migrated to the storage-computing separation system; the target system is a storage-computing integrated system.
[0024] In a second aspect, an embodiment of the present application provides a data processing apparatus, applied to a computing module in a memory-compute separation system, the memory-compute separation system further comprising a storage module, the computing module and the storage module being separately arranged;
[0025] The data processing apparatus comprises:
[0026] a receiving module, configured to receive an access instruction of a target object;
[0027] a reading module, configured to read target data corresponding to the access instruction from the storage module through an access path of the target object based on the access instruction;
[0028] a responding module, configured to respond to the access instruction by using the target data, and store a response result into the storage module.
[0029] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor.
[0030] The memory stores computer execution instructions.
[0031] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.
[0032] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, the computer execution instructions being executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.
[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, the computer program being executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.
[0034] The data processing method, apparatus, device, storage medium and program product provided by the embodiments of the present application adopt the architecture of a memory-compute separation system, and by separately arranging a storage module and a computing module, the rational separation of resources can be achieved, and the close coupling of computing resources and storage resources can be avoided. In addition, a target object accesses target data in the storage module through a specific path, which helps to achieve efficient positioning and management of data access. The data processing method uses a pre-defined access path, which not only reduces the complexity of data retrieval, but also optimizes the access strategy, reduces the delay, and improves the data response speed, thereby enhancing the response capability and throughput of the system. The data processing method effectively combines the resource optimization and efficient access mechanism of the memory-compute separation architecture, and significantly improves the response speed, expansion capability and management flexibility of the system. Attached Figure Description
[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0036] Figure 1 This is a schematic diagram of the storage-computing separation system provided in this application;
[0037] Figure 2 A flowchart illustrating the data processing method provided in this application;
[0038] Figure 3 A schematic diagram of the data processing apparatus provided in this application;
[0039] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.
[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0042] The information and data involved in this application are information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.
[0043] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 The storage-compute separation system 100 shown is described. The storage-compute separation system 100 includes a computing module 102, a storage module 104, and a management module 106, with the computing module and storage module being set up separately.
[0044] In the compute-storage separation system 100, the compute nodes and the storage nodes can adopt a plurality of different distribution manners. As an example, the plurality of compute modules 102 and the plurality of storage modules 104 can be respectively deployed in different servers or clusters; or alternatively, the plurality of storage modules 104 can be centrally deployed, while the plurality of compute modules 102 can be distributed in servers or clusters and connected through network access; or alternatively, part of the compute modules 102 and the storage modules 104 are deployed in the same server, and the remaining compute modules 102 and the storage modules 104 are deployed in different servers. The deployment manner of the compute modules 102 and the storage modules 104 is not limited in the present application. The above-mentioned servers can be implemented by independent servers or server clusters composed of a plurality of servers.
[0045] The storage module 104 is responsible for managing data and does not undertake computing tasks; the compute module 102 is used to obtain data from the storage module 104 for processing. This manner is beneficial to elastic expansion, improves resource utilization, and enhances the maintainability of the system.
[0046] As an example, the compute module 102 can be a MapReduce service (MRS) that can provide a computing engine service, a cache service, and a scheduling service. The cache service is implemented through the cache space inside the MRS, and the cache space inside the MRS provides a global cache solution for the compute module 102, thereby reducing the performance degradation caused by the separation of the compute module 102 and the storage module 104. Different compute modules 102 have independent cache spaces.
[0047] The storage module 104 is used to store data and is also a public service for a plurality of MRS clusters, and the storage module 104 can be flexibly and independently expanded. The storage module 104 can adopt an object storage service (OBS) distributed storage mode, which is implemented through an erasure code (EC) function, which is a technology for improving data reliability and storage efficiency. The storage module 104 can divide the stored data into a plurality of segments through the erasure code and add redundant check information, so that the original data can be recovered in the case of partial data loss or damage, thereby guaranteeing the data security.
[0048] The management module 106 is used to manage table metadata, and the table metadata includes table structure, permission, and other information, and is a public service module that can share metadata for a plurality of compute modules 102. The management module 106 can be an AWS LakeFormation cloud service, and the metadata storage of the Lake Formation supports horizontal expansion, and adopts a distributed Gaussian database to store metadata.
[0049] For example, the data processing method is applied to the computing module 102. After receiving the access instruction of the target object, the computing module 102 can read the target data corresponding to the access instruction from the storage module 104 based on the access instruction through the access path of the target object. Finally, the target data is used to respond to the access instruction, and the response result is stored in the storage module 104.
[0050] In one embodiment, a data processing method is provided. The present embodiment takes the data processing method applied to a computing module as an example. As shown in the figure, the data processing method comprises the following steps. Figure 2
[0051] Step 202, receiving an access instruction of a target object.
[0052] In a storage-computing separation system, the access instruction received by the computing node is usually a command for requesting data processing or operation, which has various specific forms and purposes, mainly including the following: a data reading instruction, which is used to request reading specific target data from the storage module, usually including data location (path or index), data range (offset, length) and the like, is applied to transferring data to the computing node for processing, and supports batch, random or continuous data access. This is the most common instruction form, which mainly realizes data acquisition; a data writing instruction, which carries target storage location, data content or data identifier, and writing mode (overwriting or appending, etc.) and the like, and is used to store the result processed by the computing node to the storage module; a data processing request, which contains the location information of the data to be processed, the required processing logic (such as SQL statement, MapReduce task, machine learning model inference request, etc.), and is used to instruct the storage-computing separation system to perform specific computing operation, which may contain algorithm, parameter and the like, to realize “computing in storage” or “storage-side computing”; a data deletion or modification instruction, which is used to specify the target data location to be deleted or modified, to perform data management and maintenance operation; a metadata request instruction, which is used to request metadata about data, such as size, storage state, version and the like, to assist in judging data state or optimizing storage and computing strategy. These access instructions play a bridge role between the storage module and the computing module in the storage-computing separation system, to ensure that the computing node can efficiently and flexibly access and operate the data in the storage module, thereby supporting high-performance and large-scale complex data processing tasks.
[0053] Step 204, reading the target data corresponding to the access instruction from the storage module based on the access instruction through the access path of the target object.
[0054] The computing module first receives an access instruction from an upper-layer application or a user. The access instruction usually contains the unique identification of the target object, the access type (read, write, update, etc.), and the relevant access path information.
[0055] The computing module further parses the access instruction to confirm the specific target and the required data range of the access instruction.
[0056] According to the parsing result, the computing module extracts the access path of the target object from the access instruction. The access path may be the storage location identifier of the data in the storage module (such as file path, offset address, block ID, etc.), which is used to accurately locate the target data position in the storage module. Among them, the formulation of the access path can be an absolute path, a logical path, or an indexed path after mapping, ensuring that each target object can be uniquely located.
[0057] The computing module initiates a request through the built-in or predefined interface, and sends a read command to the storage module with the access path and access type information. This read command usually includes the storage location of the target data, the range of read data (such as offset, length) or conditions (such as filtering conditions), to clearly indicate the specific data that the storage module needs to read.
[0058] The storage module retrieves data according to the received request, which is specifically manifested as that the storage module reads the corresponding target data from the storage medium according to the storage structure and the access path. The reading process may involve data decoding, decompression, and verification steps to ensure the correctness and integrity of the data.
[0059] Finally, the storage module returns the target data to the computing module that issued the read command in the specified format and protocol after completing the reading of the target data.
[0060] Step 206, responding to the access instruction with the target data and storing the response result in the storage module.
[0061] After receiving the target data from the storage module, the computing module further executes the corresponding computing or processing task according to the specific requirements of the access instruction using the target data. After processing, the computing node generates a response result. These results may be partial or complete processing output, or changes to the data. Finally, the computing node stores the processed response result in the storage module. This step includes: generating a new data object or updating an existing object, specifying the storage location (path, ID), adding necessary metadata (such as timestamp, version number), using a write operation (such as file write, object storage write operation) to store the response result in the storage medium, and ensuring the integrity and consistency of the write operation (such as confirming the success of the write operation, handling error retry mechanism).
[0062] The aforementioned data processing method adopts a storage-compute separation system architecture. By setting up separate storage and computing modules, it can achieve reasonable resource allocation and avoid tight coupling between computing and storage resources. Secondly, the target object accesses the target data in the storage module through a specific path, which helps to achieve efficient data access location and management. This data processing method utilizes predefined access paths, which not only reduces the complexity of data retrieval but also optimizes access strategies, reduces latency, and improves data response speed, thereby enhancing the system's responsiveness and throughput. This data processing method effectively combines the resource optimization and efficient access mechanism of the storage-compute separation architecture, significantly improving the system's response speed, scalability, and management flexibility.
[0063] In some optional embodiments, the storage-compute separation system further includes a management module that registers metadata for at least one object;
[0064] Step 204 includes:
[0065] Based on the access instructions and the metadata stored in the management module, determine the access permissions corresponding to the target object;
[0066] When the access permissions corresponding to the target object indicate that the target object can access the storage module, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0067] The metadata stored in the management module typically includes the unique identifier of the corresponding object, the object's location, the user or permission group to which the object belongs, access permission settings (such as read / write permissions, access control lists (ACLs), role permissions), and other metadata information.
[0068] After receiving an access instruction, the computing module can first retrieve metadata entries related to the target object from a table formed by the metadata of at least one object stored in the management module, in order to determine the access permissions corresponding to the target object.
[0069] The calculation module can compare the access type (such as "read") contained in the access command with the permission information of the target object in the metadata entries matched in the management module. The comparison may include: verifying whether the target object's identity (user ID, role, permission group, etc.) has the corresponding access permissions in the metadata; and comparing whether the target object's access type is permitted (e.g., whether it has read permissions). If the target object's permission status allows the access, the corresponding access permissions indicate that the target object can access the storage module; otherwise, the corresponding access permissions indicate that the target object cannot access the storage module.
[0070] In some optional embodiments, when the access permission corresponding to the target object indicates that the target object cannot access the storage module, the computing module can generate an alarm to refuse the computing module from retrieving data from the storage module based on the access command initiated by the target object.
[0071] The above data processing method ensures that in a storage-compute separation system, data access is not only based on the request itself, but also on the permission information in the management module to strictly control the access permissions of different objects, thereby enhancing the security and controllability of the storage-compute separation system. At the same time, by verifying permissions before reading data, it is ensured that only authorized objects can access the target data, thus improving the security level of the storage-compute separation system.
[0072] In some optional embodiments, prior to step 204, the method further includes:
[0073] Determine if the target data corresponding to the access instruction exists in the cache space; if so, read the target data corresponding to the access instruction from the cache space.
[0074] Step 204 includes:
[0075] When the target data corresponding to the access instruction does not exist in the cache space, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0076] Before the computing module reads the target data from the storage module, the computing node can first check whether a copy of the target data already exists in the local cache space. For example, it can use the access path or unique identifier of the target object (such as object ID or hash value) as an index to perform a fast search in the local cache space.
[0077] If the target data exists in the local cache space, the computing module can directly read the target data from the cache space, thereby saving the time spent accessing the storage module and retrieving data from the storage module.
[0078] If the target data is not present in the local cache, the computing module will then read the target data from the storage module through the access path of the target object.
[0079] It should be noted that, in one embodiment, after the computing module reads the target data from the storage module, or after responding to an access command with the target data and storing the response result in the storage module, the read target data can be stored in the cache space to update the data stored in the cache space. As an example, the computing module can update the data stored in the cache space using the least recently used principle.
[0080] The aforementioned data processing method reduces the frequency of access to the storage module through a caching mechanism, thereby extending the lifespan of the corresponding hardware device and reducing maintenance costs. Furthermore, by adopting the least recently used principle, it retains frequently accessed data in the cache space, thereby improving the hit rate of the data stored in the cache space for the target data corresponding to the access command. This allows for more efficient use of limited high-performance cache resources, reduces system latency, and improves overall throughput.
[0081] In some optional embodiments, prior to step 202, the following steps are also included:
[0082] In response to the migration command, the data in the target system is migrated to the storage-compute separation system; the target system is a storage-compute integrated system.
[0083] Migration commands can be issued by users through terminals that communicate with the storage-compute separation system. The process of migrating data from the target system to the storage-compute separation system can be performed by the computing module, the management module, or a migration tool that communicates with the storage-compute separation system. The following example illustrates the migration of data in a storage-compute integrated system using a migration tool.
[0084] Based on the received migration request, the migration tool clarifies the migration scope (including permission data, metadata, and storage data) and the migration target location (i.e., the specific location in the storage-compute separation system).
[0085] Furthermore, the migration tools are launched to prepare for the permission verification, path conversion mechanisms, and resource scheduling required during the migration process.
[0086] During the data migration process, the migration tool can migrate the permission configurations such as tenant information, users, user groups and role permissions in the storage-compute integrated system to the management module of the storage-compute separated system. This ensures that the permission management system remains consistent after the migration, data access permissions are not affected, and permission confusion or leakage is avoided after the migration.
[0087] During the migration process of Hive metadata (such as databases, tables, partitions, data structure definitions, etc.) from an integrated storage and compute system to the management module of a separate storage and compute system, the migration tool can change the storage path of the metadata from the original HDFS path to the target storage path to ensure consistency in path management. The metadata is stored in a Gaussian database managed by the management module, ensuring the integrity and consistency of the metadata.
[0088] Throughout the data migration process, data is copied step by step according to the migration plan to ensure data integrity and accuracy. Data structure can be adjusted or optimized during the migration process to ensure data adaptability to the new environment. It also supports breakpoint resume to ensure uninterrupted and complete migration.
[0089] The above data processing method can automatically migrate data in the storage and computing system throughout the entire process, thereby greatly shortening the migration time. During the migration, it can ensure that the permission management remains consistent after the migration, avoiding security risks caused by missing or misconfigured permission information.
[0090] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0091] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this data processing apparatus is similar to the implementation scheme described in the data processing method above. Therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the data processing method above, and will not be repeated here.
[0092] In one embodiment, such as Figure 3 As shown, a data processing device 300 is provided, applied to the computing module 102 in a storage-computing separation system 100. The storage-computing separation system 100 also includes a storage module 104. The computing module 102 and the storage module 104 are separately configured. The data processing device 300 includes:
[0093] Receiver module 302 is used to receive access instructions for the target object;
[0094] The reading module 304 is used to read the target data corresponding to the access instruction from the storage module based on the access instruction and the access path of the target object;
[0095] The response module 306 is used to respond to the access command with the target data and store the response result in the storage module.
[0096] In some optional embodiments, the storage-compute separation system further includes a management module that registers metadata for at least one object;
[0097] Read module 304 is also configured as follows:
[0098] Based on the access instructions and the metadata stored in the management module, determine the access permissions corresponding to the target object;
[0099] When the access permissions corresponding to the target object indicate that the target object can access the storage module, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0100] In some optional embodiments, the reading module 304 is further configured to:
[0101] An alarm is generated when the access permissions corresponding to the target object indicate that the target object cannot access the storage module.
[0102] In some optional embodiments, the reading module 304 is further configured to:
[0103] Determine if the target data corresponding to the access instruction exists in the cache space; if so, read the target data corresponding to the access instruction from the cache space.
[0104] When the target data corresponding to the access instruction does not exist in the cache space, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
[0105] In some optional embodiments, the reading module 304 is further configured to:
[0106] Based on the target data corresponding to the access instruction, the data stored in the cache space is updated using the least recently used principle.
[0107] In some optional embodiments, the receiving module 302 is further configured to:
[0108] In response to the migration command, data is migrated from the target system to the in-memory computing system; the target system is an in-memory computing system.
[0109] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0110] The electronic device provided in this embodiment can execute the item search method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0111] Figure 4 A schematic diagram of the structure of the computer device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0112] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.
[0113] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0114] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0115] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0116] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0117] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described item search method.
[0118] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described item search method.
[0119] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0120] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0121] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0123] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0124] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0125] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0126] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A data processing method, characterized in that, A computing module is used in a storage-compute separation system, wherein the storage-compute separation system also includes a storage module, and the computing module and the storage module are set up separately; The method includes: Receive access instructions for the target object; Based on the access instruction, the target data corresponding to the access instruction is read from the storage module through the access path of the target object; The target data is used to respond to the access command, and the response result is stored in the storage module.
2. The method according to claim 1, characterized in that, The storage-compute separation system also includes a management module, which registers metadata for at least one object. The step of reading the target data corresponding to the access instruction from the storage module based on the access instruction and through the access path of the target object includes: Based on the access command and the metadata stored in the management module, the access permissions corresponding to the target object are determined; When the access permission corresponding to the target object indicates that the target object can access the storage module, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
3. The method according to claim 2, characterized in that, After determining the access permissions corresponding to the target object, the process further includes: An alarm is generated when the access permission corresponding to the target object indicates that the target object cannot access the storage module.
4. The method according to claim 1, characterized in that, Before reading the target data corresponding to the access instruction from the storage module through the access path of the target object, the method further includes: Determine whether the target data corresponding to the access instruction exists in the cache space; if so, read the target data corresponding to the access instruction from the cache space. The step of reading the target data corresponding to the access instruction from the storage module through the access path of the target object includes: When the target data corresponding to the access instruction does not exist in the cache space, the target data corresponding to the access instruction is read from the storage module through the access path of the target object.
5. The method according to claim 4, characterized in that, After storing the response result in the storage module, the method further includes: Based on the target data corresponding to the access instruction, the data stored in the cache space is updated using the least recently used principle.
6. The method according to claim 1, characterized in that, Before receiving the access instruction for the target object, the method further includes: In response to the migration command, data in the target system is migrated to the storage-compute separation system; the target system is a storage-compute integrated system.
7. A data processing apparatus, characterized in that, A computing module is used in a storage-compute separation system, wherein the storage-compute separation system also includes a storage module, and the computing module and the storage module are set up separately; The data processing device includes: The receiving module is used to receive access instructions for the target object; The reading module is used to read the target data corresponding to the access instruction from the storage module based on the access instruction and through the access path of the target object; The response module is used to respond to the access command with the target data and store the response result in the storage module.
8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.