Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a processing request initiated for target processing data, obtaining a first data type and a processing type of the target processing data; under the condition that the first data type is cold data, calling a uniform interface service to access a first remote file system to which the target processing data belongs according to first attribute information of the target processing data, and obtaining the target processing data; storing the target processing data to a local storage area; and obtaining the target processing data from the local storage area according to the processing type for data processing. By adopting the method, the computing power and time cost consumed in the data acquisition process can be reduced.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In scenarios where business analysis is required, the data analysis process is often performed using analytical databases, which periodically import data from external storage media and then process the data.

[0003] However, with the development of the data era, the quantity of data has grown exponentially, and the external data sources that need to be analyzed have become more diverse. For example, it is necessary to obtain the corresponding data from remote file systems provided by different vendors. The access methods used by the storage products provided by different vendors are not the same, which brings great complexity to the data migration and import work and increases the computing power and time costs consumed in the data acquisition process. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can reduce the computing power and time costs consumed in the data acquisition process, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a data processing method. The method includes:

[0006] In response to a processing request initiated for target processing data, the first data type and processing type of the target processing data are obtained;

[0007] In the case where the first data type is cold data, the unified interface service is invoked to access the first remote file system to which the target data belongs, based on the first attribute information of the target data, and to obtain the target data.

[0008] The target processing data is stored in the local storage area;

[0009] According to the processing type, the target processing data is obtained from the local storage area and processed.

[0010] In one embodiment, the step of calling the unified interface service to access the first remote file system to which the target processing data belongs, based on the first data attribute information of the target processing data, and to obtain the target processing data, includes:

[0011] The unified interface service is invoked to extract the storage medium identifier and connection parameters of the target data from the first data attribute information of the target data;

[0012] The first remote file system to which the target processed data belongs is determined based on the storage medium identifier;

[0013] An access connection channel with the first remote file system is created based on the connection parameters;

[0014] The first remote file system is accessed using the access connection channel to obtain the target processing data.

[0015] In one embodiment, storing the target processing data in a local storage area includes:

[0016] The sharding capacity corresponding to the target processing data is determined based on the data capacity of the target processing data and the processing type.

[0017] The target data to be processed is divided into data fragments based on the fragment capacity to obtain each data fragment.

[0018] Each of the data fragments is stored in the local storage area in fragment order.

[0019] In one embodiment, the step of calling the unified interface service to access the first remote file system to which the target processing data belongs, based on the first attribute information of the target processing data, and to obtain the target processing data, includes:

[0020] The unified interface service is invoked to generate a data acquisition task for the target data based on the first attribute information, and the data acquisition task is written into the task pool of the unified interface service.

[0021] If the data acquisition task meets the task execution conditions, a processing thread is allocated to the data acquisition task.

[0022] The processing thread uses the first attribute information to access the first remote file system to which the target processing data belongs, and obtains the target processing data.

[0023] In one embodiment, the method further includes:

[0024] If it is determined that the target processing data has associated data based on preset association conditions, the second data type of the associated data is obtained;

[0025] If the second data type is cold data, then the unified interface service is invoked to create an asynchronous processing thread for the associated data;

[0026] The asynchronous processing thread uses the second attribute information of the associated data to access the second remote file system to obtain the associated data.

[0027] The associated data is stored in the local storage area.

[0028] In one embodiment, the method further includes:

[0029] In response to a remote storage command for target cold data, the target cold data and its data information are obtained;

[0030] Based on the security requirement level of the target cold data as represented by the data information, a target storage system matching the security requirement level is determined from each of the remote file systems;

[0031] The unified interface service is invoked to access the target storage system based on the third attribute information of the target cold data, and the target cold data is stored in the target storage system.

[0032] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0033] The request-response module is used to respond to a processing request initiated for target processing data and obtain the first data type and processing type of the target processing data;

[0034] The unified interface access module is used to, when the first data type is cold data, call the unified interface service to access the remote file system to which the target processing data belongs based on the first attribute information of the target processing data, and obtain the target processing data.

[0035] A local storage module is used to store the target processing data in a local storage area;

[0036] The data processing module is used to obtain the target processing data from the local storage area according to the processing type and perform data processing.

[0037] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0038] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0039] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0040] The aforementioned data processing methods, apparatus, computer equipment, storage media, and computer program products all incorporate a pre-configured unified interface service in their database executors. When the target data to be processed is determined to be cold data (i.e., the target data is stored in a remote file system), the unified interface service can be directly invoked to access the first remote file system to which the target data belongs based on its first attribute information. The target data is then retrieved and stored in the local storage area. Subsequently, the executor can retrieve the target data from the local storage area for processing according to the processing type. By automatically connecting to the remote file system based on the first attribute information through the unified interface service, retrieving the target data, and storing it in the local storage area, the database executor only needs to retrieve the corresponding target data from the local storage area when performing data processing operations. This achieves loose coupling between data processing and storage. The database's kernel computing layer can access various remote storage media using a unified interface, reducing the computational and time costs consumed in the data acquisition process and improving data acquisition efficiency. Attached Figure Description

[0041] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0042] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0043] Figure 3 This is a schematic diagram of the process in one embodiment of calling a unified interface service to access the first remote file system to which the target data belongs and obtain the target data based on the first data attribute information of the target data.

[0044] Figure 4 This is a schematic diagram illustrating the process of storing the target processing data to a local storage area in one embodiment;

[0045] Figure 5 This is a flowchart illustrating the process of calling a unified interface service to access the first remote file system to which the target processing data belongs, based on the first data attribute information of the target processing data, and obtaining the target processing data in another embodiment.

[0046] Figure 6 This is a flowchart illustrating the data processing method in another embodiment;

[0047] Figure 7 This is a flowchart illustrating the data processing method in another embodiment;

[0048] Figure 8 This is a schematic diagram of the database structure in one embodiment;

[0049] Figure 9 This is a flowchart illustrating the data processing method in another embodiment;

[0050] Figure 10 This is a schematic diagram illustrating the interaction between the database and various file systems in one embodiment;

[0051] Figure 11 This is a structural block diagram of a data processing device in one embodiment;

[0052] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, database 100 includes a database executor 102 and local storage 104. Database executor 102 can communicate with user terminal 106 and is equipped with a unified interface service, allowing it to connect to various remote file systems 108. Local storage 104 stores the data that database 100 needs to process; local storage 104 can be implemented using a local disk.

[0055] In actual use, users can initiate processing requests for target data through user terminal 106. Database executor 102 responds to the processing request by obtaining the first data type and processing type of the target data. If the first data type is cold data, it calls the unified interface service to access the first remote file system to which the target data belongs based on the first attribute information of the target data, obtains the target data, stores the target data in the local storage area of ​​local storage 104, and then retrieves the target data from the local storage area according to the processing type for data processing.

[0056] Database 100 can be a database that needs to retrieve data from remote file system 108, such as a transactional database or an analytical database. Database executor 102 is a key module in the database kernel responsible for performing specific data processing operations, specifically enabling database 100 to perform tasks such as reading, writing, updating, querying, and deleting data.

[0057] The remote file system 108 is a distributed storage architecture that provides file storage and access services over a network. By storing data in the remote file system 108, loose coupling between computing and storage can be achieved.

[0058] In one embodiment, the remote file system 108 may include an object storage system (OSS), which is a cloud storage service in cloud services that can store all data as objects on the cloud platform.

[0059] In one embodiment, the remote file system 108 may include a distributed file system, such as the Hadoop Distributed File System (HDFS), which can provide the database 100 with a reliable file system for large data volumes that runs on low-cost hardware.

[0060] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the database executor 102 as an example, the following steps are included:

[0061] S202, in response to a processing request initiated for the target processing data, obtain the first data type and processing type of the target processing data.

[0062] The target processing data is a collection of data that requires data processing. Understandably, the target processing data can be stored in the local storage of the database or in a remote file system.

[0063] Among them, a processing request initiated for the target data is an operation instruction used to instruct the execution of data processing on the target data to obtain the corresponding data processing result. It can be understood that a processing request can be triggered by a user based on the target data via a user terminal. For example, a user can set the first attribute information, data type, and processing type of the target data through the user terminal, and then click on the data processing space to initiate a processing request to the database for the target data.

[0064] The first data type is a preset classification parameter that classifies the target data based on the data access characteristics and usage frequency, and can include cold data and hot data.

[0065] Cold data refers to static datasets that are rarely accessed or are read-only. These are typically used for long-term archiving, historical analysis, or compliance purposes and are not sensitive to access speed. Examples of cold data include user login logs, complete historical audit records from annual audits, and design documents and source code archived after project completion. Because cold data is accessed infrequently, it is usually stored on a remote file service to save local database storage capacity.

[0066] Hot data refers to frequently accessed or modified active data in a database, requiring extremely high access speed and low latency. For example, in e-commerce websites, hot data can include product detail pages and user shopping carts. In gaming scenarios, hot data can include players' real-time status and location.

[0067] Understandably, data types are not fixed for data; that is, the data types of different data can be changed according to the actual access characteristics and usage frequency of the data.

[0068] The processing type is a classification parameter used to categorize the specific operations performed on the data. For example, data types may include, but are not limited to, data query, data transformation, data analysis, data cleaning, data aggregation, and machine learning.

[0069] Specifically, the database executor can respond to a processing request initiated for the target data to be processed, parse the processing request, and obtain the first data type and processing type of the target data to be processed.

[0070] S204, if the first data type is cold data, call the unified interface service to access the first remote file system to which the target data belongs based on the first attribute information of the target data to obtain the target data.

[0071] Among them, the unified interface service is a data access abstraction layer pre-set in the database. It can shield the technical differences of the underlying heterogeneous storage system through a standardized operation interface, and provide a unified file access interface for the database executor so as to achieve a consistent data access experience when the database executor accesses different underlying storage media.

[0072] In one embodiment, when the database executor calls the unified interface service, it can be considered as building a unified file system at the database's computation layer. For example, this unified file system can be called xblock. The xblock unified file system can connect to other remote file systems through its own abstract interface. When the database kernel accesses different underlying storage media, it doesn't need to concern itself with the specific implementation details of the underlying layer, but can directly access the xblock abstract interface. The xblock abstract interface is the unified file access interface provided by the database's computation layer, which can implement basic functions such as open, write, read, seek, and close. The different underlying storage media can include local files, remote file systems such as HDFS and OSS provided by different vendors.

[0073] The first attribute information is a set of metadata describing the data storage location, access method, and connection configuration of the target data to be processed. It can include all parameters required to establish a data access channel with the first remote file system. Understandably, the first attribute information can be generated by the user when generating the table structure corresponding to the target data. For example, the user can determine the target data to be processed based on their own data processing needs, generate the corresponding first attribute information based on the actual storage location of the target data and the relevant authentication information at that storage location, and configure the first attribute information in the metadata information of the table corresponding to the target data. The database executor only needs to access the metadata of the table containing the target data to obtain this data information and then use this data information to call the unified interface service to access the remote file system.

[0074] In one embodiment, the first attribute information may include storage location information, authentication credential information, and connection configuration information. The storage location information may include, but is not limited to, media type and Uniform Resource Identifier (URI). The authentication credential information may include, but is not limited to, user identifier and password. The connection configuration information may include, but is not limited to, endpoint information, region information, and connection duration threshold.

[0075] In one embodiment, when the database computing layer represented by the database executor accesses the unified file system, it only needs to add an option when opening the file to access all data as if it were a local file. The information in the option is stored in the first attribute information of the corresponding target data being processed, i.e., the metadata information of the table. When the metadata of the table is opened for access, this data information can be obtained to access the file.

[0076] Among them, the first remote file system is the remote file system that actually stores the target processing data in each remote file system.

[0077] Specifically, if the database executor determines that the first data type of the target data to be processed is cold data, it can determine that the target data to be processed is stored in a remote file system. The database executor can call a pre-set unified interface service to access the first remote file system to which the target data to be processed belongs, based on the first attribute information of the target data to be processed, and obtain the target data to be processed.

[0078] In one embodiment, the database executor can call a pre-set unified interface service to abstract and generate a unified file system at the computing layer. Then, by simply opening the unified file system and inputting the relevant connection parameters for remote connection with the first remote file system according to the first attribute information, the target processing data can be obtained from the first remote file system through the unified interface provided by the unified file system.

[0079] In one embodiment, if the database executor determines that the first data type of the target processing data is hot data, it can assume that the target processing data is in local storage. The database executor can directly obtain the target processing data from local storage through the basic interface and process the target processing data according to the processing type.

[0080] S206, store the target processing data in the local storage area.

[0081] The local storage area is the physical storage area provided by the database for local storage. During data processing, the database may need to read portions of the target data sequentially for processing until all the target data has been read. Since each remote file system can be considered a remote storage medium isolated from the database computation, and file updates on each remote file system must go through the database, the database executor can directly store the target data in the local storage area without worrying about the risk of the target data stored in the local storage area becoming invalid due to data updates in the remote file systems.

[0082] Specifically, in order to reduce the cost and risk of file transfer failures with remote file systems, the unified interface service can directly retrieve all target processing data from the first remote file and return it to the database executor. The database executor can then store the target processing data in the local storage area, instead of retrieving portions of the target processing data from the first remote file system sequentially to perform data processing until all target processing data has been retrieved.

[0083] S208: Retrieve target processing data from the local storage area according to the processing type and perform data processing.

[0084] Specifically, the database executor can sequentially retrieve the target processing data from the local storage area according to the processing type, perform data processing, and obtain the data processing result.

[0085] In the aforementioned data processing method, the database executor is pre-configured with a unified interface service. When the target data to be processed is determined to be cold data (i.e., the target data is stored in a remote file system), the unified interface service can be directly invoked to access the first remote file system to which the target data belongs based on its first attribute information. The target data is then obtained and stored in the local storage area. Subsequently, the executor can retrieve the target data from the local storage area for processing according to the processing type. By automatically connecting to the remote file system based on the first attribute information through the unified interface service, retrieving the target data, and storing it in the local storage area, the database executor only needs to retrieve the corresponding target data from the local storage area when performing data processing operations. This achieves loose coupling between data processing and storage. The database kernel computing layer can access various remote storage media using a unified interface, reducing the computational and time costs consumed in the data acquisition process and improving data acquisition efficiency.

[0086] In one embodiment, such as Figure 3 As shown, in S204, the unified interface service accesses the first remote file system to which the target processing data belongs, based on the first data attribute information of the target processing data, to obtain the target processing data, including:

[0087] S302, call the unified interface service to extract the storage medium identifier and connection parameters of the target processing data from the first data attribute information of the target processing data.

[0088] The storage medium identifier is a unique identifier used to identify a remote file system. It can be represented by specific characters or the system name of the remote file system. For example, if the storage medium identifier is OSS, the corresponding remote file system is an object storage system.

[0089] The connection parameters are a set of configuration parameters required to establish a connection between the unified file system created by the unified interface service and the first remote file system storing the target processing data. Through these connection parameters, the unified interface service can build an access connection channel with the corresponding remote file system to retrieve data from it.

[0090] In one embodiment, connection parameters may include information such as URI, user, and password. The database executor only needs to add connection parameters when the file is opened to access all data in the remote file system as if it were a local disk.

[0091] Specifically, the database executor can call the unified interface service to perform data extraction operations on the first data attribute information, and extract the storage medium identifier and connection parameters of the target data to be processed.

[0092] S304, determine the first remote file system to which the target data to be processed belongs based on the storage medium identifier.

[0093] Specifically, the unified interface service can automatically determine the first remote file system to which the target data to be processed belongs, based on the storage medium identifier.

[0094] In one embodiment, the unified interface service can obtain the mapping relationship between each storage medium identifier and each remote file system from the local storage, and determine the first remote file system to which the target data to be processed belongs by looking up the mapping relationship based on the storage medium identifier.

[0095] S306, Create an access connection channel with the first remote file system based on the connection parameters.

[0096] The access connection channel is the communication link between the unified file system abstracted by the unified interface service and the first remote file system. Through the access connection channel, relevant data stored in the first remote file system can be obtained.

[0097] Specifically, the unified interface service can create an access connection channel with the first remote file system based on the connection parameters.

[0098] In one embodiment, the unified interface service can create an access connection channel to a first remote file system based on connection parameters through the unified interface provided by the abstract unified file system.

[0099] S308 uses the access connection channel to access the first remote file system and obtain the target processing data.

[0100] Specifically, the unified interface service can use the access connection channel to access the first remote file system and obtain the target processing data from the first remote file system.

[0101] In the above embodiments, the database executor can accurately and quickly construct an access connection channel with the first remote file system by calling the unified interface service and based on the storage medium identifier and connection parameters extracted from the first attribute information. This effectively improves the real-time performance of obtaining target processing data from the first remote file system and reduces the time and computing power costs consumed in the data acquisition process.

[0102] In one embodiment, such as Figure 4 As shown, S206, storing the target processing data in the local storage area includes:

[0103] S402, determine the fragment capacity corresponding to the target data to be processed based on the data capacity and processing type of the target data to be processed.

[0104] Among them, the data capacity of the target data being processed is an indicator parameter used to measure the size of the target data being processed, usually measured in units such as bytes, kilobytes, and megabytes.

[0105] The partition capacity of the target processing data refers to the size of each part when the target processing data is divided into multiple parts. Understandably, the size of the allocated capacity for the target processing data will affect the parallel speed and efficiency of data processing. Different processing types require different data processing methods, which may influence the data allocation strategy.

[0106] Specifically, after obtaining the target processing data, the database executor can first determine the data capacity of the target processing data, thereby determining the size of the target processing data, and then determine the corresponding shard capacity of the target processing data based on the data capacity and processing type.

[0107] In one embodiment, the database executor can determine the first theoretical shard capacity corresponding to the data capacity and the second theoretical shard capacity corresponding to the processing type, respectively. Then, based on the capacity calculation weight corresponding to the data capacity and the capacity calculation weight corresponding to the processing type, the first theoretical shard capacity and the second theoretical shard capacity are weighted and averaged to obtain the shard capacity corresponding to the target processing data.

[0108] S404: The target data is divided into fragments based on the fragment capacity to obtain each data fragment.

[0109] Among them, fragmentation processing is a process of dividing the target data into multiple data fragments based on fragment capacity as the core basis and constraint.

[0110] Specifically, the database executor can divide the target data into shards based on the shard capacity to obtain each data shard.

[0111] In one embodiment, the database executor can perform sharding planning on the target processing data under the constraint of sharding capacity, obtain sharding planning information, and then perform sharding action on the target processing data based on the sharding planning information to cut the target processing data into candidate data shards. The integrity of each candidate data allocation is verified, and if the verification is successful, each candidate data shard is determined as the data shard of the target processing data.

[0112] In one embodiment, if the verification fails, the fragmentation planning parameters can be adjusted, and the process can return to the step of fragmenting the target data under the constraint of fragmentation capacity to obtain fragmentation planning information, until the integrity verification is successful.

[0113] S406 stores each data fragment in the local storage area according to the fragment order.

[0114] The fragmentation order refers to the logical sequence of each data fragment in the original target processing data.

[0115] Specifically, after obtaining the data shards, the database executor can store each data shard in the local storage area according to the sharding order. By storing the data shards in the sharding order, data integrity, data content dependencies, and processing dependencies can be improved.

[0116] In the above embodiments, by caching the target processing data in segments, the network transmission cost and resource consumption of the disk can be reduced when accessing local storage subsequently, thereby improving data processing efficiency. At the same time, it can provide basic data units for subsequent distributed parallel data processing.

[0117] In one embodiment, such as Figure 5 As shown, in S204, the unified interface service accesses the first remote file system to which the target processing data belongs, based on the first attribute information of the target processing data, to obtain the target processing data, including:

[0118] S502, the unified interface service is invoked to generate a data acquisition task for the target data based on the first attribute information, and the data acquisition task is written into the task pool of the unified interface service.

[0119] The data acquisition task is a complete data loading execution unit generated for the target processing data, describing how to obtain the target processing data from the first remote file system. By executing the data acquisition task, the corresponding target processing data can be obtained from the first remote file system.

[0120] The task pool of the unified interface service is a queue system that manages tasks waiting to be executed within the unified interface service. The task pool can include task queues, task status, schedulers, etc. It can realize functions such as priority scheduling, load balancing, fault tolerance, resource control, and status monitoring to meet the data access requirements under high concurrency.

[0121] Specifically, the database executor can call the unified interface service to generate a data acquisition task for the target data based on the first attribute information, and write the data acquisition task into the task pool of the unified interface service.

[0122] S504: If the data acquisition task meets the task execution conditions, allocate a processing thread to the data acquisition task.

[0123] Among them, the task execution condition is a preset judgment condition used to determine whether the data acquisition task can be executed. The data acquisition task is stored in the task queue of the task pool. Only when the data acquisition task meets the task execution condition will the unified interface service create a processing thread for the data acquisition task.

[0124] In one embodiment, task execution conditions may include the task priority reaching the processing requirement, or the task scheduling time reaching the preset scheduling time, or the system resources meeting the task execution requirements, etc.

[0125] Among them, the processing thread is a thread in the unified interface service used to perform data acquisition tasks. The operations it needs to perform include establishing a connection with the remote file system, reading data, processing data, and storing data in the local storage area.

[0126] Specifically, when the data acquisition task meets the task execution conditions, the unified interface service will allocate a processing thread for the data acquisition task.

[0127] In one embodiment, if the data acquisition task does not meet the task execution conditions, it will continue to wait for execution in the task pool.

[0128] S506, the processing thread accesses the first remote file system to which the target processing data belongs based on the first attribute information, and obtains the target processing data.

[0129] Specifically, the unified interface service uses a processing thread to access the first remote file system to which the target data belongs, based on the first attribute information, and then retrieves the target data. Understandably, the specific steps by which the processing thread accesses the first remote file system to which the target data belongs based on the first attribute information have been explained in detail above and will not be repeated here.

[0130] In one embodiment, such as Figure 6As shown, the data processing method may also include the following steps:

[0131] S602, if it is determined that there is associated data in the target processing data based on preset association conditions, the second data type of the associated data is obtained.

[0132] Among them, the preset association conditions are preset judgment conditions used to determine whether there is related data in the target data. Related data are other data that have a predefined relationship with the target data. These can be set by designers in advance based on the possible relationships between data. For example, preset association conditions may include, but are not limited to, primary and foreign key association conditions, business logic association conditions, time series association conditions, spatial location association conditions, semantic similarity association conditions, and generation dependency association conditions.

[0133] The second data type is a preset classification parameter that classifies related data based on the data access characteristics and usage frequency. It can also include cold data and hot data.

[0134] Specifically, during the process of acquiring target processing data, the database executor can determine whether there is related data that satisfies the preset association conditions, based on the data information of the target processing data and preset association conditions. If related data that satisfies the preset association conditions exists, the database executor can acquire the second data type of the related data.

[0135] In one embodiment, the database executor can determine the second data type of the associated data by finding a preset correspondence between each data identifier and each data type based on the data identifier of the associated data.

[0136] S604 If the second data type is cold data, then the unified interface service is called to create an asynchronous processing thread for the associated data.

[0137] Asynchronous processing threads are execution threads that can run independently in the background without blocking the thread that retrieves the target data. By creating asynchronous processing threads for related data, related data can be retrieved independently without affecting the process of retrieving the target data.

[0138] Specifically, if the database executor determines that the second data type of the associated data is cold data, it can call the unified interface service to create an asynchronous processing thread for the associated data.

[0139] S606 uses an asynchronous processing thread to access the second remote file system to which the associated data belongs, based on the second attribute information of the associated data, and obtains the associated data.

[0140] Specifically, the unified interface service can use an asynchronous processing thread to access the second remote file system to which the associated data belongs, based on the second attribute information of the associated data, and obtain the associated data. Understandably, the specific steps for the asynchronous processing thread to access the second remote file system based on the second attribute information of the associated data are basically the same as the specific implementation steps for accessing the first remote file system based on the first attribute information; only the first attribute information needs to be modified to the second attribute information, and the first remote file system needs to be modified to the second remote file system. Therefore, these details will not be elaborated further.

[0141] S608 stores associated data in the local storage area.

[0142] Specifically, after obtaining the associated data, the database executor can store the associated data in the local storage area.

[0143] In one embodiment, after obtaining the associated data, the database executor can determine the shard capacity corresponding to the associated data based on the data capacity and processing type of the associated data, perform sharding processing on the associated data based on the shard capacity corresponding to the associated data to obtain each associated data shard, and allocate and store each associated data in the local storage area according to the sharding order.

[0144] In the above embodiments, when there is related data in the target processing data, the database executor can obtain the related data from the remote file system through an asynchronous processing thread. Without affecting the efficiency of obtaining the target processing data, the data to be processed is cached in advance for subsequent data processing operations of the target processing data, which effectively improves the data processing efficiency.

[0145] In one embodiment, such as Figure 7 As shown, the data processing method may also include the following steps:

[0146] S702, in response to a remote storage command for target cold data, acquires the target cold data and its data information.

[0147] Among them, target cold data refers to the specific data content that needs to be stored in the remote file system, which is usually database data that is accessed less frequently than a preset frequency threshold.

[0148] A remote storage command is a signal used to instruct the migration of target cold data from local storage to a remote file system. The remote storage command carries the target cold data and its associated information. Understandably, a remote storage command can be generated by a user based on the target cold data triggered by their terminal.

[0149] Among them, the data information of the target cold data refers to the metadata of the target cold data, which may include descriptive information such as the size, format, creation time, verification and storage location, access permissions, and security requirement level of the data.

[0150] Specifically, the database executor can respond to remote storage instructions for target cold data and retrieve the target cold data and its data information from the remote storage instructions.

[0151] S704, based on the security requirement level of the target cold data represented by the data information, determines the target storage system that matches the security requirement level from each remote file system.

[0152] The security requirement level is a parameter that quantifies the security protection requirements for the target cold data. A higher security requirement level indicates higher storage security requirements for the target cold data. Understandably, different remote file systems have different storage security levels. For example, OSS remote file systems are generally private cloud systems, while HDFS remote file systems are generally public cloud systems. Therefore, the storage security of OSS remote file systems is higher than that of HDFS remote file systems.

[0153] Specifically, the database executor can extract information from the data, determine the security requirement level of the target cold data, obtain the storage security of each remote file system, compare the security requirement level with the storage security of each system, and determine the target storage system that matches the security requirement level from the remote file systems.

[0154] S706 calls the unified interface service to access the target storage system based on the third attribute information of the target cold data and stores the target cold data in the target storage system.

[0155] The third attribute information is data attribute information that includes the connection parameters of the target storage system.

[0156] Specifically, after determining the target storage system for which the target cold data needs to be migrated, the database executor can call the unified interface service to access the target storage system based on the third attribute information of the target cold data and store the target cold data in the target storage system.

[0157] In the above embodiments, when it is determined that there is cold data in the database, the database executor can automatically match the corresponding target storage system for the cold data according to the security requirement level of the cold data, and call the unified interface service to access and connect with the target storage system, so as to migrate the cold data from the local storage of the database to the remote file system for storage, which effectively improves the utilization efficiency of local storage.

[0158] In one embodiment, a data processing method is provided, which can be applied to, for example... Figure 8 In the database shown, the database includes a database kernel. When the database kernel needs to access the storage file system, it can directly access the unified file system through a unified access interface. Both the unified access interface and the unified file system are abstracted and generated by the unified interface service set inside the database executor. The unified access interface can implement interface functions such as open, write, read, query, and close. The unified file system can connect to various storage file systems through connection information. The storage file systems can include remote file systems and local file systems. Remote file systems include HDFS file system and OSS file system, while local file systems can be local disks.

[0159] like Figure 9 As shown, the data processing method may specifically include the following steps:

[0160] S901, in response to a processing request initiated for target processing data, obtains the first data type and processing type of the target processing data.

[0161] S902, when the first data type is cold data, the unified interface service is invoked to generate a data acquisition task for the target data based on the first attribute information.

[0162] S903 writes the data acquisition task into the task pool of the unified interface service.

[0163] S904: If the data acquisition task meets the task execution conditions, allocate a processing thread to the data acquisition task.

[0164] S905 uses a processing thread to extract the storage medium identifier and connection parameters of the target data from the first data attribute information of the target data.

[0165] S906, determine the first remote file system to which the target processing data belongs based on the storage medium identifier.

[0166] S907, creates an access connection channel with the first remote file system based on the connection parameters.

[0167] S908 uses the access connection channel to access the first remote file system and obtain the target processing data.

[0168] S909 determines the fragment capacity corresponding to the target data to be processed based on the data capacity and processing type of the target data.

[0169] S910 performs fragment processing on the target data based on the fragment capacity to obtain each data fragment.

[0170] S911 stores each data fragment in the local storage area according to the fragment order.

[0171] S912 retrieves the target processing data from the local storage area according to the processing type and performs data processing.

[0172] While executing S904, the database executor can also asynchronously perform related data retrieval operations, specifically including the following steps:

[0173] S913, if it is determined that there is associated data in the target processing data based on preset association conditions, the second data type of the associated data is obtained.

[0174] S914 If the second data type is cold data, then the unified interface service is called to create an asynchronous processing thread for the associated data.

[0175] S915 uses an asynchronous processing thread to access the second remote file system to which the associated data belongs, based on the second attribute information of the associated data, and obtains the associated data.

[0176] S916 stores associated data in the local storage area.

[0177] In one embodiment, such as Figure 10 As shown, the unified interface service will generate an abstract unified file system at the database's computation layer. This unified file system can act as a server. When access to the unified file system is needed, a client can be built into the database executor. This client corresponds to the file access request and is used to send file access requests to the unified file system, passing connection information to establish a connection. After a successful connection, the database executor can use the basic interface based on the client to access the corresponding remote file system, such as HDFS or OSS, just like accessing local files. It can also access the local file system, i.e., local storage.

[0178] The client can generate a file access request based on the target processing data. Upon receiving the request, the server accesses the corresponding remote file system to retrieve the target processing data based on the connection information and then transmits the data to the client. Connection information may include media type, URI, user, password, etc. The client and server connect via local sockets, achieving transmission efficiency comparable to local file transfers.

[0179] The server-side component can include functional modules such as thread pools, LRU caching media, asynchronous processing threads, message processing, and connection pools. Specifically, when returning data to the client, the server caches file chunks, preventing data from being successfully transferred to disk or remote storage during subsequent accesses. Through thread pools and asynchronous processing mechanisms, it can handle high-concurrency client accesses, and with the caching media, it can quickly respond to clients.

[0180] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0181] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0182] In one embodiment, such as Figure 11 As shown, a data processing device 1100 is provided, including: a request-response module 1101, a unified interface access module 1102, a local storage module 1103, and a data processing module 1104, wherein:

[0183] The request-response module 1101 is used to respond to a processing request initiated for the target processing data and obtain the first data type and processing type of the target processing data.

[0184] The unified interface access module 1102 is used to call the unified interface service to access the remote file system to which the target data belongs and obtain the target data when the first data type is cold data.

[0185] Local storage module 1103 is used to store target processing data in the local storage area.

[0186] The data processing module 1104 is used to obtain target processing data from the local storage area according to the processing type and perform data processing.

[0187] In one embodiment, the unified interface access module 1102 is used to: call the unified interface service to extract the storage medium identifier and connection parameters of the target processing data from the first data attribute information of the target processing data; determine the first remote file system to which the target processing data belongs based on the storage medium identifier; create an access connection channel with the first remote file system according to the connection parameters; and use the access connection channel to access the first remote file system to obtain the target processing data.

[0188] In one embodiment, the local storage module 1103 is used to: determine the shard capacity corresponding to the target processing data based on the data capacity and processing type of the target processing data; perform shard processing on the target processing data based on the shard capacity to obtain each data shard; and store each data shard in the local storage area in shard order.

[0189] In one embodiment, the unified interface access module 1102 is configured to: call the unified interface service to generate a data acquisition task for the target processing data based on the first attribute information, and write the data acquisition task into the task pool of the unified interface service; allocate a processing thread for the data acquisition task if the data acquisition task meets the task execution conditions; and use the processing thread to access the first remote file system to which the target processing data belongs based on the first attribute information to acquire the target processing data.

[0190] In one embodiment, the data processing apparatus 1100 further includes:

[0191] The association condition judgment module is used to obtain the second data type of the associated data when it is determined that there is associated data in the target data to be processed based on preset association conditions.

[0192] The asynchronous processing thread creation module is used to call the unified interface service to create an asynchronous processing thread for the associated data if the second data type is cold data.

[0193] The associated data acquisition module is used to access the second remote file system to which the associated data belongs based on the second attribute information of the associated data using an asynchronous processing thread, and obtain the associated data.

[0194] The associated data storage module is used to store associated data in the local storage area.

[0195] In one embodiment, the data processing apparatus 1100 further includes:

[0196] The storage instruction response module is used to respond to remote storage instructions for target cold data and to obtain the target cold data and its data information.

[0197] The security level matching module is used to determine the target storage system that matches the security requirement level of the target cold data from various remote file systems, based on the security requirement level represented by the data information.

[0198] The data storage module is used to call the unified interface service, access the target storage system based on the third attribute information of the target cold data, and store the target cold data in the target storage system.

[0199] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0200] In one embodiment, a computer device is provided, which may be a server to which a database belongs, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to data processing methods. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0201] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0202] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the specific implementation steps in the above-described data processing method embodiment.

[0203] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the specific implementation steps in the above-described data processing method embodiment.

[0204] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the specific implementation steps in the above-described data processing method embodiment.

[0205] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the acquisition, storage, processing, and transmission of the data all comply with relevant laws and regulations.

[0206] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0208] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to a processing request initiated for target processing data, the first data type and processing type of the target processing data are obtained; In the case where the first data type is cold data, the unified interface service is invoked to access the first remote file system to which the target data belongs, based on the first attribute information of the target data, and to obtain the target data. The target processing data is stored in the local storage area; According to the processing type, the target processing data is obtained from the local storage area and processed.

2. The method according to claim 1, characterized in that, The process of calling the unified interface service to access the first remote file system to which the target processing data belongs, based on the first data attribute information of the target processing data, and to obtain the target processing data includes: The unified interface service is invoked to extract the storage medium identifier and connection parameters of the target data from the first data attribute information of the target data; The first remote file system to which the target processed data belongs is determined based on the storage medium identifier; An access connection channel with the first remote file system is created based on the connection parameters; The first remote file system is accessed using the access connection channel to obtain the target processing data.

3. The method according to claim 1, characterized in that, The step of storing the target processing data in the local storage area includes: The sharding capacity corresponding to the target processing data is determined based on the data capacity of the target processing data and the processing type. The target data to be processed is divided into data fragments based on the fragment capacity to obtain each data fragment. Each of the data fragments is stored in the local storage area in fragment order.

4. The method according to claim 1, characterized in that, The process of calling the unified interface service to access the first remote file system to which the target processing data belongs, based on the first attribute information of the target processing data, and obtaining the target processing data includes: The unified interface service is invoked to generate a data acquisition task for the target data based on the first attribute information, and the data acquisition task is written into the task pool of the unified interface service. If the data acquisition task meets the task execution conditions, a processing thread is allocated to the data acquisition task. The processing thread uses the first attribute information to access the first remote file system to which the target processing data belongs, and obtains the target processing data.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: If it is determined that the target processing data has associated data based on preset association conditions, the second data type of the associated data is obtained; If the second data type is cold data, then the unified interface service is invoked to create an asynchronous processing thread for the associated data; The asynchronous processing thread uses the second attribute information of the associated data to access the second remote file system to obtain the associated data. The associated data is stored in the local storage area.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to a remote storage command for target cold data, the target cold data and its data information are obtained; Based on the security requirement level of the target cold data as represented by the data information, a target storage system matching the security requirement level is determined from each of the remote file systems; The unified interface service is invoked to access the target storage system based on the third attribute information of the target cold data, and the target cold data is stored in the target storage system.

7. A data processing apparatus, characterized in that, The device includes: The request-response module is used to respond to a processing request initiated for target processing data and obtain the first data type and processing type of the target processing data; The unified interface access module is used to, when the first data type is cold data, call the unified interface service to access the remote file system to which the target processing data belongs based on the first attribute information of the target processing data, and obtain the target processing data. A local storage module is used to store the target processing data in a local storage area; The data processing module is used to obtain the target processing data from the local storage area according to the processing type and perform data processing.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images