Cold and hot data processing method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of big data, and provides a cold and hot data processing method and device, electronic equipment and a storage medium, the method comprises the following steps: receiving an access request of a cold and hot data table, and acquiring a first file handle; cold and hot data in a first storage space corresponding to the first file handle is classified to obtain cold data; the cold data is migrated to a second storage space, and a first file handle of the cold data is updated to a second file handle of the second storage space; when a processing request of the cold and hot data table is detected, a processing operation instruction of the processing request is acquired; and the cold and hot data table is processed based on the processing operation instruction. According to the application, the cold data of the first storage space is migrated to the second storage space, the storage logic of the cold and hot data is not changed, programs written before data migration can still run after data migration, and the processing efficiency of the cold and hot data is improved.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a method, apparatus, electronic device, and storage medium for processing hot and cold data. Background Technology

[0002] Existing cold and hot data processing technologies store cold and hot data separately, requiring the maintenance of two systems. The business logic needs to display the location of the queried data, and even the query syntax may be different, resulting in high data processing complexity. At the same time, in terms of program design, after data migration, the previously written business programs need to be rewritten, resulting in high design costs and low efficiency in cold and hot data processing. Summary of the Invention

[0003] In view of the above, it is necessary to propose a method, apparatus, electronic device and storage medium for processing cold and hot data. By migrating cold data in a first storage space to a second storage space, the storage logic of cold and hot data is not changed, and the program written before the data migration can still run after the data migration, thereby improving the processing efficiency of cold and hot data.

[0004] A first aspect of the present invention provides a method for processing hot and cold data, the method comprising:

[0005] Receive access requests for hot and cold data tables and obtain the first file handle;

[0006] The cold and hot data in the first storage space corresponding to the first file handle are classified to obtain cold data, wherein the first storage space is located in the first cluster data storage space of the file processing system;

[0007] The cold data is migrated to the second storage space, and the first file handle of the cold data is updated to the second file handle of the second storage space, wherein the second storage space is located in the second cluster data storage space of the file processing system;

[0008] When a processing request for the hot and cold data table is detected, the processing operation instruction for the processing request is obtained;

[0009] The hot and cold data table is processed based on the processing operation instructions.

[0010] Optionally, processing the hot and cold data table based on the processing operation instructions includes:

[0011] When the processing operation instruction is a data write operation, the write data and the first partition identifier code of the write data are obtained from the processing request of the hot and cold data table.

[0012] Write the data to a preset temporary storage directory;

[0013] In response to the received write data completion instruction, the source code corresponding to the hot and cold data table is queried to obtain the third file handle corresponding to the first partition identifier;

[0014] Determine whether the fourth file handle of the temporary directory is the same as the third file handle;

[0015] When the fourth file handle of the temporary storage directory is the same as the third file handle, a preset first function is called to update the fourth file handle of the temporary storage directory to the third file handle, and the written data is mapped to the hot and cold data table.

[0016] Optionally, the method further includes;

[0017] When the fourth file handle of the temporary directory is different from the third file handle, the written data in the temporary directory is copied to the third storage space corresponding to the third file handle, the temporary directory is deleted, and the written data is mapped to the hot and cold data table.

[0018] Optionally, processing the hot and cold data table based on the processing operation instructions includes:

[0019] When the processing operation instruction is a partition deletion operation, the second partition identifier code is obtained from the processing request of the hot and cold data table;

[0020] By querying the source code corresponding to the hot and cold data table, the fifth file handle corresponding to the second partition identifier is obtained;

[0021] Obtain the default file handle associated with the hot and cold data table;

[0022] Determine whether the fifth file handle is the same as the default file handle;

[0023] When the fifth file handle is the same as the default file handle, the second partition corresponding to the second partition identifier is deleted by calling the preset second function through the default file handle.

[0024] Optionally, the method further includes:

[0025] When the fifth file handle is different from the default file handle, the fifth file handle is assigned to the default file handle to obtain a new file handle;

[0026] The second partition corresponding to the second partition identifier is deleted by calling the preset second function through the new file handle.

[0027] Optionally, classifying the hot and cold data in the first storage space corresponding to the first file handle to obtain cold data includes:

[0028] The hot and cold data are input into a pre-trained hot and cold data classification model, and the cold data output by the hot and cold data model is received.

[0029] Optionally, before inputting the hot and cold data into a pre-trained hot and cold data classification model, the method further includes:

[0030] Obtain the historical dataset and the access frequency of each historical data point;

[0031] Based on the access frequency of each historical data point, the corresponding historical data in the historical dataset are labeled, and the labeled historical dataset is used as the training set.

[0032] The training set is used to train a preset neural network to obtain a hot and cold data classification model.

[0033] A second aspect of the present invention provides a cold and hot data processing apparatus, the apparatus comprising:

[0034] The receiving module is used to receive access requests for the hot and cold data tables and obtain the first file handle;

[0035] The classification module is used to classify the hot and cold data in the first storage space corresponding to the first file handle to obtain the cold data;

[0036] The migration module is used to migrate the cold data to the second storage space and update the first file handle of the cold data to the second file handle of the second storage space;

[0037] The acquisition module is used to acquire the processing operation instructions of the processing request when a processing request for the hot and cold data table is detected.

[0038] The processing module is used to process the hot and cold data table based on the processing operation instructions.

[0039] A third aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the processor is configured to implement the hot and cold data processing method by executing a computer program stored in the memory.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the hot and cold data processing method described above.

[0041] In summary, the cold and hot data processing method, apparatus, electronic device, and storage medium described in this invention can promote the construction of smart cities and be applied in fields such as smart buildings, smart security, smart communities, smart living, and the Internet of Things. By migrating cold data from the first storage space to the second storage space, the storage logic of the cold and hot data remains unchanged, and programs written before the data migration can still run after the migration, thus improving the processing efficiency of cold and hot data. When a processing request for the cold and hot data table is detected, the processing operation instructions for the processing request are obtained, and the cold and hot data table is processed. During the processing, data is written or partitioned for deletion according to the operation strategy corresponding to different processing operation instructions. During the data writing or partition deletion process, it is not necessary to modify the code of the data before migration, which facilitates user operation, improves user experience and satisfaction, and simultaneously improves the processing efficiency of cold and hot data. Attached Figure Description

[0042] Figure 1 This is a flowchart of the hot and cold data processing method provided in Embodiment 1 of the present invention.

[0043] Figure 2 This is a structural diagram of the cold and hot data processing device provided in Embodiment 2 of the present invention.

[0044] Figure 3 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0045] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0047] Example 1

[0048] Figure 1 This is a flowchart of the hot and cold data processing method provided in Embodiment 1 of the present invention.

[0049] In this embodiment, the hot and cold data processing method can be applied to electronic devices. For electronic devices that need to perform hot and cold data processing, the hot and cold data processing function provided by the method of this invention can be directly integrated into the electronic device, or it can run in the electronic device in the form of a software development kit (SDK).

[0050] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0051] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, as well as machine learning and deep learning.

[0052] like Figure 1 As shown, the cold and hot data processing method specifically includes the following steps. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0053] 101. Receive the access request for the hot and cold data table and obtain the first file handle.

[0054] In this embodiment, the user requests the access to cold and hot data in the cold and hot data table to process the cold and hot data. Here, cold data refers to offline data that is not frequently accessed, such as backup and archive data.

[0055] In this embodiment, the cold and hot data are stored in the HDFS (Hadoop Distributed File System) file processing system. HDFS first splits each file into many partitions, such as 128MB each, and then each partition is stored on a node and stored with multiple replicas.

[0056] In this embodiment, the file handle is used to uniquely identify the opened file. Specifically, in file input and output, to read data from a file, the application first calls the operating system function and passes the file name, and selects a path to the file to open the file. The function retrieves a sequence number, which is the file handle.

[0057] In this embodiment, the first file handle is used to uniquely identify the accessed hot and cold data tables.

[0058] 102. Classify the hot and cold data in the first storage space corresponding to the first file handle to obtain the cold data.

[0059] In this embodiment, the first storage space is located in the first cluster data storage space of the HDFS file processing system.

[0060] In this embodiment, after obtaining the first file handle, the corresponding first storage space can be identified based on the first file handle, and cold data can be obtained from the cold and hot data in the first storage space.

[0061] In an optional embodiment, classifying the hot and cold data in the first storage space corresponding to the first file handle to obtain cold data includes:

[0062] The hot and cold data are input into a pre-trained hot and cold data classification model, and the cold data output by the hot and cold data model is received.

[0063] In this embodiment, a hot and cold data classification model can be pre-trained, and hot and cold data in the first storage space can be input into the hot and cold data classification model to output cold data.

[0064] In an optional embodiment, before inputting the hot and cold data into a pre-trained hot and cold data classification model, the method further includes:

[0065] Obtain the historical dataset and the access frequency of each historical data point;

[0066] Based on the access frequency of each historical data point, the corresponding historical data in the historical dataset are labeled, and the labeled historical dataset is used as the training set.

[0067] The training set is used to train a preset neural network to obtain a hot and cold data classification model.

[0068] In this embodiment, cold data tags can be set in advance based on the access frequency. For example, if the access rate is less than or equal to N times per month, it is determined as a cold data tag.

[0069] In this embodiment, a tagging tool is used to tag the corresponding historical data according to the access frequency of each historical data. The tagging tool can be labelimg, labelme, or the NLP annotation tool BRAT. The tagging tool is an existing technology, and different tagging tools are determined according to the specific scenario. This embodiment does not impose any restrictions on this.

[0070] In other optional embodiments, classifying the hot and cold data in the first storage space corresponding to the first file handle to obtain cold data includes:

[0071] Sort the hot and cold data in the first storage space according to their chronological order;

[0072] Select the data that appears last in the sorted list and define them as cold data.

[0073] In this embodiment, cold data can be determined based on the order of storage time of hot and cold data. For example, data from the current three months can be determined as hot data, and data from three months later can be determined as cold data.

[0074] In other alternative embodiments, hot and cold data can be divided according to user needs. For example, business data that has a smaller impact on business metrics can be identified as cold data.

[0075] 103. The cold data is migrated to the second storage space, and the first file handle of the cold data is updated to the second file handle of the second storage space.

[0076] In this embodiment, the second storage space is located in the second cluster data storage space of the file processing system.

[0077] In this embodiment, the first storage space and the second storage space belong to different clusters of the file processing system, but the storage logic of the data in different clusters is the same.

[0078] In this embodiment, the first storage space corresponds to the path of an HDFS cluster, and the second storage space corresponds to the path of an HDFS cluster. Users can quickly determine the storage space for cold data and the storage space for hot data based on the cluster path, which is transparent to users. At the same time, no previous code needs to be modified when accessing cold data and hot data, which is convenient for users and improves user experience and satisfaction.

[0079] In this embodiment, HDFS storage is generally partitioned based on date. Infrequently used cold data is migrated to a high-density storage cluster, i.e., the second storage space. The first file handle of the cold data is updated to the second file handle of the second storage space. In this way, hot and cold data are separated in the same table, and there are two different HDFS cluster paths. Users know the spaces where hot and cold data are stored, which is transparent to users. At the same time, the storage logic of hot and cold data does not change. Hot and cold data processing can be achieved through operations on a single table. Furthermore, a single system is used to store hot and cold data, which solves the problems in the prior art that require two separate systems to store hot and cold data and that programs written before data migration cannot run after data migration.

[0080] 104. When a processing request for the hot and cold data table is detected, the processing operation instruction for the processing request is obtained.

[0081] In this embodiment, after a table is separated into hot and cold data, different partitions may correspond to different paths in the HDFS cluster. When Spark is used to access the hot and cold data tables, the Spark source code needs to be modified to add processing instructions based on the different paths in the HDFS cluster, so that the business code before migration can continue to run.

[0082] 105. Based on the processing operation instructions, process the hot and cold data table.

[0083] In this embodiment, the processing operation instructions include a data write operation and a partition delete operation. The data write operation refers to writing the data in the processing request into the hot and cold data table, and the partition delete operation refers to deleting the partition in the processing request.

[0084] In an optional embodiment, processing the hot and cold data table based on the processing operation instructions includes:

[0085] When the processing operation instruction is a data write operation, the write data and the first partition identifier code of the write data are obtained from the processing request of the hot and cold data table.

[0086] Write the data to a preset temporary storage directory;

[0087] In response to the received write data completion instruction, the source code corresponding to the hot and cold data table is queried to obtain the third file handle corresponding to the first partition identifier;

[0088] Determine whether the fourth file handle of the temporary directory is the same as the third file handle;

[0089] When the fourth file handle of the temporary storage directory is the same as the third file handle, a preset first function is called to update the fourth file handle of the temporary storage directory to the third file handle, and the written data is mapped to the hot and cold data table.

[0090] Furthermore, the method also includes;

[0091] When the fourth file handle of the temporary directory is different from the third file handle, the written data in the temporary directory is copied to the third storage space corresponding to the third file handle, the temporary directory is deleted, and the written data is mapped to the hot and cold data table.

[0092] In this embodiment, when performing a data write operation, to prevent the program written before the data migration from still running after the data migration, after the data is written to a preset temporary directory, a preset first function is called through the fourth file handle corresponding to the temporary directory to modify the fourth file handle of the temporary directory to the third file handle corresponding to the data being written. Specifically, by determining whether the fourth file handle of the temporary directory is the same as the third file handle, if they are the same, it means that the temporary directory and the target directory of the data being written are located on the same HDFS, and it is only necessary to call the preset first function (rename function) through the fourth file handle to modify the fourth file handle of the temporary directory to the third file handle; if they are different, it means that the temporary directory and the target directory of the data being written are not located on the same HDFS, and in this case, it is necessary to call the file copy function FileUtil.copy to copy the data being written in the temporary directory to the third storage space corresponding to the third file handle.

[0093] In an optional embodiment, processing the hot and cold data table based on the processing operation instructions includes:

[0094] When the processing operation instruction is a partition deletion operation, the second partition identifier code is obtained from the processing request of the hot and cold data table;

[0095] By querying the source code corresponding to the hot and cold data table, the fifth file handle corresponding to the second partition identifier is obtained;

[0096] Obtain the default file handle associated with the hot and cold data table;

[0097] Determine whether the fifth file handle is the same as the default file handle;

[0098] When the fifth file handle is the same as the default file handle, the second partition corresponding to the second partition identifier is deleted by calling the preset second function through the default file handle.

[0099] Furthermore, the method also includes:

[0100] When the fifth file handle is different from the default file handle, the fifth file handle is assigned to the default file handle to obtain a new file handle;

[0101] The second partition corresponding to the second partition identifier is deleted by calling the preset second function through the new file handle.

[0102] In this embodiment, the hot and cold data tables are associated with a default file handle, which refers to the handle corresponding to the HDFS file system where the hot and cold data reside.

[0103] In this embodiment, the corresponding fifth file handle is obtained through the second partition identifier code, and it is determined whether the fifth file handle is the same as the default file handle. Based on the determination result, different deletion methods are used to delete the corresponding partition. This operation is performed within the same system, which solves the problem of low partition deletion efficiency caused by the fact that in the prior art, after implementing cold and hot separation operation, some partition file handles are the handles corresponding to the file processing system where cold data is located. If the delete operation is called through the file handle of the current file processing system, two sets of code need to be written to delete the corresponding partition.

[0104] In this embodiment, after completing the hot and cold data separation process, the storage logic of the hot and cold data tables on HDFS remains unchanged. This storage method reduces storage costs and avoids the inconvenience to users caused by traditionally storing hot and cold data in two separate systems. In this embodiment, accessing the separated hot and cold data through Spark does not require modifying the code of the data before migration, which facilitates user operations, improves user experience and satisfaction, and also improves the efficiency of hot and cold data processing.

[0105] In summary, the hot and cold data processing method described in this embodiment migrates cold data from the first storage space to the second storage space without altering the storage logic of the hot and cold data. Programs written before the data migration can still run after the migration, thus improving the processing efficiency of hot and cold data. When a processing request for the hot and cold data table is detected, the processing operation instructions for the request are obtained, and the hot and cold data tables are processed. During the processing, data writing or partition deletion is performed according to the operation strategy corresponding to different processing operation instructions. During data writing or partition deletion, it is not necessary to modify the code of the data before migration, which facilitates user operation, improves user experience and satisfaction, and simultaneously improves the efficiency of hot and cold data processing.

[0106] Example 2

[0107] Figure 2 This is a structural diagram of the cold and hot data processing device provided in Embodiment 2 of the present invention.

[0108] In some embodiments, the hot and cold data processing device 20 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the hot and cold data processing device 20 may be stored in the memory of an electronic device and executed by the at least one processor to perform (see details). Figure 1 (Description) Functionality for processing hot and cold data.

[0109] In this embodiment, the hot and cold data processing device 20 can be divided into multiple functional modules according to the functions it performs. These functional modules may include: a receiving module 201, a classification module 202, a migration module 203, an acquisition module 204, and a processing module 205. The module referred to in this invention is a series of computer-readable instruction segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0110] The receiving module 201 is used to receive access requests for the hot and cold data tables and obtain the first file handle.

[0111] The classification module 202 is used to classify the hot and cold data in the first storage space corresponding to the first file handle to obtain the cold data.

[0112] The migration module 203 is used to migrate the cold data to the second storage space and update the first file handle of the cold data to the second file handle of the second storage space.

[0113] The acquisition module 204 is used to acquire the processing operation instructions of the processing request when a processing request for the hot and cold data table is detected.

[0114] The processing module 205 is used to process the hot and cold data table based on the processing operation instructions.

[0115] In an optional embodiment, the classification module 202 is configured to: input the hot and cold data into a pre-trained hot and cold data classification model, and receive the cold data output by the hot and cold model.

[0116] In an optional embodiment, before the classification module 202 inputs the hot and cold data into the pre-trained hot and cold data classification model, a historical dataset and the access frequency of each historical data are obtained; the corresponding historical data in the historical dataset are labeled based on the access frequency of each historical data, and the labeled historical dataset is used as the training set; the training set is used to train a preset neural network to obtain the hot and cold data classification model.

[0117] In other optional embodiments, classifying the hot and cold data in the first storage space corresponding to the first file handle to obtain cold data includes: sorting the hot and cold data in the first storage space according to time sequence; and selecting the data that are ranked last to be determined as cold data.

[0118] In an optional embodiment, the processing module 205 is configured to: when the processing operation instruction is a data write operation, obtain the write data and the first partition identifier of the write data from the processing request of the hot and cold data table; write the write data to a preset temporary storage directory; in response to the received write data completion instruction, query the source code corresponding to the hot and cold data table to obtain the third file handle corresponding to the first partition identifier; determine whether the fourth file handle of the temporary storage directory is the same as the third file handle; when the fourth file handle of the temporary storage directory is the same as the third file handle, call a preset first function to update the fourth file handle of the temporary storage directory to the third file handle, and map the write data to the hot and cold data table; when the fourth file handle of the temporary storage directory is not the same as the third file handle, copy the write data under the temporary storage directory to the third storage space corresponding to the third file handle, delete the temporary storage directory, and map the write data to the hot and cold data table.

[0119] In an optional embodiment, the processing module 205 is configured to: when the processing operation instruction is a partition deletion operation, obtain a second partition identifier from the processing request of the hot and cold data table; query the source code corresponding to the hot and cold data table to obtain a fifth file handle corresponding to the second partition identifier; and obtain a default file handle associated with the hot and cold data table.

[0120] Determine whether the fifth file handle is the same as the default file handle; if the fifth file handle is the same as the default file handle, call the preset second function through the default file handle to delete the second partition corresponding to the second partition identifier; if the fifth file handle is different from the default file handle, assign the fifth file handle to the default file handle to obtain a new file handle; call the preset second function through the new file handle to delete the second partition corresponding to the second partition identifier.

[0121] In this embodiment, the hot and cold data tables are associated with a default file handle, which refers to the handle corresponding to the HDFS file system where the hot and cold data reside.

[0122] In this embodiment, the corresponding fifth file handle is obtained through the second partition identifier code, and it is determined whether the fifth file handle is the same as the default file handle. Based on the determination result, different deletion methods are used to delete the corresponding partition. This operation is performed within the same system, which solves the problem of low partition deletion efficiency caused by the fact that in the prior art, after implementing cold and hot separation operation, some partition file handles are the handles corresponding to the file processing system where cold data is located. If the delete operation is called through the file handle of the current file processing system, two sets of code need to be written to delete the corresponding partition.

[0123] In this embodiment, after completing the hot and cold data separation process, the storage logic of the hot and cold data tables on HDFS remains unchanged. This storage method reduces storage costs and avoids the inconvenience to users caused by traditionally storing hot and cold data in two separate systems. In this embodiment, accessing the separated hot and cold data through Spark does not require modifying the code of the data before migration, which facilitates user operations, improves user experience and satisfaction, and also improves the efficiency of hot and cold data processing.

[0124] In summary, the hot and cold data processing device described in this embodiment migrates cold data from the first storage space to the second storage space without altering the storage logic of the hot and cold data. Programs written before the data migration can still run after the migration, thus improving the processing efficiency of hot and cold data. When a processing request for the hot and cold data table is detected, the processing operation instructions for the request are obtained, and the hot and cold data tables are processed. During the processing, data is written or partitioned for deletion according to the operation strategy corresponding to different processing operation instructions. During data writing or partition deletion, it is not necessary to modify the code of the data before migration, which facilitates user operation, improves user experience and satisfaction, and simultaneously increases the efficiency of hot and cold data processing.

[0125] Example 3

[0126] See Figure 3 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0127] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0128] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 3 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0129] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0130] In some embodiments, the memory 31 is used to store program code and various data, such as the hot and cold data processing device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0131] In some embodiments, the at least one processor 32 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control unit of the electronic device 3, connecting various components of the entire electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0132] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0133] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Optionally, the power supply may be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0134] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0135] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of the present invention.

[0136] In a further embodiment, combined with Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed applications (such as the hot and cold data processing device 20), program code, etc., for example, the various modules mentioned above.

[0137] The memory 31 stores program code, and the at least one processor 32 can call the program code stored in the memory 31 to execute related functions. For example, Figure 2 The modules described herein are program codes stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of cold and hot data processing.

[0138] For example, the program code can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete this application. The one or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the program code in the electronic device 3. For example, the program code can be divided into a receiving module 201, a classification module 202, a migration module 203, an acquisition module 204, and a processing module 205.

[0139] In one embodiment of the present invention, the memory 31 stores a plurality of computer-readable instructions, which are executed by the at least one processor 32 to perform the function of hot and cold data processing.

[0140] Specifically, the specific implementation method of the above instructions by the at least one processor 32 can be referred to Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0141] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0142] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0144] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements, and the singular does not exclude the plural. Multiple elements or devices recited in the present invention may also be implemented by a single element or device in software or hardware. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for processing hot and cold data, characterized in that, The method includes: Receive access requests for hot and cold data tables and obtain the first file handle; The cold and hot data in the first storage space corresponding to the first file handle are classified to obtain cold data, wherein the first storage space is located in the first cluster data storage space of the file processing system; The cold data is migrated to the second storage space, and the first file handle of the cold data is updated to the second file handle of the second storage space, wherein the second storage space is located in the second cluster data storage space of the file processing system; When a processing request for the hot and cold data table is detected, the processing operation instruction for the processing request is obtained; Based on the processing operation instruction, the hot and cold data tables are processed, including: when the processing operation instruction is a data write operation, obtaining the write data and the first partition identifier of the write data from the processing request of the hot and cold data tables; writing the write data into a preset temporary storage directory; responding to the received write data completion instruction, querying the source code corresponding to the hot and cold data tables to obtain the third file handle corresponding to the first partition identifier; determining whether the fourth file handle of the temporary storage directory is the same as the third file handle; when the fourth file handle of the temporary storage directory is the same as the third file handle, calling a preset first function to update the fourth file handle of the temporary storage directory to the third file handle, and mapping the write data to the hot and cold data tables; when the fourth file handle of the temporary storage directory is not the same as the third file handle, copying the write data in the temporary storage directory to the third storage space corresponding to the third file handle, deleting the temporary storage directory, and mapping the write data to the hot and cold data tables.

2. The cold and hot data processing method as described in claim 1, characterized in that, The processing of the hot and cold data table based on the processing operation instructions further includes: When the processing operation instruction is a partition deletion operation, the second partition identifier code is obtained from the processing request of the hot and cold data table; Query the source code corresponding to the hot and cold data table to obtain the fifth file handle corresponding to the second partition identifier; Obtain the default file handle associated with the hot and cold data table, wherein the default file handle refers to the handle associated with the hot and cold data in the hot and cold data table in the file processing system where the hot and cold data is located; Determine whether the fifth file handle is the same as the default file handle; When the fifth file handle is the same as the default file handle, the second partition corresponding to the second partition identifier is deleted by calling the preset second function through the default file handle.

3. The cold and hot data processing method as described in claim 2, characterized in that, The method further includes: When the fifth file handle is different from the default file handle, the fifth file handle is assigned to the default file handle to obtain a new file handle; The second partition corresponding to the second partition identifier is deleted by calling the preset second function through the new file handle.

4. The cold and hot data processing method as described in claim 1, characterized in that, The classification of hot and cold data in the first storage space corresponding to the first file handle to obtain cold data includes: The hot and cold data are input into a pre-trained hot and cold data classification model, and the cold data output by the hot and cold data classification model is received.

5. The cold and hot data processing method as described in claim 4, characterized in that, Before inputting the hot and cold data into the pre-trained hot and cold data classification model, the method further includes: Obtain the historical dataset and the access frequency of each historical data point; Based on the access frequency of each historical data point, the corresponding historical data in the historical dataset are labeled, and the labeled historical dataset is used as the training set. The training set is used to train a preset neural network to obtain a hot and cold data classification model.

6. A cold and hot data processing device, characterized in that, The apparatus is used to implement the hot and cold data processing method as described in any one of claims 1 to 5, the apparatus comprising: The receiving module is used to receive access requests for the hot and cold data tables and obtain the first file handle; The classification module is used to classify hot and cold data in the first storage space corresponding to the first file handle to obtain cold data, wherein the first storage space is located in the first cluster data storage space of the file processing system; A migration module is used to migrate the cold data to a second storage space and update the first file handle of the cold data to the second file handle of the second storage space, wherein the second storage space is located in the second cluster data storage space of the file processing system; The acquisition module is used to acquire the processing operation instructions of the processing request when a processing request for the hot and cold data table is detected. The processing module is used to process the hot and cold data table based on the processing operation instructions.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is configured to implement the hot and cold data processing method as described in any one of claims 1 to 5 when executing a computer program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the hot and cold data processing method as described in any one of claims 1 to 5.

Images