Data processing method, device, system and electronic device

Abstract

The present invention provides a data processing method, device, system and electronic device. After receiving user data, the data in the memory is updated according to the user data. During the updating process, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is written to the disk. The first data includes: the data in the memory when the instruction is received. In this method, during the updating process of the data in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the updating of the data in the memory and the writing of the data in the memory to the disk are performed simultaneously, avoiding data update blockage and improving data processing performance.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, system and electronic equipment. Background Art

[0002] When the contents of a memory data page are inconsistent with those on a disk data page, the memory data page is called a dirty page. When the data in the memory data page is written to the disk data page, the contents of the memory data page and the disk data page are consistent. The process of writing this data to the disk is the process of flushing dirty pages. Since the memory is constantly receiving user data, the data on the memory data page is constantly updated. However, in the process of flushing dirty pages, for a certain data page in the memory, the data page needs to be locked first. At this time, the data page can no longer receive user data or update data. Then the data of the data page is written to the disk. After the data of the data page is written to the disk, the data page is unlocked. At this time, the data page can continue to receive user data and update data. This method of locking the data page when flushing dirty pages requires data pages to be locked. Data needs to wait until the memory data page is written to the disk before it can be updated to the memory data page. This is easy to cause data update blockage and low data processing performance. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a data processing method, device, system and electronic device to avoid data update blockage and improve data processing performance.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, which is applied to a first device running a memory; a disk is presupposed to run in a second device, and the method comprises: receiving user data, and updating the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, writing the first data in the memory to the disk; wherein the first data comprises: the data in the memory when the instruction is received.

[0005] Furthermore, before the step of writing the first data in the memory to the disk, the method also includes: recording the current logical sequence number of the redo log of the disk; the method also includes: if the second device crashes during the process of writing the first data in the memory to the disk, writing the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0006] Furthermore, the memory is divided into multiple storage blocks; the disk stores multiple disk files; each of the disk files is used to store updated data of at least a part of the multiple storage blocks; the step of writing the first data in the memory to the disk includes: determining the target disk file corresponding to the storage block to which the first data belongs from the multiple disk files; and writing the first data to the target disk file.

[0007] Furthermore, the storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

[0008] In a second aspect, an embodiment of the present invention provides a data processing method, which is applied to a second device running a disk; it is assumed that a memory runs in a first device, and the method includes: receiving first data in the memory from the first device; wherein the first data is sent by the first device in the following manner: the first device receives user data, and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received; and the first data is written to the disk.

[0009] Furthermore, before the step of receiving the first data in the memory from the first device, the method also includes: if a first instruction from the first device is received, recording the current logical sequence number of the redo log of the disk; the method also includes: if the second device crashes during the process of writing the first data in the memory to the disk, and receives a second instruction from the first device, writing the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0010] Furthermore, the memory is divided into multiple storage blocks; the disk stores multiple disk files; each of the disk files is used to store updated data of at least a part of the multiple storage blocks; the step of writing the first data to the disk includes: writing the first data to a target disk file among the multiple disk files; wherein the target disk file is: the disk file corresponding to the storage block to which the first data belongs among the multiple disk files.

[0011] Furthermore, the storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

[0012] In a third aspect, an embodiment of the present invention provides a data processing device, which is arranged in a first device running a memory; a disk is preset to run in a second device, and the device includes: an update module, used to receive user data and update the data in the memory according to the user data; a first write module, used to write the first data in the memory to the disk if an instruction to write the data in the memory to the disk is triggered during the process of updating the data in the memory; wherein the first data includes: the data in the memory when the instruction is received.

[0013] In a fourth aspect, an embodiment of the present invention provides a data processing device, which is arranged in a second device running a disk; a memory is preset in the first device, and the device includes: a receiving module, used to receive first data in the memory from the first device; wherein the first data is sent by the first device in the following manner: the first device receives user data, and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received; a second writing module, used to write the first data to the disk.

[0014] In a fifth aspect, an embodiment of the present invention provides a data update system, the system comprising a first device running a memory and a second device running a disk; the first device is used to: receive user data, and update the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device; wherein the first data includes: the data in the memory when the instruction is received; the second device is used to: receive the first data in the memory from the first device; and write the first data to the disk.

[0015] In a sixth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data processing method described in any one of the first aspect, or the data processing method described in any one of the second aspect.

[0016] In the seventh aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the data processing method described in any one of the first aspect, or the data processing method described in any one of the second aspect.

[0017] The data processing method, device, system and electronic device provided by the present invention, when receiving user data, updates the data in the memory according to the user data; during the updating process, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is written to the disk; wherein the first data includes: the data in the memory when the instruction is received. In this method, during the updating process of the data in the memory, it is not necessary to lock the data page in the memory being updated, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are carried out simultaneously, avoiding data update blockage and improving data processing performance.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of another data processing method provided by an embodiment of the present invention;

[0023] Figure 3 A flowchart of another data processing method provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of another data processing method provided by an embodiment of the present invention;

[0025] Figure 5 A flowchart of another data processing method provided by an embodiment of the present invention;

[0026] Figure 6 A flowchart of another data processing method provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a data processing method provided by an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0029] Fig. 9 A schematic diagram of the structure of another data processing device provided by an embodiment of the present invention;

[0030] Fig.10 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0031] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] Due to the large amount of information currently, a huge amount of data is stored in the database, and the number of rows of many single table data has reached tens of millions or even hundreds of millions of data. In database storage, the large amount of data in a table is stored in a disk file, which makes the single disk file too large, which will lead to low efficiency in flushing dirty pages; and when data needs to be queried, due to the large step length of traversal, positioning search is required, which will cause more random searches and low query efficiency. In the process of flushing dirty pages, for a certain data page in memory, the data page needs to be locked first. At this time, the data page can no longer receive user data or update data, or the dirty page cannot be flushed when receiving user data or updating the data page; this method of locking the data page when flushing dirty pages, in order to ensure the correctness of the data page, makes it impossible to flush dirty pages and update memory data pages at the same time, which is easy to cause data update blockage, low efficiency in flushing dirty pages, and low data processing performance.

[0034] Based on this, an embodiment of the present invention provides a data processing method, device, system and electronic device. This technology can be applied to applications in which data in the memory is written to a disk during the data update process in the memory. To facilitate the understanding of this embodiment, a data processing method disclosed in an embodiment of the present invention is first described in detail. The method is applied to a first device running a memory; a disk is presupposed to run in the second device. In actual implementation, the first device and the second device may be the same device or different devices. For example, if the disk adopts a shared storage method, the memory and the disk may run on different devices. In other words, the memory and the disk may run on the same device or on different devices respectively. The above-mentioned disk file can be understood as a file stored in a computer, and the storage information in the disk file is not affected by power outages. Figure 1 As shown, the method comprises the following steps:

[0035] Step S102, receiving user data, and updating the data in the memory according to the user data.

[0036] The above-mentioned user data can be understood as data input by the user that needs to be calculated or instructions such as compiled code segments; when the user data is updated, the first device running with memory receives the user data and updates the data in the memory according to the user data; for example, if the first device receives a set of new calculation data, the first device can update this set of new calculation data to the memory.

[0037] Step S104, during the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is written to the disk; wherein the first data includes: the data in the memory when the instruction is received.

[0038] During the process of the first device updating the data in the memory according to the user data, if the flush dirty page instruction is triggered, that is, the data in the memory needs to be written to the disk, the data in the memory can be written to the disk when the flush dirty page instruction is received; for example, when the flush dirty page instruction is received, the total number of bytes of data in the memory is 5000, then the total number of 5000 bytes of data in the memory can be written to the disk.

[0039] A data processing method provided by an embodiment of the present invention updates the data in the memory according to the user data after receiving the user data; during the updating process, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is written to the disk; wherein the first data includes: the data in the memory when the instruction is received. In this method, during the updating process of the data in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the updating of the data in the memory and the writing of the data in the memory to the disk are performed simultaneously, thus avoiding data update blockage and improving data processing performance.

[0040] The embodiment of the present invention also provides another data processing method, which is implemented on the basis of the method in the above embodiment; Figure 2 As shown, the method comprises the following steps:

[0041] Step S202, receiving user data, and updating the data in the memory according to the user data.

[0042] Step S204: During the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the current logical sequence number of the redo log on the disk is recorded.

[0043] The redo log can be used to record the address of data refresh before refreshing the data in the memory to the disk. The purpose of using the redo log is to save the records that have not been refreshed to the disk to prevent loss. As the data gradually increases, the redo log also gradually increases, such as the redo log can increase from 1M to 10M, and then to 100M. In actual implementation, the logical sequence number can be represented by LSN (Log Sequence Number), which is used to identify the location of a specific log file record in the redo log. The growth rate of LSN in the redo log is directly related to the database activity, that is, as transactions occur and data is written to the redo log, the LSN will continue to increase. The more activities in the database, the faster the LSN grows. It can also be understood that LSN is an integer number that increases continuously, indicating the total amount of bytes of data written to the redo log, or the size of the redo log. For example, when the total amount of bytes in the redo log is 1k, the LSN is 1024; when the total amount of bytes in the redo log is 2k, the LSN is 2048.

[0044] When the first device updates the data in the memory according to the user data, if the dirty page flush instruction is triggered, that is, the data in the memory needs to be written to the disk, it is usually necessary to record the current logical sequence number of the redo log of the disk first; in actual implementation, the logical sequence number can be recorded in a marker file. For example, when the current LSN of the redo log is 1024, 1024 can be recorded in a marker file.

[0045] Step S206, writing the first data in the memory to the disk; wherein the first data includes: the data in the memory when the instruction is received.

[0046] Step S208, if the second device crashes during the process of writing the first data in the memory to the disk, write the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0047] In actual implementation, if the second device running the disk crashes during the process of writing the first data in the memory to the disk, the data in the disk may be incomplete. When restoring the data, the current data in the memory can be written to the disk according to the above-mentioned logical serial number; the amount of data of the current data may be greater than the amount of data of the first data, or may be equal to the amount of data of the first data.

[0048] For example, when receiving the disk flush instruction, the amount of the first data in the memory is 5000, and the current LSN of the recorded disk redo log is 2048. If the second device crashes during the process of writing the first data in the memory to the disk, if at the moment of the crash, the data in the disk has been updated to 3000, that is, part of the data has been written to the disk, and part of the data has not been written to the disk, if there is no abnormality in the device, the dirty page flushing and the process of updating the data in the memory can continue. However, if the second device crashes, considering that at the moment of the crash, the data in the memory and the data in the disk are not equal, then, when restoring the data in the disk, in order to ensure the integrity of the data in the disk, it is necessary to write the current data in the memory to the disk starting from the position of LSN 2048, which is equivalent to writing the data between 2048 and 3000 to the disk twice. Although part of the disk space is sacrificed, the data processing efficiency is improved because the two processes of updating the data in the memory and writing the data in the memory to the disk can be performed simultaneously.

[0049] In addition, if the data in the memory is not updated during the downtime of the second device, when the second device resumes normal operation and needs to restore the data in the disk, the amount of current data in the memory is the same as the amount of first data before the downtime, both are 5000; if the data in the memory is updated during the downtime of the second device, when the second device resumes normal operation and needs to restore the data in the disk, the amount of current data in the memory is greater than the amount of first data before the downtime, for example, the amount of current data in the memory may reach 6000, 7000 or more.

[0050] Another data processing method provided by an embodiment of the present invention, when receiving user data, first updates the data in the memory according to the user data; during the update process, if the instruction to write the data in the memory to the disk is triggered, the current logical sequence number of the redo log of the disk is recorded, and then the first data in the memory is written to the disk; if the second device crashes during the writing process, the current data in the memory is written to the disk according to the logical sequence number. In this method, during the data update process in the memory, it is not necessary to lock the data page in the memory being updated, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are carried out simultaneously, avoiding data update blockage and improving data processing performance.

[0051] The embodiment of the present invention further provides another data processing method, which is implemented on the basis of the method of the above embodiment. The method focuses on describing the specific process of writing the first data in the memory to the disk, which specifically corresponds to the following step S304.

[0052] In the method, the memory is divided into multiple storage blocks; the storage block can be understood as a data unit for processing memory data, and each storage block is equivalent to an independent storage unit; for easy distinction, the multiple storage blocks can be numbered respectively, such as storage block No. 1, storage block No. 2, etc.; multiple disk files are stored in the disk; each disk file is used to store update data of at least a part of the multiple storage blocks; in the prior art, the disk usually only stores one larger disk file, and the large amount of data of a data table is stored in this larger disk file, that is, a data table corresponds to a larger disk file, resulting in a single table data volume that is too large and a single disk file that is too large. For example, the size of a larger disk file may reach 1G or more, so when reading, updating or randomly searching for data in the larger disk file, the efficiency is relatively low. In this embodiment, a larger disk file corresponding to the large amount of data in the data table can be split into multiple disk files. To facilitate distinction, the multiple disk files can be numbered, such as disk file A, disk file B, etc.; each disk file can be used to store updated data of one or more storage blocks of the memory; wherein the size of each disk file can be configured according to actual needs, for example, the size of each disk file can be configured to 1M or 2M, etc., usually the default is 1M.

[0053] The storage blocks in the multiple storage blocks correspond to the disk files in the multiple disk files one by one. In actual implementation, the originally larger disk files are usually split based on the storage blocks in the memory. One storage block in the memory can correspond to one disk file. For example, if the memory is divided into 10 storage blocks, 10 disk files can be stored in the disk. One storage block corresponds to one disk file, such as storage block No. 1 corresponds to disk file A, data block No. 2 corresponds to disk file B, etc.; or, in order to further ensure the security of data in the disk, multiple disk files can correspond to the same storage block. For example, a disk file group can be composed of disk file A and disk file B, and the disk file group corresponds to storage block No. 1. The specific implementation method can be determined according to actual needs. Figure 3 As shown, the method comprises the following steps:

[0054] Step S302, receiving user data, and updating the data in the memory according to the user data.

[0055] Step S304, in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, determine the target disk file corresponding to the storage block to which the first data belongs from multiple disk files; write the first data to the target disk file; wherein the first data includes: the data in the memory when the instruction is received.

[0056] The above-mentioned target disk file can be understood as a disk file of the data in the storage block that is about to be written into the memory; in actual implementation, it is considered that there are multiple storage blocks divided in the memory, and each storage block has a corresponding disk file, such as storage block No. 1 corresponds to disk file A, storage block No. 2 corresponds to disk file B, etc.; in the process of the first device updating the data in the memory according to the user data, if the dirty page flush instruction is triggered, it is necessary to determine the storage block to which the first data belongs based on the first data in the memory when the dirty page flush instruction is received, and then determine the target disk file corresponding to the storage block from multiple disk files, and then write the first data to the target disk file; for example, if when the dirty page flush instruction is received, the storage block to which the first data in the memory belongs is storage block No. 1, and the target disk file corresponding to storage block No. 1 is disk file A, then the data is written to disk file A.

[0057] Another data processing method provided by an embodiment of the present invention, when receiving user data, first updates the data in the memory according to the user data; during the updating process, if the instruction to write the data in the memory to the disk is triggered, the target disk file corresponding to the storage block to which the first data belongs is determined from multiple disk files; and the first data is written to the target disk file. In this method, during the updating process of the data in the memory, it is not necessary to lock the data page in the memory being updated, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are carried out simultaneously, avoiding data update blockage and improving data processing performance.

[0058] The embodiment of the present invention also provides another data processing method, which is applied to a second device running a disk; it is assumed that a first device runs a memory, such as Figure 4 As shown, the method comprises the following steps:

[0059] Step S402, receiving first data from the memory of the first device; wherein the first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received.

[0060] Step S404, writing the first data to the disk.

[0061] Another data processing method provided by an embodiment of the present invention receives first data in a memory of a first device and writes the first data to a disk. In the method, during the data update process in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are performed simultaneously, avoiding data update blocking and improving data processing performance.

[0062] The embodiment of the present invention also provides another data processing method, such as Figure 5 As shown, the method comprises the following steps:

[0063] Step S502: If a first instruction is received from the first device, the current logical sequence number of the redo log of the disk is recorded.

[0064] The above-mentioned first instruction can be understood as an instruction issued by a first device running a memory, which needs to write the first data in the memory to the disk; in actual implementation, if the second device receives the first instruction, it is usually necessary to first record the current logical sequence number of the redo log of the disk; in actual implementation, the logical sequence number can be recorded in a marker file, for example, when the current LSN of the redo log is 1024, 1024 can be recorded in a marker file.

[0065] Step S504, receiving first data from the memory of the first device; wherein the first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received.

[0066] Step S506, writing the first data to the disk.

[0067] Step S508, if the second device crashes during the process of writing the first data in the memory to the disk, and receives a second instruction from the first device, write the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0068] The above-mentioned second instruction can be understood as an instruction issued by the first device running with memory, which requires the recovery of data in the disk; in actual implementation, if the second device running with the disk crashes during the process of writing the first data in the memory to the disk, and the second device receives the second instruction issued by the first device to recover the data in the disk, considering that the data in the disk may be incomplete at the moment of crash, when recovering the data, the current data in the memory can be written to the disk according to the above-mentioned logical serial number.

[0069] Another data processing method provided by an embodiment of the present invention records the current logical sequence number of the redo log of the disk if a first instruction is received from a first device; receives first data in the memory of the first device and writes the first data to the disk; if the second device crashes during the writing process and receives a second instruction from the first device, writes the current data in the memory to the disk according to the logical sequence number; in this method, during the data update process in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are performed simultaneously, avoiding data update blocking and improving data processing performance.

[0070] The embodiment of the present invention also provides another data processing method, which is implemented on the basis of the above embodiment method; the method focuses on describing the specific process of writing the first data to the disk, which specifically corresponds to the following step S604; in the method, the memory is divided into multiple storage blocks; the disk stores multiple disk files; each disk file is used to store the update data of at least a part of the storage blocks in the multiple storage blocks; the storage blocks in the multiple storage blocks correspond to the disk files in the multiple disk files one by one. Figure 6 As shown, the method comprises the following steps:

[0071] Step S602, receiving first data from the memory of the first device; wherein the first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received.

[0072] Step S604, writing the first data into a target disk file among the multiple disk files; wherein the target disk file is: a disk file corresponding to the storage block to which the first data belongs among the multiple disk files.

[0073] Since each of the multiple storage blocks in the memory has a corresponding disk file, such as storage block 1 corresponds to disk file A, storage block 2 corresponds to disk file B, etc.; when the first data in the memory needs to be updated to the target disk file, it is necessary to determine the storage block to which it belongs based on the first data, and then determine the target disk file corresponding to the storage block from multiple disk files, and then write the first data to the target disk file. For example, if the storage block to which the data in the memory belongs is storage block 1 when the dirty page flush instruction is received, and the target disk file corresponding to storage block 1 is disk file A, then the data will be written to disk file A.

[0074] Another data processing method provided by an embodiment of the present invention receives first data in the memory of a first device; writes the first data to a target disk file among multiple disk files; in this method, during the data update process in the memory, it is not necessary to lock the data page in the memory being updated, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are carried out simultaneously, avoiding data update blockage and improving data processing performance.

[0075] To further understand the above embodiments, the following is provided: Figure 7 A schematic diagram of a data processing method is shown in FIG. Figure 7 It includes a memory cache area and a disk storage area. In the memory cache area, the memory is divided into 10 storage blocks, namely storage block 1, storage block 2, and storage block 10, that is, the data is cached in blocks in the memory; in the disk cache area, multiple disk files are stored, and the storage blocks correspond to the disk files one by one. In actual implementation, in order to further ensure the integrity of the data in the disk file corresponding to each storage block, multiple disk files can also correspond to the same storage block. Figure 7 In the example, two disk files form a disk file group, and one disk file group corresponds to one storage block. Therefore, Figure 7 There are 10 disk file groups stored in the disk, such as disk file group No. 1, disk file group No. 2, up to disk file group No. 10, each disk file group includes two disk files, disk file A and disk file B. Each disk file group can be provided with an update record, and the update record stores the file identifier of the disk file whose data was most recently updated. For example, for disk file group No. 1 corresponding to storage block No. 1, if the disk file whose data was most recently updated stored in the update record is disk file A.

[0076] In actual implementation, it is usually necessary to first create a data table according to user instructions. The size of a single table space of the data table can be specified according to requirements, such as 2M, which is generally 1M by default, and a disk file can be created. When the user updates the data, the memory first receives the user's updated data and saves it to the memory. Then, the background process periodically flushes dirty pages to update the data in the memory to the disk. The following is an introduction to the dirty page flushing process:

[0077] like Figure 7 As shown, taking the storage block No. 1 in the memory and the target disk file being the disk file A of the disk file group No. 1 as an example, after refreshing the data block No. 1 in the cache, the first step is to record the LSN of the redo log to a mark file. Usually, a group of disk files can correspond to one mark file. Since the target disk file is the first disk file in the disk file group No. 1, that is, disk file A, the record position in the mark file is the first line; the second step is to update the data in the memory to the disk. Specifically, the data in each data block in the memory data is updated to the target disk file corresponding to the data block. For storage block No. 1, the data in storage block No. 1 is updated to disk file A in the disk file group No. 1; the third step is to update the file identifier of the disk file that last refreshed the data in the memory, that is, disk file A, to record the disk file that last refreshed the data.

[0078] If the second device running the disk crashes while writing the first data in the memory to the disk, the data in the disk may be incomplete. When restoring the data in the disk, the current data in the memory can be written to the disk according to the above logical sequence number.

[0079] In the prior art, for example, the current LSN of the recorded redo log is 2048, and the data in the memory has been updated to 5000. When the data in the memory needs to be written to the disk, for the data page in the storage block, it is necessary to lock the data page, and then write the data from 2048 to 5000 to the disk. After the data of the data page is written to the disk, the data page is unlocked. At this time, the data page can continue to receive user data and update data, which is equivalent to the two processes of flushing dirty pages and updating memory data can only be performed serially. In this solution, when it is necessary to flush dirty pages, the current LSN of the redo log of the disk is recorded in advance. For the data page in the storage block, it is not necessary to lock the data page. In the process of flushing dirty pages, the data in the data page can be updated according to the user data at the same time without waiting, thereby avoiding data update blocking and improving data processing performance. Even if the second device running the disk crashes, it only needs to restore the data in the disk according to the recorded LSN.

[0080] An embodiment of the present invention provides a structural schematic diagram of a data processing device, wherein the device is arranged in a first device running a memory; a disk is arranged in a second device, such as Figure 8 As shown, the device includes: an update module 80, which is used to receive user data and update the data in the memory according to the user data; a first writing module 81, which is used to write the first data in the memory to the disk if an instruction to write the data in the memory to the disk is triggered during the process of updating the data in the memory; wherein the first data includes: the data in the memory when the instruction is received.

[0081] A data processing device provided by an embodiment of the present invention updates data in a memory according to the user data after receiving user data; during the updating process, if an instruction to write the data in the memory to a disk is triggered, the first data in the memory is written to the disk; wherein the first data includes: the data in the memory when the instruction is received. In the device, during the updating process of the data in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the updating of the data in the memory and the writing of the data in the memory to the disk are performed simultaneously, thus avoiding data update blockage and improving data processing performance.

[0082] Furthermore, the device is also used to: record the current logical sequence number of the redo log of the disk; if the second device crashes during the process of writing the first data in the memory to the disk, write the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0083] Furthermore, the memory is divided into multiple storage blocks; multiple disk files are stored in the disk; each disk file is used to store updated data of at least a part of the multiple storage blocks; the first writing module 81 is also used to: determine the target disk file corresponding to the storage block to which the first data belongs from the multiple disk files; and write the first data to the target disk file.

[0084] Furthermore, the storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

[0085] The data processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data processing method embodiment. For the sake of brief description, for matters not mentioned in the data processing device embodiment, reference may be made to the corresponding contents in the aforementioned data processing method embodiment.

[0086] The embodiment of the present invention provides a structural schematic diagram of another data processing device, the device is set in a second device running a disk; it is preset that a first device runs a memory, such as Fig. 9As shown, the apparatus includes: a receiving module 90, for receiving first data in a memory from a first device; wherein the first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received; a second writing module 91, for writing the first data to the disk.

[0087] Another data processing device provided by an embodiment of the present invention receives first data in a memory of a first device and writes the first data to a disk. In the device, during the data update process in the memory, the data page in the memory being updated does not need to be locked, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are performed simultaneously, thereby avoiding data update blocking and improving data processing performance.

[0088] Furthermore, the device is also used for: if a first instruction is received from a first device, recording the current logical sequence number of the redo log of the disk; if the second device crashes during the process of writing the first data in the memory to the disk, and receives a second instruction from the first device, writing the current data in the memory to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

[0089] Furthermore, the memory is divided into a plurality of storage blocks; the disk stores a plurality of disk files; each disk file is used to store update data of at least a portion of the storage blocks in the plurality of storage blocks;

[0090] The second writing module 91 is further used to write the first data to a target disk file among the multiple disk files; wherein the target disk file is a disk file corresponding to the storage block to which the first data belongs in the multiple disk files.

[0091] Furthermore, the storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

[0092] The data processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data processing method embodiment. For the sake of brief description, for matters not mentioned in the data processing device embodiment, reference may be made to the corresponding contents in the aforementioned data processing method embodiment.

[0093] The embodiment of the present invention also provides a structural diagram of a data processing system, such as Fig.10As shown, the system includes a first device 100 running a memory and a second device 101 running a disk; wherein the first device 100 and the second device 101 can be the same device or different devices.

[0094] The first device 100 is used to: receive user data, and update the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, send the first data in the memory to the second device; wherein the first data includes: the data in the memory when the instruction is received;

[0095] The second device 101 is used to: receive first data from the memory of the first device; and write the first data to a disk.

[0096] An embodiment of the present invention provides a data update system, the system includes a first device running a memory and a second device running a disk; after receiving user data, the first device updates the data in the memory according to the user data; during the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first device sends the first data in the memory to the second device, and the second device writes the first data to the disk. In this system, during the process of updating the data in the memory, it is not necessary to lock the data page in the memory being updated, and the data in the memory can be directly written to the disk, so that the two processes of updating the data in the memory and writing the data in the memory to the disk are carried out simultaneously, avoiding data update blockage and improving data processing performance.

[0097] The embodiment of the present invention further provides an electronic device, see Fig.11 As shown, the electronic device includes a processor 130 and a memory 131 , wherein the memory 131 stores machine executable instructions that can be executed by the processor 130 , and the processor 130 executes the machine executable instructions to implement the above data processing method.

[0098] Further, Fig.11 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0099] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0100] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0101] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned data processing method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0102] The computer program products of the data processing methods, devices, systems and electronic devices provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0103] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0104] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method is applied to a first device running a memory; It is assumed that a disk is running in the second device, and the method includes: receiving user data, and updating data in the memory according to the user data; In the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is written to the disk; wherein the first data includes: the data in the memory when the instruction is received; Before the step of writing the first data in the memory to the disk, the method further includes: recording a current logical sequence number of a redo log of the disk; the redo log is used to record address information of data refresh before refreshing the data in the memory to the disk; the logical sequence number is represented by LSN; The method also includes: if the second device crashes during the process of writing the first data in the memory to the disk, writing the current data in the memory to the disk according to the logical serial number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

2. The method according to claim 1, characterized in that: The memory is divided into a plurality of storage blocks; the disk stores a plurality of disk files; each of the disk files is used to store update data of at least a portion of the storage blocks; The step of writing the first data in the memory to the disk includes: determining a target disk file corresponding to the storage block to which the first data belongs from the multiple disk files; The first data is written to the target disk file.

3. The method according to claim 2, characterized in that The storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

4. A data processing method, characterized in that: The method is applied to a second device running a disk; It is assumed that a memory is running in the first device, and the method includes: receiving first data in the memory from the first device; The first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received; Writing the first data to the disk; Before the step of receiving the first data in the memory from the first device, the method further comprises: if a first instruction from the first device is received, recording a current logical sequence number of a redo log of the disk; the redo log is used to record address information of data refresh before refreshing the data in the memory to the disk; the logical sequence number is represented by LSN; The method also includes: if the second device crashes during the process of writing the first data in the memory to the disk and receives a second instruction from the first device, the current data in the memory is written to the disk according to the logical serial number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

5. The method according to claim 4, characterized in that The memory is divided into a plurality of storage blocks; the disk stores a plurality of disk files; Each of the disk files is used to store update data of at least a portion of the storage blocks in the plurality of storage blocks; The step of writing the first data to the disk includes: writing the first data to a target disk file among the multiple disk files; wherein the target disk file is: a disk file corresponding to a storage block to which the first data belongs among the multiple disk files.

6. The method according to claim 5, characterized in that The storage blocks in the plurality of storage blocks correspond one-to-one to the disk files in the plurality of disk files.

7. A data processing device, characterized in that: The apparatus is arranged on a first device running a memory; It is assumed that a disk is running in the second device, and the apparatus comprises: An updating module, used for receiving user data and updating the data in the memory according to the user data; A first writing module is used to write the first data in the memory to the disk if an instruction to write the data in the memory to the disk is triggered during the process of updating the data in the memory; wherein the first data includes: the data in the memory when the instruction is received; The device is also used to: record the current logical sequence number of the redo log of the disk; If the second device crashes during the process of writing the first data in the memory to the disk, the current data in the memory is written to the disk according to the logical sequence number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

8. A data processing device, characterized in that: The device is arranged on a second device running a disk; It is assumed that a memory is running in the first device, and the apparatus includes: A receiving module, configured to receive first data from the memory of the first device; The first data is sent by the first device in the following manner: the first device receives user data and updates the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, the first data in the memory is sent to the second device, and the first data includes: the data in the memory when the instruction is received; A second writing module, used for writing the first data to the disk; The device is also used for: If a first instruction from the first device is received, the current logical sequence number of the redo log of the disk is recorded; the redo log is used to record address information of data refresh before refreshing the data in the memory to the disk; the logical sequence number is represented by LSN; If the second device crashes during the process of writing the first data in the memory to the disk and receives a second instruction from the first device, the current data in the memory is written to the disk according to the logical serial number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

9. A data processing system, characterized in that: The system includes a first device running a memory and a second device running a disk; The first device is used to: receive user data, and update the data in the memory according to the user data; in the process of updating the data in the memory, if an instruction to write the data in the memory to the disk is triggered, send the first data in the memory to the second device; wherein the first data includes: the data in the memory when the instruction is received; The second device is used to: receive the first data in the memory from the first device; and write the first data to the disk; Before the step of receiving the first data in the memory from the first device, the second device is further used to: if receiving the first instruction from the first device, record the current logical sequence number of the redo log of the disk; the redo log is used to record the address information of the data refresh before refreshing the data in the memory to the disk; the logical sequence number is represented by LSN; The second device is also used for: if the second device crashes during the process of writing the first data in the memory to the disk, and receives a second instruction from the first device, writing the current data in the memory to the disk according to the logical serial number; wherein the data volume of the current data is greater than or equal to the data volume of the first data.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data processing method according to any one of claims 1 to 6.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the data processing method described in any one of claims 1 to 6.

Images