Data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present invention relate to a data processing method, device, electronic device, and storage medium, comprising: when data to be written is acquired, writing the data to be written into a memory queue established in the device memory, and when it is determined that the data in the memory queue meets the set disk write conditions, moving the data in the memory queue to the device disk; when a data dequeue instruction is received, if it is determined that data exists on the device disk, reading the data from the device disk; if it is determined that data does not exist on the device disk and data exists in the memory queue, dequeuing the data in the memory queue. In this way, the memory queue can be expanded by using a disk, which can break through the capacity limitations of using a single memory queue for data storage.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] A memory queue is a queue created in device memory. Like regular queues, it supports both queue entry and exit operations and adheres to the first-in, first-out principle of data processing. In practice, memory queues are widely applicable in many scenarios due to their flexibility and ease of use.

[0003] However, the storage capacity of the memory queue is limited. When a large amount of data or messages needs to be stored, the memory queue storage capacity will be limited. Summary of the Invention

[0004] In view of this, in order to solve the technical problem of limited storage capacity of memory queues in the scenario of applying memory queues in the prior art, embodiments of the present invention provide a data processing method, device, electronic device and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] When data to be written is obtained, the data to be written is written into a memory queue established in the device memory, and when it is determined that the data in the memory queue meets the set disk writing conditions, the data in the memory queue is moved to the device disk;

[0007] When a data dequeue instruction is received, if it is determined that data exists in the device disk, the data is read from the device disk; if it is determined that data does not exist in the device disk and data exists in the memory queue, the data in the memory queue is dequeued.

[0008] In one possible implementation, writing the data to be written into a memory queue established in a device memory includes:

[0009] Writing the data to be written into a first memory queue established in the device memory;

[0010] When it is determined that the data in the memory queue meets the set disk-writing condition, the data in the memory queue is moved to the device disk, including:

[0011] When it is determined that the data in the first memory queue meets a set condition, marking the first memory queue as a second memory queue, and moving the data in the second memory queue to a device disk;

[0012] The method further comprises:

[0013] After marking the first memory queue as the second memory queue, a new first memory queue is created in the device memory.

[0014] In a possible implementation, after writing the data to be written into a first memory queue established in a device memory, the method further includes:

[0015] Updating the value of the first attribute of the first memory queue to the sum of the current value of the first attribute and the amount of the data to be written, where the first attribute is used to represent the current amount of data in the memory queue;

[0016] The determining whether the data in the first memory queue meets a set disk flushing condition includes:

[0017] When it is determined that the value of the first attribute of the first memory queue reaches a set threshold, it is determined that the data in the first memory queue meets a set disk write condition.

[0018] In one possible implementation, moving the data in the second memory queue to a device disk includes:

[0019] Encapsulating the second memory queue into a file;

[0020] Write said file to the device disk.

[0021] In one possible implementation, the method further includes:

[0022] Setting the file index value of the file to the current value of the first metadata, where the first metadata is used to represent the file index value that needs to be occupied when the data is written to the disk;

[0023] After writing the file to the device disk, adding the file index value of the file to the file index value set, and updating the value of the first metadata to the sum of the current value of the first metadata and 1;

[0024] The step of determining whether data exists on the device disk includes:

[0025] Determine whether the file index value set is empty;

[0026] When it is determined that the file index value set is not empty, it is determined that data exists in the device disk.

[0027] In one possible implementation, reading data from a device disk includes:

[0028] Determine a file on the device disk whose file index value is the current value of the second metadata as a target file, and determine the current value of the third metadata as a target index value, wherein the second metadata is used to represent the file index value of the currently dequeued file, and the third metadata is used to represent the index value of the next dequeued data;

[0029] Read data whose subscript value is the target subscript value from the target file.

[0030] In a possible implementation, reading data whose subscript value is the target subscript value from the target file includes:

[0031] When reading data from the target file for the first time, the target file is loaded from the device disk into the device memory to obtain a corresponding backup memory queue; and data whose index value in the backup memory queue is the target index value is dequeued;

[0032] When data is not read from the target file for the first time, data with an index value of the target index value in the backup memory queue is dequeued.

[0033] In a possible implementation, after dequeuing the data whose index value is the target index value from the backup memory queue, the method further includes:

[0034] Updating the value of the second attribute of the backup memory queue to a value obtained by subtracting 1 from the current value of the second attribute, where the second attribute is used to represent the amount of data in the memory queue;

[0035] When it is determined that the current value of the second attribute of the backup memory queue is not 0, the value of the third metadata is updated to the value of the current value of the third metadata plus 1; when it is determined that the current value of the second attribute of the backup memory queue is 0, the value of the third metadata is updated to the corresponding initial value, the value of the second metadata is updated to the value of the current value of the second metadata plus 1, and the current value of the second metadata is deleted from the file index value set.

[0036] In one possible implementation, dequeuing the data in the memory queue includes:

[0037] Determining whether the second memory queue exists in the device memory;

[0038] When it is determined that the second memory queue exists in the device memory, dequeueing the data in the second memory queue;

[0039] When it is determined that the second memory queue does not exist in the device memory, determining whether the first memory queue exists in the device memory;

[0040] When it is determined that the first memory queue exists in the device memory, data in the first memory queue is dequeued.

[0041] In one possible implementation, dequeuing the data in the second memory queue includes:

[0042] Backing up the second memory queue to obtain a corresponding backup memory queue;

[0043] Dequeueing the data in the backup memory queue;

[0044] Dequeuing the data in the first memory queue includes:

[0045] Creating a new backup memory queue in the device memory, and soft-linking the address of the first memory queue to the backup memory queue;

[0046] Dequeue the data in the backup memory queue.

[0047] In a second aspect, an embodiment of the present invention provides a data processing device, including:

[0048] The writing module is used to write the data to be written into the memory queue established in the device memory when the data to be written is obtained;

[0049] A disk write module, configured to move the data in the memory queue to the device disk when it is determined that the data in the memory queue meets a set disk write condition;

[0050] The reading module is used to, when receiving a data dequeue instruction, read the data from the device disk if it is determined that there is data in the device disk; if it is determined that there is no data in the device disk and it is determined that there is data in the memory queue, dequeue the data in the memory queue.

[0051] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a data processing program stored in the memory to implement the data processing method described in any one of the first aspects.

[0052] In a fourth aspect, an embodiment of the present invention provides a storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method described in any one of the first aspects.

[0053] The technical solution provided by the embodiment of the present invention is to write the data to be written into the memory queue established in the device memory when the data to be written is obtained, and to move the data in the memory queue to the device disk when it is determined that the data in the memory queue meets the set disk write conditions. This realizes that when the data in the memory queue reaches the upper limit of the memory queue, the data in the memory queue is moved to the device disk, thereby realizing the expansion of the memory queue by using the disk, which can break through the capacity limitation of using only the memory queue for data storage. When a data dequeue instruction is received, if it is determined that the device disk has data, the data is read from the device disk. If it is determined that the device disk does not have data and it is determined that the memory queue has data, the data in the memory queue is dequeued. Since the data in the device disk is queued first, when the data exists in the device disk, the data is read from the device disk first. After the data in the device disk is read, the data is read from the memory queue. This realizes the expansion of the memory queue by using the disk while meeting the queue's first-in-first-out data processing principle. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of an embodiment of a data processing method provided by an embodiment of the present invention;

[0055] Figure 2 A block diagram of a data processing device according to an embodiment of the present invention;

[0056] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] The data processing method provided by the present invention is explained below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0059] See also Figure 1 , is a flow chart of an embodiment of a data processing method provided by an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0060] Step 101: When data to be written is obtained, the data to be written is written into a memory queue established in a device memory.

[0061] Step 102: When it is determined that the data in the memory queue meets the set disk-writing conditions, the data in the memory queue is moved to the device disk.

[0062] The following is a unified description of step 101 and step 102:

[0063] The memory queue in the embodiment of the present invention refers to a common memory queue, that is, a queue established in the device memory. The memory queue is the same as a queue in the usual sense, can support queue entry operations, can also support queue exit operations, and meets the first-in-first-out data processing principle.

[0064] Furthermore, the memory queue in the embodiment of the present invention has two attributes, which are respectively recorded as a first attribute (Memsize) and a second attribute (ElemNum) for the convenience of description. Among them, the first attribute is used to characterize the current data size of the data in the memory queue, and the second attribute is used to characterize the number of data in the memory queue. It is understandable that when the memory queue is enqueued or dequeued, the values ​​of the first attribute and the second attribute also change accordingly.

[0065] Specifically, after an enqueue operation is performed on a memory queue, that is, after data is written to the memory queue, the value of its first attribute will be updated to the sum of the current value of the first attribute and the amount of the most recently written data, and the value of the second attribute will be updated to the current value of the second attribute plus 1. After a dequeue operation is performed on a memory queue, the value of its first attribute will be updated to the difference between the current value of the first attribute and the amount of the most recently dequeued data, and the value of the second attribute will be updated to the current value of the second attribute minus 1.

[0066] In addition, the data in the memory queue can have a subscript value to represent the order in which the data is enqueued.

[0067] For example, after the first data (assuming the data size is 2KB) is written into the memory queue, the first attribute value of the memory queue will be updated to 2KB, the second attribute value will be updated to 1, and the subscript value of the first data will be 1. After the second data (assuming the data size is 4KB) is written into the memory queue, the first attribute value of the memory queue will be updated to 6KB, the second attribute value will be updated to 2, and the subscript value of the second data will be 2. After the first enqueued data is dequeued, the first attribute value of the memory queue will be updated to 4KB, and the second attribute value will be updated to 1.

[0068] The above is an explanation of the memory queue provided by the embodiment of the present invention. The following is an explanation of the specific implementation of step 101 and step 102:

[0069] As can be seen from the description of step 101 and step 102, in the embodiment of the present invention, a memory queue and a disk are combined for data storage, so that the memory queue can be expanded by using a disk, breaking through the capacity limitation of using only a memory queue for data storage.

[0070] Specifically, as an embodiment, when data to be written is acquired, the data to be written is written to a first memory queue. When it is determined that the data in the first memory queue meets the set disk write conditions, the first memory queue is marked as a second memory queue, and the data in the second memory queue is moved to the device disk. Simultaneously, a new first memory queue is created in the device memory.

[0071] As can be seen, in this embodiment of the present invention, a first memory queue and a second memory queue are maintained in the device memory. The first memory queue is specifically used to enqueue data to be written, and the second memory queue is specifically used to write data to disk. This allows writing data in the first memory queue to disk and enqueuing data to be written to disk to be processed in parallel, which improves data processing performance compared to asynchronous processing.

[0072] Optionally, the above-mentioned disk flushing condition includes: the value of the first attribute of the first memory queue reaches a set threshold (for example, 16 KB), that is, the amount of data in the first memory queue reaches the set threshold.

[0073] In one embodiment, moving the data in the second memory queue to the device disk includes packaging the second memory queue into a file and writing the file to the device disk. Thus, in this embodiment, the second memory queue can be moved to the device disk as a whole. This improves disk write performance and is faster than dequeuing the data in the second memory queue and moving them to the device disk one by one.

[0074] Step 103: When a data dequeue instruction is received, if it is determined that data exists in the device disk, the data is read from the device disk; if it is determined that data does not exist in the device disk and data exists in the memory queue, the data in the memory queue is dequeued.

[0075] As can be seen from the description of steps 101 and 102 above, the data on the device disk is obtained by writing the data in the memory queue to disk. Therefore, the data on the device disk is first enqueued. According to the first-in, first-out data processing principle, when it is determined that data exists on the device disk, the data is first read from the device disk. When it is determined that the data does not exist on the device disk and that data exists in the memory queue, the data in the memory queue is then dequeued. In addition, when it is determined that data does not exist on both the device disk and the memory queue, feedback information indicating that there is no data to dequeue can be provided to the user.

[0076] The following describes in detail how to read data from the device disk:

[0077] Before describing how to read data from a device disk, the metadata file provided by an embodiment of the present invention is first introduced. The metadata file is a type of statistical identification data designed to manage a large amount of queue data.

[0078] In one embodiment, the metadata file includes the following four identifiers: first metadata (NowEnqueueFileNum), second metadata (NowDequeueFileNum), third metadata (NowDeQueueElemIndex), and a file index value set (FileList). The first metadata is used to indicate the file index value that will be occupied when data is written to the disk, and its initial value is 1; the second metadata is used to indicate the file index value of the currently dequeued file, and its initial value is 1; the third metadata is used to indicate the index value of the next dequeued data, and its initial value is 1; the file index value set is used to record the file index values ​​of files stored on the device disk, and is initially empty.

[0079] Based on the above metadata file, in order to facilitate understanding of the specific implementation of step 103, it is necessary to further explain that in the above step 102, when the file is written to the disk, the file index value of the file can be set to the current value of the first metadata, and after the file is written to the disk, the file index value of the file is added to the file index value set, and the value of the first metadata is updated to the sum of the current value of the first metadata and 1.

[0080] For example, when the file is written to disk for the first time, the file index value of the file is set to the current value of the first metadata, specifically 1. After file 1 (that is, the file with a file index value of 1) is written to disk, the file index value 1 of file 1 is added to the file index value set, and the value of the first metadata is updated to 2. In this way, when the file is written to disk for the second time, the file index value of the file is set to the current value of the first metadata, specifically 2.

[0081] Based on this, as an embodiment, determining whether there is data in the device disk includes: determining whether the file index value set is empty. When it is determined that the file index value set is empty, it can be determined that there is no data in the device disk. When it is determined that the file index value set is not empty, it can be determined that there is data in the device disk.

[0082] Furthermore, when reading data from the device disk, the first-in-first-out data processing principle is followed, and the data written earlier is read from the file written earlier. Specifically, the file on the device disk whose file index value is the current value of the second metadata is determined as the target file, and the current value of the third metadata is determined as the target index value. The data with the index value of the target index value is read from the target file.

[0083] Among them, when reading data with an index value of the target index value from the target file, if this is the first time to read data from the target file, the target file is loaded from the device disk to the device memory, and the corresponding backup memory queue is obtained, and the data with an index value of the target index value in the backup memory queue is dequeued; if this is not the first time to read data from the target file, the data with an index value of the target index value in the backup memory queue can be dequeued.

[0084] Furthermore, after reading data with an index value equal to the target index value from the backup memory queue, the value of the second attribute of the backup memory queue is updated to a value obtained by subtracting 1 from the current value of the second attribute.

[0085] Afterwards, when it is determined that the value of the second attribute of the backup memory queue is not 0, the value of the third metadata is updated to the value of the current value of the third metadata plus 1; when it is determined that the value of the second attribute of the backup memory queue is 0, the value of the third metadata is updated to the corresponding initial value, the value of the second metadata is updated to the value of the current value of the second metadata plus 1, and the current value of the second metadata is deleted from the file index value set.

[0086] For example, assume that file 1 and file 2 are written to disk one after another, and that file 2 contains three pieces of data. The subscript values ​​of these three pieces of data are 1, 2, and 3 in the order in which they are written. Also, assume that the current metadata file is as follows:

[0087] FileList:[1,2];

[0088] NowDequeueFileNum: 1;

[0089] NowDeQueueElemIndex: 1;

[0090] NowEnqueueFileNum: 3.

[0091] Based on the above metadata file, when the data dequeue instruction is received for the first time, it is determined that FileList is not empty, that is, it is determined that there is data on the device disk. According to the above description, the file with a file index value of 1, that is, file 1, is determined as the target file. Since it is the first time to read data from file 1, file 1 is loaded from the device disk to the device memory, and the corresponding backup memory queue is obtained. The data with an index value of 1 in the backup memory queue is dequeued. At this time, the second attribute value of the backup memory queue is updated from 3 to 2, and the updated metadata file is as follows:

[0092] FileList:[1,2];

[0093] NowDequeueFileNum: 1;

[0094] NowDeQueueElemIndex: 2;

[0095] NowEnqueueFileNum: 3.

[0096] When the second data dequeue instruction is received, it is determined that FileList is not empty, that is, that data exists on the device disk. According to the above description, the file with a file index value of 1, that is, file 1, is determined as the target file. Since this is not the first time data is read from file 1, the data with an index value of 2 in the backup memory queue is directly dequeued. At this time, the second attribute value of the backup memory queue is updated from 2 to 1. The updated metadata file is as follows:

[0097] FileList:[1,2];

[0098] NowDequeueFileNum: 1;

[0099] NowDeQueueElemIndex: 2;

[0100] NowEnqueueFileNum: 3.

[0101] When the third data dequeue instruction is received, it is determined that FileList is not empty, that is, it is determined that there is data on the device disk. According to the above description, the file with a file index value of 1, that is, file 1, is determined as the target file. Since this is not the first time data is read from file 1, the data with an index value of 3 in the backup memory queue is directly dequeued. At this time, the second attribute value of the backup memory queue is updated from 1 to 0. The updated metadata file is as follows:

[0102] FileList:【2】;

[0103] NowDequeueFileNum:2;

[0104] NowDeQueueElemIndex: 1;

[0105] NowEnqueueFileNum: 3.

[0106] When the data dequeue instruction is received for the fourth time, it is determined that FileList is not empty, that is, it is determined that there is data in the device disk. According to the above description, the file with a file index value of 2, that is, file 2, is determined as the target file. Then, according to the above description, data is read from file 2 according to the first-in-first-out principle. As for how to read data from file 2, please refer to the above description of the process of reading data from file 1, which will not be described in detail here.

[0107] When the data in file 2 is also read, the metadata file is as follows:

[0108] FileList:【】;

[0109] NowDequeueFileNum:3;

[0110] NowDeQueueElemIndex: 1;

[0111] NowEnqueueFileNum: 3.

[0112] Afterwards, if a data dequeue instruction is received again and it is determined that FileList is empty, that is, it is determined that there is no data in the device disk, then according to the description in step 103, it is determined that there is no data in the device disk and that there is data in the memory queue, and the data in the memory queue is dequeued.

[0113] The following describes how to dequeue the data in the memory queue:

[0114] First, since the data in the second memory queue is written earlier than the data in the first memory queue, when it is determined that the second memory queue exists in the device memory, the data in the second memory queue is first dequeued. When it is determined that the second memory queue does not exist in the device memory and it is determined that the first memory queue exists in the device memory, the data in the first memory queue is dequeued.

[0115] Specifically, when it is determined that a second memory queue exists in the device memory, the second memory queue is backed up to obtain a corresponding backup memory queue, and then the data in the backup memory queue is dequeued. This process allows data to be written to the disk and dequeued in parallel, improving data processing performance.

[0116] Similarly, when it is determined that the first memory queue exists in the device memory, a backup memory queue can be established in the device memory, the address of the first memory queue is soft-linked to the backup memory queue, and the data in the backup memory queue is dequeued. In this way, the data in the first memory queue is dequeued.

[0117] As for the specific operation of dequeuing data in the memory queue, this application will not go into details.

[0118] Furthermore, as can be seen from the above description, metadata files are crucial in the data processing methods provided by the embodiments of the present invention. Therefore, the embodiments of the present invention propose periodically backing up the metadata files in the device memory to the device disk. For example, every second, the metadata files in the device memory are synchronously refreshed to the corresponding files on the device disk. This allows the metadata files to be restored from the device disk even if a process crashes and exits.

[0119] Based on this, in an embodiment of the present invention, when initializing a metadata file, the device disk is first checked to see if there is a corresponding metadata file. If so, the metadata therein is immediately loaded into the device memory. If not, a metadata file is created. The content of the newly created metadata file is as follows:

[0120] FileList:【】;

[0121] NowDequeueFileNum: 1;

[0122] NowDeQueueElemIndex: 1;

[0123] NowEnqueueFileNum: 1.

[0124] At the same time, a background thread is started to be responsible for the timing task, which synchronizes the metadata file in the device memory to the corresponding file in the device disk every 1 second.

[0125] The technical solution provided by the embodiment of the present invention is to write the data to be written into the memory queue established in the device memory when the data to be written is obtained, and to move the data in the memory queue to the device disk when it is determined that the data in the memory queue meets the set disk write conditions. This realizes that when the data in the memory queue reaches the upper limit of the memory queue, the data in the memory queue is moved to the device disk, thereby realizing the expansion of the memory queue by using the disk, which can break through the capacity limitation of using only the memory queue for data storage. When a data dequeue instruction is received, if it is determined that the device disk has data, the data is read from the device disk. If it is determined that the device disk does not have data and it is determined that the memory queue has data, the data in the memory queue is dequeued. Since the data in the device disk is queued first, when the data exists in the device disk, the data is read from the device disk first. After the data in the device disk is read, the data is read from the memory queue. This realizes the expansion of the memory queue by using the disk while meeting the queue's first-in-first-out data processing principle.

[0126] Corresponding to the aforementioned embodiments of the data processing method, the present invention further provides embodiments of a data processing device.

[0127] See also Figure 2 , is a block diagram of an embodiment of a data processing device provided by an embodiment of the present invention. Figure 2 As shown, the device includes:

[0128] The writing module 21 is configured to write the data to be written into a memory queue established in the device memory when the data to be written is acquired;

[0129] The disk write module 22 is configured to move the data in the memory queue to the device disk when it is determined that the data in the memory queue meets the set disk write conditions;

[0130] The reading module 23 is used to, when receiving a data dequeue instruction, read the data from the device disk if it is determined that there is data in the device disk; if it is determined that there is no data in the device disk and there is data in the memory queue, dequeue the data in the memory queue.

[0131] In a possible implementation, the writing module 21 is specifically configured to: write the data to be written into a first memory queue established in a device memory;

[0132] The disk migration module 22 is specifically configured to: when it is determined that the data in the first memory queue meets a set condition, mark the first memory queue as a second memory queue and move the data in the second memory queue to the device disk;

[0133] The device also includes (not shown in the figure):

[0134] A new queue module is used to create a new first memory queue in the device memory after marking the first memory queue as the second memory queue.

[0135] In one possible embodiment, the apparatus further includes (not shown in the figure): a first updating module, configured to, after writing the data to be written into a first memory queue established in the device memory, update the value of a first attribute of the first memory queue to the sum of a current value of the first attribute and the amount of the data to be written, wherein the first attribute is used to represent the current amount of data in the memory queue;

[0136] The disk placement module 22 is specifically configured to: when it is determined that the value of the first attribute of the first memory queue reaches a set threshold, determine that the data in the first memory queue meets a set disk placement condition.

[0137] In one possible implementation, the tray placement module 22 includes:

[0138] An encapsulation submodule, configured to encapsulate the second memory queue into a file;

[0139] The write disk module is used to write the file to the device disk.

[0140] In one possible embodiment, the device further includes (not shown in the figure):

[0141] a setting module, configured to set the file index value of the file to the current value of first metadata, where the first metadata is used to represent the file index value that needs to be occupied when the data is written to the disk;

[0142] An adding module, configured to add the file index value of the file to the file index value set after the file is written to the device disk;

[0143] A second updating module, configured to update the value of the first metadata to the sum of the current value of the first metadata and 1;

[0144] The disk write module 22 is specifically configured to: determine whether the file index value set is empty; and when it is determined that the file index value set is not empty, determine that data exists in the device disk.

[0145] In one possible implementation, the reading module 23 includes (not shown in the figure):

[0146] A first determining submodule is configured to determine a file in a device disk whose file index value is a current value of second metadata as a target file, where the second metadata is used to represent the file index value of a currently dequeued file;

[0147] A second determining submodule, configured to determine a current value of third metadata as a target index value, wherein the third metadata is used to represent an index value of next dequeued data;

[0148] The file reading submodule is used to read data with a subscript value of the target subscript value from the target file.

[0149] In one possible implementation, the file reading submodule is specifically configured to:

[0150] When reading data from the target file for the first time, the target file is loaded from the device disk into the device memory to obtain a corresponding backup memory queue; and data whose index value in the backup memory queue is the target index value is dequeued;

[0151] When data is not read from the target file for the first time, data with an index value of the target index value in the backup memory queue is dequeued.

[0152] In one possible embodiment, the device further includes (not shown in the figure):

[0153] a third updating module, configured to update the value of a second attribute of the backup memory queue to a value obtained by subtracting 1 from a current value of the second attribute after dequeuing the data having the subscript value of the target subscript value in the backup memory queue, wherein the second attribute is used to represent the number of data in the memory queue;

[0154] a fourth updating module, configured to update the value of the third metadata to a value obtained by adding 1 to the current value of the third metadata when determining that the current value of the second attribute of the backup memory queue is not 0;

[0155] a fifth updating module, configured to update the value of the third metadata to a corresponding initial value when determining that the current value of the second attribute of the backup memory queue is 0;

[0156] a sixth updating module, configured to, when determining that the current value of the second attribute of the backup memory queue is 0, update the value of the second metadata to a value obtained by adding 1 to the current value of the second metadata;

[0157] The deleting module is configured to delete the current value of the second metadata from the file index value set when determining that the current value of the second attribute of the backup memory queue is 0.

[0158] In a possible implementation, the reading module 23 includes (not shown in the figure):

[0159] a determination submodule, configured to determine whether the second memory queue exists in the device memory;

[0160] a dequeue submodule, configured to dequeue data in the second memory queue when it is determined that the second memory queue exists in the device memory;

[0161] The determining submodule is further configured to determine whether the first memory queue exists in the device memory when it is determined that the second memory queue does not exist in the device memory;

[0162] The dequeue submodule is further configured to dequeue the data in the first memory queue when it is determined that the first memory queue exists in the device memory.

[0163] In one possible implementation, the dequeue submodule is specifically configured to:

[0164] Backing up the second memory queue to obtain a corresponding backup memory queue; dequeuing data in the backup memory queue;

[0165] In one possible implementation, the dequeue submodule is specifically configured to:

[0166] A backup memory queue is newly created in the device memory, and the address of the first memory queue is soft-linked to the backup memory queue; and data in the backup memory queue is dequeued.

[0167] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 3 The electronic device 300 shown includes: at least one processor 301, memory 302, at least one network interface 304 and other user interfaces 303. The various components in the electronic device 300 are coupled together via a bus system 303. It is understood that the bus system 305 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 305 is not shown in FIG. Figure 3 Various buses are labeled as bus system 305 .

[0168] The user interface 303 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0169] It is understood that the memory 302 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 302 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0170] In some embodiments, the memory 302 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 3021 and application programs 3022 .

[0171] The operating system 3021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 3022 include various application programs, such as a media player (MediaPlayer) and a browser (Browser), for implementing various application services. Programs implementing the methods of the embodiments of the present invention may be included in application programs 3022.

[0172] In an embodiment of the present invention, by calling a program or instruction stored in the memory 302, specifically, a program or instruction stored in the application 3022, the processor 301 is configured to execute the method steps provided in each method embodiment, for example, including:

[0173] When data to be written is obtained, the data to be written is written into a memory queue established in the device memory, and when it is determined that the data in the memory queue meets the set disk writing conditions, the data in the memory queue is moved to the device disk;

[0174] When a data dequeue instruction is received, if it is determined that data exists in the device disk, the data is read from the device disk; if it is determined that data does not exist in the device disk and data exists in the memory queue, the data in the memory queue is dequeued.

[0175] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 301. Processor 301 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 301 or by software instructions. The above processor 301 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 302 , and the processor 301 reads the information in the memory 302 and completes the steps of the above method in combination with its hardware.

[0176] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0177] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0178] The electronic device provided in this embodiment may be Figure 3 The electronic device shown in FIG. 1 can perform the following operations: Figure 1 All steps of the data processing method in Figure 1 For details on the technical effects of the data processing method shown, please refer to Figure 1 For the sake of brevity, the relevant description will not be repeated here.

[0179] An embodiment of the present invention further provides a storage medium (computer-readable storage medium). The storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and the memory may also include a combination of the aforementioned types of memory.

[0180] When one or more programs in the storage medium can be executed by one or more processors, the data processing method executed on the electronic device side can be implemented.

[0181] The processor is configured to execute a data processing program stored in the memory to implement the following steps of a data processing method executed on the electronic device side:

[0182] When data to be written is obtained, the data to be written is written into a memory queue established in the device memory, and when it is determined that the data in the memory queue meets the set disk writing conditions, the data in the memory queue is moved to the device disk;

[0183] When a data dequeue instruction is received, if it is determined that data exists in the device disk, the data is read from the device disk; if it is determined that data does not exist in the device disk and data exists in the memory queue, the data in the memory queue is dequeued.

[0184] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0185] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0186] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: When data to be written is obtained, the data to be written is written into a memory queue established in the device memory, and when it is determined that the data in the memory queue meets the set disk writing conditions, the data in the memory queue is moved to the device disk; When receiving a data dequeue instruction, if it is determined that data exists in the device disk, reading the data from the device disk; If it is determined that there is no data in the device disk and that there is data in the memory queue, dequeueing the data in the memory queue; Writing the data to be written into a memory queue established in the device memory includes: Writing the data to be written into a first memory queue established in the device memory; When it is determined that the data in the first memory queue meets a set condition, the first memory queue is marked as a second memory queue, the data in the second memory queue is encapsulated into a file, and the file is moved to a device disk; The step of reading data from the device disk includes: Determine a file on the device disk whose file index value is the current value of the second metadata as a target file, and determine the current value of the third metadata as a target index value, wherein the second metadata is used to represent the file index value of the currently dequeued file, and the third metadata is used to represent the index value of the next dequeued data; Read data whose subscript value is the target subscript value from the target file.

2. The method according to claim 1, characterized in that The method further comprises: After marking the first memory queue as the second memory queue, a new first memory queue is created in the device memory.

3. The method according to claim 2, characterized in that After writing the data to be written into the first memory queue established in the device memory, the method further includes: Updating the value of the first attribute of the first memory queue to the sum of the current value of the first attribute and the amount of the data to be written, where the first attribute is used to represent the current amount of data in the memory queue; The determining whether the data in the first memory queue meets a set disk flushing condition includes: When it is determined that the value of the first attribute of the first memory queue reaches a set threshold, it is determined that the data in the first memory queue meets a set disk write condition.

4. The method according to claim 1, wherein The method further comprises: Setting the file index value of the file to the current value of the first metadata, where the first metadata is used to represent the file index value that needs to be occupied when the data is written to the disk; After writing the file to the device disk, adding the file index value of the file to the file index value set, and updating the value of the first metadata to the sum of the current value of the first metadata and 1; The step of determining whether data exists on the device disk includes: Determine whether the file index value set is empty; When it is determined that the file index value set is not empty, it is determined that data exists in the device disk.

5. The method according to claim 1, wherein The step of reading data having a subscript value of the target subscript value from the target file includes: When reading data from the target file for the first time, the target file is loaded from the device disk into the device memory to obtain a corresponding backup memory queue; and data whose index value in the backup memory queue is the target index value is dequeued; When data is not read from the target file for the first time, data with an index value of the target index value in the backup memory queue is dequeued.

6. The method according to claim 5, characterized in that After dequeuing the data whose index value is the target index value in the backup memory queue, the method further includes: Updating the value of the second attribute of the backup memory queue to a value obtained by subtracting 1 from the current value of the second attribute, where the second attribute is used to represent the amount of data in the memory queue; When it is determined that the current value of the second attribute of the backup memory queue is not 0, the value of the third metadata is updated to the value of the current value of the third metadata plus 1; when it is determined that the current value of the second attribute of the backup memory queue is 0, the value of the third metadata is updated to the corresponding initial value, the value of the second metadata is updated to the value of the current value of the second metadata plus 1, and the current value of the second metadata is deleted from the file index value set.

7. The method according to claim 2, characterized in that Dequeuing the data in the memory queue includes: Determining whether the second memory queue exists in the device memory; When it is determined that the second memory queue exists in the device memory, dequeueing the data in the second memory queue; When it is determined that the second memory queue does not exist in the device memory, determining whether the first memory queue exists in the device memory; When it is determined that the first memory queue exists in the device memory, data in the first memory queue is dequeued.

8. The method according to claim 7, characterized in that Dequeuing the data in the second memory queue includes: Backing up the second memory queue to obtain a corresponding backup memory queue; Dequeueing the data in the backup memory queue; Dequeuing the data in the first memory queue includes: Creating a new backup memory queue in the device memory, and soft-linking the address of the first memory queue to the backup memory queue; Dequeue the data in the backup memory queue.

9. A data processing device, characterized in that: include: The writing module is used to write the data to be written into the memory queue established in the device memory when the data to be written is obtained; A disk write module, configured to move the data in the memory queue to the device disk when it is determined that the data in the memory queue meets a set disk write condition; a reading module, configured to read the data from the device disk when receiving a data dequeue instruction and determining that the data exists in the device disk; If it is determined that there is no data in the device disk and that there is data in the memory queue, dequeueing the data in the memory queue; Writing the data to be written into a memory queue established in the device memory includes: Writing the data to be written into a first memory queue established in the device memory; When it is determined that the data in the first memory queue meets a set condition, the first memory queue is marked as a second memory queue, the data in the second memory queue is encapsulated into a file, and the file is moved to a device disk; The step of reading data from the device disk includes: Determine a file on the device disk whose file index value is the current value of the second metadata as a target file, and determine the current value of the third metadata as a target index value, wherein the second metadata is used to represent the file index value of the currently dequeued file, and the third metadata is used to represent the index value of the next dequeued data; Read data whose subscript value is the target subscript value from the target file.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 8.

Images