Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The application discloses a data processing method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: obtaining a write request for a to-be-written file; determining a read-write unit corresponding to each to-be-written data in a plurality of to-be-written data according to a first offset, a second offset, and sequence information and capacity information of each read-write unit in the plurality of read-write units; and writing each to-be-written data into the read-write unit corresponding to each to-be-written data. The application solves the technical problem of low read-write performance caused by the fact that, in the related art, the read-write data needs to be recorded according to the metadata, which leads to data redundancy and a long data addressing time.

Description

Technical Field

[0001] This application relates to the field of data storage, and more specifically, to a data processing method and apparatus, a non-volatile storage medium, and an electronic device. Background Technology

[0002] File storage stores data in the form of files on physical devices. Object-based file storage has the following characteristics: 1. Each file's content is composed of multiple objects, rather than a one-to-one correspondence; 2. The file's metadata information is not in the object storage, but in another service, such as a database.

[0003] In related file storage technologies based on storage objects, since the storage objects themselves do not support partial updates, it is necessary to add metadata to the files to record valid data during the implementation process, which leads to technical problems such as data redundancy and waste of data resources.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a data processing method and apparatus, a non-volatile storage medium, and an electronic device to at least solve the technical problem of low read and write performance caused by data redundancy and long data addressing time due to the need to record read and write data based on metadata in related technologies.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: obtaining a write request for a file to be written, wherein the file to be written includes: a plurality of data to be written, the write request including: a file identifier of the file to be written, used to determine a plurality of read / write units for storing the file to be written, the write request further including: a first offset and a second offset, the first offset being the offset of each of the plurality of data to be written relative to a first position, the second offset being the offset of each of the data to be written relative to a second position, the first position being the starting position of the file to be written in the plurality of read / write units, and the second position being the starting position of each of the plurality of read / write units; determining a read / write unit corresponding one-to-one with each of the plurality of data to be written based on the first offset, the second offset, and sequence information and capacity information of each of the plurality of read / write units; and writing each of the data to be written to the read / write unit corresponding to each of the data to be written.

[0007] Optionally, writing each piece of data to be written to the read / write unit corresponding to each piece of data to be written includes: when the read / write unit includes an initial data block, processing each piece of data to be written and the initial data block respectively in a left-to-right order according to the overlap relationship between each piece of data to be written and the initial data block, and the data length of each piece of data to be written and the initial data block, to obtain multiple target data blocks, wherein the data length of each piece of data to be written is a first length, and the data length of the initial data block is an initial length; and writing each target data block in the multiple target data blocks to the read / write unit corresponding to each piece of data to be written in each target data block.

[0008] Optionally, following a left-to-right order, each piece of data to be written and the initial data block are processed according to the overlap relationship between each piece of data to be written and the initial data block, as well as the data lengths of each piece of data to be written and the initial data block, to obtain multiple target data blocks. This includes: if each piece of data to be written is to the left of the initial data block and has no intersection with it, merging each piece of data to be written with the initial data block to obtain a first target data block, where the length of the first target data block is the sum of a first length and an initial length; if each piece of data to be written is to the left of the initial data block and has an intersection with it, merging each piece of data to be written with the initial data block to obtain a second target data block, where the length of the second target data block is the sum of the first length and the initial length minus the first target length, where the first target length is the length of each piece of data to be written that intersects with the initial data block; if each piece of data to be written is completely contained within the initial data block, merging each piece of data to be written with the initial data block to obtain a third target data block, where the length of the third target data block is the initial length.

[0009] Optionally, following a left-to-right order, each piece of data to be written is processed with the initial data block according to the overlap relationship between each piece of data to be written and the initial data block, as well as the data length of each piece of data to be written and the initial data block, to obtain multiple target data blocks. The method further includes: if each piece of data to be written is to the right of the initial data block and has no intersection with the initial data block, comparing each piece of data to be written with the next initial data block until each piece of data to be written is to the left of the initial data block, or each piece of data to be written is to the right of the initial data block and has an intersection with the initial data block; if each piece of data to be written is to the right of the initial data block and has an intersection with the initial data block, merging each piece of data to be written with the initial data block to obtain a fifth target data block, wherein the length of the fifth target data block is the sum of the first length and the initial length minus the second target length, and the second target length is the length of each piece of data to be written that has an intersection with the initial data block; if each piece of data to be written completely contains the initial data block, merging each piece of data to be written with the initial data block to obtain a sixth target data block, wherein the length of the sixth target data block is the first length.

[0010] Optionally, each read / write unit corresponds one-to-one with a storage unit.

[0011] Optionally, writing each piece of data to be written to the corresponding read / write unit includes: writing each piece of data to be written to the corresponding read / write unit to obtain multiple first target read / write units, wherein each of the multiple first target read / write units includes: written data; after obtaining multiple first target read / write units, if a target condition exists, the written data is sent to the storage unit, wherein the target condition includes at least one of the following: receiving a target instruction carrying target semantics, the establishment time of the read / write unit exceeding a preset threshold, and no space to be written in the read / write unit, wherein the target semantics are used to instruct the read / write unit to perform persistence operations.

[0012] Optionally, during the process of sending the written data to the storage unit, multiple first target read / write units refuse to receive the file to be written.

[0013] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: obtaining a read request for a file to be read, wherein the read request includes: a file identifier of the file to be read, used to determine a plurality of read / write units for storing the file to be read, the read request further including: a third offset and the length of the data to be read in the file to be read, the third offset being the offset of the data to be read relative to a third position, the third position being the starting position of the file to be read in the plurality of read / write units; determining a second target read / write unit among the plurality of read / write units according to the third offset and the length of the data to be read; and reading the data to be read in the second target read / write unit.

[0014] Optionally, reading data to be read in the second target read / write unit includes: if the number of data to be read with the same target length in the second target read / write unit exceeds a preset target number, then asynchronously pre-reading the unread data in the second target read / write unit according to the target length.

[0015] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: an acquisition module, configured to acquire a write request for a file to be written, wherein the file to be written includes: a plurality of data to be written, the write request including: a file identifier of the file to be written, configured to determine a plurality of read / write units for storing the file to be written, the write request further including: a first offset and a second offset, the first offset being the offset of each of the plurality of data to be written relative to a first position, the second offset being the offset of each of the data to be written relative to a second position, the first position being the starting position of the file to be written in the plurality of read / write units, and the second position being the starting position of each of the plurality of read / write units; a determination module, configured to determine a read / write unit corresponding one-to-one with each of the plurality of data to be written based on the first offset, the second offset, and sequence information and capacity information of each of the plurality of read / write units; and a write module, configured to write each of the data to be written to the read / write unit corresponding to each of the data to be written.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the above data processing method when it runs.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the above-described data processing method when it runs.

[0018] In this embodiment, a write request for a file to be written is obtained. The file to be written includes multiple pieces of data to be written. The write request includes a file identifier for the file to be written, used to determine multiple read / write units for storing the file to be written. The write request also includes a first offset and a second offset. The first offset is the offset of each piece of data to be written relative to a first position, and the second offset is the offset of each piece of data to be written relative to a second position. The first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each read / write unit. Based on the first offset... The second offset, along with the sequence and capacity information of each read / write unit in the multiple read / write units, determines the read / write unit that corresponds one-to-one with each piece of data to be written. By writing each piece of data to be written into the read / write unit corresponding to that piece of data, the goal of automatic addressing based on the data to be written is achieved without recording the data location through metadata. This optimizes data redundancy, saves data resources, and improves read / write performance. It also solves the technical problem of low read / write performance caused by data redundancy and long data addressing time due to the need for related technologies to record read / write data based on metadata. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 It is a flowchart of a data processing method based on relevant technologies;

[0021] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of another data processing method according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a data processing method according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram illustrating the processing of data to be written and an initial data block according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application;

[0026] Figure 7This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application;

[0027] Figure 8 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application;

[0028] Figure 9 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application;

[0029] Figure 10 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application;

[0030] Figure 11 This is a schematic diagram illustrating asynchronous pre-reading of unread data in a second target read / write unit according to an embodiment of this application;

[0031] Figure 12 This is a structural diagram of a data processing apparatus according to an embodiment of this application;

[0032] Figure 13 This is a hardware structure block diagram of a computer terminal according to an embodiment of the present application for a data processing method. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] In related technologies, particularly file storage technologies based on storage objects, since the storage objects themselves do not support partial updates, it is necessary to add metadata to the files to record valid data during the implementation process, such as... Figure 1 As shown. Therefore, the following problems exist: 1. Data redundancy and wasted costs, requiring a large amount of metadata space to record the valid data of each file; 2. Long read / write paths and poor performance, each time data is read or written, metadata needs to be accessed first to obtain the data location, affecting read / write performance; 3. Severe fragmentation, seriously affecting file read / write performance, with data overlapping and causing data redundancy. In the case of random file writes, a large number of fragment objects will be generated. When reading data, more metadata access is required to concatenate the valid data before accessing the data, further generating a large amount of performance loss and seriously affecting read performance.

[0036] Especially when using Rados as a storage platform, since Rados objects themselves support random writes, using the above read and write logic will not only fail to leverage Rados's inherent characteristics, but will also result in poor file read and write performance due to significant performance overhead.

[0037] Rados is a scalable distributed storage system used for storing and managing large-scale data. Part of the Ceph storage system, Rados provides a high-performance, highly available, and reliable storage solution. Rados employs a distributed architecture, automatically managing data replication and recovery, and supports multiple storage interfaces and data access methods. Rados also offers flexible data management and storage strategies that can be customized and configured according to specific needs.

[0038] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0039] According to an embodiment of this application, a method embodiment for data processing is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0041] Step S202: Obtain a write request for the file to be written, wherein the file to be written includes: multiple data to be written, and the write request includes: a file identifier of the file to be written, used to determine multiple read / write units for storing the file to be written, and the write request also includes: a first offset and a second offset, wherein the first offset is the offset of each of the multiple data to be written relative to a first position, and the second offset is the offset of each of the multiple data to be written relative to a second position, the first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each of the multiple read / write units.

[0042] A file identifier is an identifier used within a file to identify information such as file type, format, version, author, and creation date. File identifiers typically include a unique identifier, file type, file size, creation date, modification date, and file format. File identifiers help users manage and identify files more easily and play an important role in file transfer, storage, and backup. In this embodiment, the file identifier is, for example, the unique identifier of a file within the file system: inode, where inode is a 64-bit integer.

[0043] As we understand it, the read / write unit is used to write files and data to be written, and also to read files and data to be read. The read / write unit is a memory read / write unit, which is the smallest unit in a computer used for storing and accessing data. A memory read / write unit is a storage unit in computer memory that can store one byte of data and can be read or written. A memory read / write unit typically consists of an address and a memory element used to identify and store data. During computer operation, the central processing unit (CPU) accesses the memory read / write unit through the address, and can read or write data within it. The speed and capacity of the memory read / write unit have a significant impact on the computer's performance and storage capacity.

[0044] Step S204: Based on the first offset, the second offset, and the sequence information and capacity information of each of the multiple read / write units, determine the read / write unit that corresponds one-to-one with each of the multiple data to be written.

[0045] Optionally, each of the multiple read / write units has the same storage capacity.

[0046] According to some optional embodiments of this application, for example, the number of multiple read / write units for receiving the file to be written is 10, wherein each of the 10 read / write units has 4M of memory, and the total capacity of the multiple read / write units is 40M. It is understood that the writing order of the file to be written in the multiple read / write units is from 0M to 40M, so the starting position of the file to be written in the multiple read / write units is 0M. Since each read / write unit has 4M of memory, for each read / write unit, its starting position is 0M and its ending position is 4M.

[0047] For example, the target object indicates that 2MB of data to be written will be written into multiple read / write units starting from the 6MB mark of the file to be written, where the file to be written is 10MB. The specific steps are: obtaining a target request to write the file to be written into multiple read / write units, wherein the target request includes: a first offset, a second offset, and sequence information and capacity information of each read / write unit in the multiple read / write units. For example, the number of multiple read / write units used to receive the file to be written is 10, namely the first read / write unit, the second read / write unit, ..., the tenth read / write unit, where each of the 10 read / write units has 4MB of memory, then the first offset is 6MB, the second offset is 2MB, and so on. Figure 4 As shown, IO represents the data to be written. In summary, the read / write unit corresponding to the aforementioned 2MB of data to be written is the second read / write unit.

[0048] According to some alternative embodiments of this application, the read / write unit corresponding to the data to be written is determined according to the following formula:

[0049] writeIndex=offset / rados_object_size

[0050] objOffset=offset%rados_object_size

[0051] Where offset is the first offset; rados_object_size is the memory size of the read / write unit, which defaults to 4M; writeIndex is the sequence of read / write units; and objOffset is the second offset.

[0052] Step S206: Write each piece of data to be written to the read / write unit corresponding to each piece of data to be written.

[0053] It is understandable that the initial data block is the data block that already exists in the read / write unit before the data to be written is written to the read / write unit.

[0054] In some optional embodiments of this application, step S206 specifically includes the following steps:

[0055] Step S2061: When the read / write unit includes an initial data block, each data to be written and the initial data block are processed in a left-to-right order according to the overlap relationship between each data to be written and the initial data block, as well as the data length of each data to be written and the initial data block, to obtain multiple target data blocks. The data length of each data to be written is the first length, and the data length of the initial data block is the initial length.

[0056] Following a left-to-right order, each piece of data to be written and the initial data block are processed according to the overlap relationship between each piece of data to be written and the initial data block, as well as the data length of each piece of data to be written and the initial data block, to obtain multiple target data blocks. This can be achieved through the following method:

[0057] Scenario 1: Figure 5 This is a schematic diagram illustrating the processing of data to be written and an initial data block according to an embodiment of this application. Off1 represents the initial data block, and Off2 represents the data to be written. Figure 5 As shown, if each piece of data to be written is to the left of the initial data block and has no intersection with the initial data block, each piece of data to be written is merged with the initial data block to obtain the first target data block, wherein the length of the first target data block is the sum of the first length and the initial length.

[0058] Scenario 2: Figure 6 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application. Off1 is the initial data block, and Off2 is the data to be written. Figure 6 As shown, if each piece of data to be written is to the left of the initial data block and intersects with the initial data block, each piece of data to be written is merged with the initial data block to obtain a second target data block. The length of the second target data block is the sum of the first length and the initial length minus the first target length. The first target length is the length of each piece of data to be written that intersects with the initial data block.

[0059] Scenario 3: Figure 7 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application. Off1 is the initial data block, and Off2 is the data to be written. Figure 7 As shown, if each piece of data to be written is completely contained within the initial data block, each piece of data to be written is merged with the initial data block to obtain a third target data block, wherein the length of the third target data block is the initial length.

[0060] Scenario 4: Figure 8This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application. Off1 is the initial data block, and Off2 is the data to be written. Figure 8 As shown, if each piece of data to be written is to the right of the initial data block and has no intersection with the initial data block, each piece of data to be written is compared with the next initial data block until each piece of data to be written is to the left of the initial data block, or each piece of data to be written is to the right of the initial data block and has an intersection with the initial data block.

[0061] Scenario 5: Figure 9 This is a schematic diagram illustrating another embodiment of the present application of processing the data to be written with the initial data block, where off1, off2, and off3 are the initial data blocks, and off4 is the data to be written. Figure 9 As shown, if each piece of data to be written is to the right of the initial data block and intersects with the initial data block, each piece of data to be written is merged with the initial data block to obtain the fifth target data block. The length of the fifth target data block is the sum of the first length and the initial length minus the second target length. The second target length is the length of each piece of data to be written that intersects with the initial data block.

[0062] Scenario Six: Figure 10 This is a schematic diagram illustrating another method of processing the data to be written with the initial data block according to an embodiment of this application. Off1 is the initial data block, and Off2 is the data to be written. Figure 10 As shown, if each piece of data to be written completely contains the initial data block, each piece of data to be written is merged with the initial data block to obtain the sixth target data block, wherein the length of the sixth target data block is the first length.

[0063] Step S2062: Write each target data block in the plurality of target data blocks into the read / write unit corresponding to each data to be written in each target data block.

[0064] Preferably, each read / write unit corresponds one-to-one with a storage unit, such as a Rados object. Step S206 further includes the following steps:

[0065] Step S2063: Write each piece of data to be written into the read / write unit corresponding to each piece of data to be written, to obtain multiple first target read / write units, wherein each of the multiple first target read / write units includes: written data;

[0066] Step S2064: After obtaining multiple first target read / write units, if a target situation exists, the written data is sent to the storage unit. The target situation includes at least one of the following: receiving a target instruction carrying target semantics, the establishment time of the read / write unit exceeds a preset threshold, and there is no space to be written in the read / write unit. The target semantics are used to instruct the read / write unit to perform persistence operations.

[0067] Preferably, during the process of sending the written data to the storage unit, multiple first target read / write units refuse to receive the file to be written.

[0068] In a preferred embodiment, the file system may experience simultaneous reading and writing of massive numbers of files. In such cases, merging read-ahead and write I / O will cause insufficient memory and may lead to memory shortage. A memory usage threshold can be set. If the memory usage threshold is exceeded, all read and write I / O will skip the memory unit and directly access the storage to the Rados object.

[0069] Based on the above steps, a write request for the file to be written is obtained. The file to be written includes multiple pieces of data to be written. The write request includes a file identifier for the file to be written, used to determine multiple read / write units for storing the file to be written. The write request also includes a first offset and a second offset. The first offset is the offset of each piece of data to be written relative to a first position, and the second offset is the offset of each piece of data to be written relative to a second position. The first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each read / write unit. Based on the first offset, the second offset, and the sequence information and capacity information of each read / write unit, a read / write unit corresponding one-to-one with each piece of data to be written is determined. Each piece of data to be written is written to the read / write unit corresponding to each piece of data to be written. This method achieves the goal of automatic addressing based on the data to be written without recording the location of the data through metadata, thereby optimizing data redundancy, saving data resources, and improving read / write performance.

[0070] Figure 3 This is a flowchart of another data processing method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0071] Step S302: Obtain a read request for the file to be read. The read request includes: a file identifier of the file to be read, used to determine multiple read / write units for storing the file to be read. The read request also includes: a third offset and the length of the data to be read in the file to be read. The third offset is the offset of the data to be read relative to a third position, and the third position is the starting position of the file to be read in the multiple read / write units.

[0072] Step S304: Determine the second target read / write unit among multiple read / write units based on the third offset and the length of the data to be read.

[0073] Step S306: Read the data to be read in the second target read / write unit.

[0074] Figure 11 This is a schematic diagram illustrating asynchronous pre-reading of unread data in a second target read / write unit according to an embodiment of this application, as shown below. Figure 11 As shown, if the number of data items of the same target length to be read in the second target read / write unit exceeds the preset target quantity, the unread data in the second target read / write unit is asynchronously pre-read according to the target length. Asynchronous data pre-reading is an optimization technique for data reading. It utilizes prediction algorithms and multi-threading technology to load data into memory in advance before it is actually needed. This reduces waiting time and improves data reading efficiency. Asynchronous data pre-reading is commonly used in scenarios such as large databases, file systems, or network data transmission. By reading data in advance, the number of I / O operations can be reduced, improving the overall performance of the system.

[0075] Through the above steps, metadata no longer needs to be recorded during the entire file data read / write process. Instead, the corresponding data location is directly calculated based on the offset of the data to be written or read, saving addressing time. This also avoids large amounts of file metadata, shortens the read / write path, reduces performance overhead, and significantly improves read / write performance. In scenarios with a large number of random read / write operations, it effectively avoids a large number of fragmented objects and useless data, ensuring that every actually existing object stores valid data.

[0076] Figure 12 This is a structural diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0077] The acquisition module 1202 is used to acquire a write request for a file to be written, wherein the file to be written includes: multiple data to be written, and the write request includes: a file identifier of the file to be written, used to determine multiple read / write units for storing the file to be written, and the write request also includes: a first offset and a second offset, wherein the first offset is the offset of each of the multiple data to be written relative to a first position, and the second offset is the offset of each of the multiple data to be written relative to a second position, the first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each of the multiple read / write units;

[0078] The determination module 1204 is used to determine the read / write unit that corresponds one-to-one with each of the multiple data to be written, based on the first offset, the second offset, and the sequence information and capacity information of each of the multiple read / write units.

[0079] The write module 1206 is used to write each piece of data to be written to the read / write unit corresponding to each piece of data to be written.

[0080] It should be noted that the above Figure 12 The modules in can be program modules (e.g., a set of program instructions that implement a specific function) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.

[0081] It should be noted that, Figure 12 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0082] Figure 13 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. Figure 13 As shown, the computer terminal 130 may include one or more processors 1302 (shown as 1302a, 1302b, ..., 1302n in the figure) 1302 (processor 1302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1304 for storing data, and a transmission module 1306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 13 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 130 may also include... Figure 13 The more or fewer components shown, or having the same Figure 13 The different configurations shown.

[0083] It should be noted that the aforementioned one or more processors 1302 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 130. As involved in the embodiments of this application, this data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0084] The memory 1304 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 1302 executes various functional applications and data processing by running the software programs and modules stored in the memory 1304, thereby implementing the aforementioned data processing method. The memory 1304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1304 may further include memory remotely located relative to the processor 1302, and these remote memories can be connected to the computer terminal 130 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] The transmission module 1306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 130. In one example, the transmission module 1306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 1306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0086] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 130.

[0087] It should be noted here that, in some optional embodiments, the above... Figure 13The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 13 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0088] It should be noted that, Figure 13 The computer terminal shown is used to execute Figure 2 The data processing method shown above is also applicable to this electronic device, and will not be repeated here.

[0089] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program controls the device where the storage medium is located to execute the above data processing method when it runs.

[0090] A program for a non-volatile storage medium performs the following functions: acquiring a write request for a file to be written, wherein the file to be written includes: multiple pieces of data to be written; the write request includes: a file identifier for the file to be written, used to determine multiple read / write units for storing the file to be written; the write request also includes: a first offset and a second offset, wherein the first offset is the offset of each piece of data to be written relative to a first position, and the second offset is the offset of each piece of data to be written relative to a second position; the first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each read / write unit in the multiple read / write units; determining a read / write unit corresponding one-to-one with each piece of data to be written based on the first offset, the second offset, and the sequence information and capacity information of each read / write unit in the multiple read / write units; and writing each piece of data to be written to the read / write unit corresponding to each piece of data to be written.

[0091] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described data processing method during runtime.

[0092] The processor is used to run a program that performs the following functions: acquiring a write request for a file to be written, wherein the file to be written includes: multiple pieces of data to be written; the write request includes: a file identifier for the file to be written, used to determine multiple read / write units for storing the file to be written; the write request also includes: a first offset and a second offset, wherein the first offset is the offset of each piece of data to be written relative to a first position, and the second offset is the offset of each piece of data to be written relative to a second position; the first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each read / write unit in the multiple read / write units; determining a read / write unit corresponding one-to-one with each piece of data to be written based on the first offset, the second offset, and the sequence information and capacity information of each read / write unit in the multiple read / write units; and writing each piece of data to be written to the read / write unit corresponding to each piece of data to be written.

[0093] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0094] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: A write request is obtained for a file to be written, wherein the file to be written includes: multiple pieces of data to be written; the write request includes: a file identifier of the file to be written, used to determine multiple read / write units for storing the file to be written; the write request also includes: a first offset and a second offset; the first offset is the offset of each piece of data to be written relative to a first position; the second offset is the offset of each piece of data to be written relative to a second position; the first position is the starting position of the file to be written in the multiple read / write units; the second position is the starting position of each read / write unit in the multiple read / write units; wherein the storage capacity of each read / write unit in the multiple read / write units is the same; the remainder obtained by dividing the first offset by the storage capacity of the read / write unit is the second offset. Based on the first offset, the second offset, and the sequence information and capacity information of each of the plurality of read and write units, a read and write unit corresponding one-to-one with each of the plurality of data to be written is determined, wherein the sequence information of the read and write unit corresponding to each data to be written is determined based on the quotient obtained by dividing the first offset by the storage capacity of the read and write unit. Each piece of data to be written is written into the read / write unit corresponding to each piece of data to be written. Writing each piece of data to be written to the read / write unit corresponding to each piece of data to be written includes: when the read / write unit includes an initial data block, processing each piece of data to be written and the initial data block respectively in a left-to-right order according to the overlap relationship between each piece of data to be written and the initial data block, and the data length of each piece of data to be written and the initial data block, to obtain a plurality of target data blocks, wherein the data length of each piece of data to be written is a first length, and the data length of the initial data block is an initial length; and writing each of the plurality of target data blocks to the read / write unit corresponding to each piece of data to be written in each target data block.

2. The method according to claim 1, characterized in that, Following a left-to-right order, each piece of data to be written and the initial data block are processed according to the overlap relationship between each piece of data to be written and the initial data block, as well as the data length of each piece of data to be written and the initial data block, to obtain multiple target data blocks, including: If each piece of data to be written is to the left of the initial data block and has no intersection with the initial data block, then each piece of data to be written is merged with the initial data block to obtain a first target data block, wherein the length of the first target data block is the sum of the first length and the initial length; If each piece of data to be written is to the left of the initial data block and intersects with the initial data block, then each piece of data to be written is merged with the initial data block to obtain a second target data block. The length of the second target data block is the sum of the first length and the initial length minus the first target length. The first target length is the length of each piece of data to be written that intersects with the initial data block. If each piece of data to be written is completely contained within the initial data block, each piece of data to be written is merged with the initial data block to obtain a third target data block, wherein the length of the third target data block is the initial length.

3. The method according to claim 1, characterized in that, Following a left-to-right order, each piece of data to be written and the initial data block are processed according to the overlap relationship between each piece of data to be written and the initial data block, as well as the data length of each piece of data to be written and the initial data block, to obtain multiple target data blocks. The method further includes: If each piece of data to be written is to the right of the initial data block and has no intersection with the initial data block, compare each piece of data to be written with the next initial data block until each piece of data to be written is to the left of the initial data block, or each piece of data to be written is to the right of the initial data block and has an intersection with the initial data block; If each piece of data to be written is to the right of the initial data block and intersects with the initial data block, then each piece of data to be written is merged with the initial data block to obtain a fifth target data block. The length of the fifth target data block is the sum of the first length and the initial length minus the second target length. The second target length is the length of each piece of data to be written that intersects with the initial data block. If each piece of data to be written completely contains the initial data block, each piece of data to be written is merged with the initial data block to obtain a sixth target data block, wherein the length of the sixth target data block is the first length.

4. The method according to claim 1, characterized in that, Each read / write unit corresponds one-to-one with a storage unit.

5. The method according to claim 4, characterized in that, Writing each piece of data to be written to the read / write unit corresponding to each piece of data to be written includes: Each piece of data to be written is written to the corresponding read / write unit to obtain a plurality of first target read / write units, wherein each of the plurality of first target read / write units includes: already written data; After obtaining the plurality of first target read / write units, if a target situation exists, the written data is sent to the storage unit. The target situation includes at least one of the following: receiving a target instruction carrying target semantics, the establishment time of the read / write unit exceeding a preset threshold, and there is no space to be written in the read / write unit. The target semantics are used to instruct the read / write unit to perform persistence operations.

6. The method according to claim 5, characterized in that, During the process of sending the written data to the storage unit, the plurality of first target read / write units refuse to receive the file to be written.

7. A data processing apparatus, characterized in that, include: An acquisition module is used to acquire a write request for a file to be written, wherein the file to be written includes: multiple pieces of data to be written, and the write request includes: a file identifier of the file to be written, used to determine multiple read / write units for storing the file to be written, and the write request also includes: a first offset and a second offset, wherein the first offset is the offset of each piece of data to be written relative to a first position, and the second offset is the offset of each piece of data to be written relative to a second position, wherein the first position is the starting position of the file to be written in the multiple read / write units, and the second position is the starting position of each read / write unit in the multiple read / write units, wherein the storage capacity of each read / write unit in the multiple read / write units is the same, and the remainder obtained by dividing the first offset by the storage capacity of the read / write unit is the second offset; The determination module is used to determine the read / write unit that corresponds one-to-one with each of the multiple data to be written, based on the first offset, the second offset, and the sequence information and capacity information of each of the multiple read / write units. The sequence information of the read / write unit corresponding to each data to be written is determined based on the quotient obtained by dividing the first offset by the storage capacity of the read / write unit. A writing module is used to write each piece of data to be written into the read / write unit corresponding to each piece of data to be written. The writing module is further configured to perform the following steps: when the read / write unit includes an initial data block, in a left-to-right order, according to the overlap relationship between each data to be written and the initial data block, and the data length of each data to be written and the initial data block, process each data to be written and the initial data block respectively to obtain multiple target data blocks, wherein the data length of each data to be written is a first length, and the data length of the initial data block is an initial length; and write each of the multiple target data blocks into the read / write unit corresponding to each data to be written in each target data block.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the data processing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data processing method according to any one of claims 1 to 6.

Images