Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method, relates to the technical field of artificial intelligence, in particular to the technical field of big data, cloud computing and distributed computing, and can be applied to an intelligent cloud scene. A specific implementation scheme is as follows: according to a data amount of initial data and a to-be-processed task corresponding to the initial data, a plurality of logical block data is determined, wherein the to-be-processed task corresponds to a related operation performed according to the initial data; according to initial metadata of the initial data, logical block metadata of each of the plurality of logical block data is determined, wherein the initial metadata is used to indicate a position of a preset value in the initial data, and the logical block metadata is used to indicate a storage position of the logical block data in a storage unit and a position of the preset value in the logical block data; and at least one logical block data is read according to the plurality of logical block metadata. The present disclosure also provides a data processing device, an electronic equipment and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of big data, cloud computing, and distributed computing technology, applicable to intelligent cloud scenarios. More specifically, this disclosure provides a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of big data technology, the application scenarios of distributed computing engines are constantly increasing. Data processed by a distributed computing engine can come from distributed storage systems. Distributed storage systems and distributed computing engines can be deployed on different servers. To perform data processing, data can be read from the distributed storage system and sent to the devices associated with the distributed computing engine. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a data processing method is provided, the method comprising: determining a plurality of logical block data based on the data volume of initial data and a task to be processed corresponding to the initial data, wherein the task to be processed corresponds to a related operation to be performed based on the initial data; determining logical block metadata for each of the plurality of logical block data based on the initial metadata of the initial data, wherein the initial metadata is used to indicate the position of a preset value in the initial data, and the logical block metadata is used to indicate the storage position of the logical block data in a storage unit and the position of the preset value in the logical block data; and reading at least one logical block data based on the plurality of logical block metadata.

[0005] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a first determining module, configured to determine a plurality of logical block data based on the data volume of initial data and a task to be processed corresponding to the initial data, wherein the task to be processed corresponds to a related operation performed based on the initial data; a second determining module, configured to determine the logical block metadata of each of the plurality of logical block data based on the initial metadata of the initial data, wherein the initial metadata is used to indicate the position of a preset value in the initial data, and the logical block metadata is used to indicate the storage position of the logical block data in a storage unit and the position of the preset value in the logical block data; and a reading module, configured to read at least one logical block data based on the plurality of logical block metadata.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided according to this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is an exemplary system architecture diagram of a data processing method and apparatus applicable according to an embodiment of the present disclosure;

[0012] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0013] Figure 3A This is a schematic diagram of a data processing method according to an embodiment of the present disclosure;

[0014] Figure 3B This is a schematic diagram of logic block data according to an embodiment of the present disclosure;

[0015] Figure 4 This is a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

[0016] Figure 5 This is a block diagram of an electronic device to which a data processing method can be applied, according to an embodiment of the present disclosure. Detailed Implementation

[0017] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] Reading data from a distributed storage system can consume significant amounts of network input / output (I / O) and memory resources. Furthermore, the process requires a central processing unit (CPU) to perform operations such as deserialization and decompression, resulting in high hardware costs for data reading.

[0019] To reduce the hardware costs required to read data from a distributed storage system, a distributed caching system can be set up for the distributed computing engine. Based on the data scanned by the distributed computing engine, the distributed caching system can generate cache files. These cache files can be stored on the local disk of the distributed computing engine. The size of the cache files can vary depending on the type of distributed caching system. For example, a cache file could be 64 megabytes (MB).

[0020] Distributed caching systems can cache the entire file containing scanned data. In a distributed storage system, a file can be up to 128 megabytes in size. However, scanned data may only comprise a portion of the entire file. This file is less likely to be reused in subsequent processing. Therefore, this file can be implemented as two cache files in the distributed caching system. For example, in interactive analytics scenarios, data analysts might use a distributed computing engine to scan large amounts of data to extract the desired information. Cacheing all the files containing the scanned data could waste significant disk resources of the distributed caching system.

[0021] Furthermore, if the cached data resides in an entire file, the generation time of the cache file will be relatively long. With frequent access to the distributed caching system within a short period, the distributed computing engine may not be able to effectively utilize the distributed caching system, leading to a decrease in cache hit rate.

[0022] Figure 1 This is a schematic diagram of an exemplary system architecture to which data processing methods and apparatus can be applied according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0023] like Figure 1 As shown, the system architecture 100 according to this embodiment may include server 110 and server 120. Network 130 is used as a medium to provide a communication link between server 110 and server 120. Network 130 may include various connection types, such as wired and / or wireless communication links, etc.

[0024] Server 110 may be equipped with a distributed storage system 111. The distributed storage system can store a large number of data files. Server 120 may be equipped with a distributed computing engine 121 and a distributed caching system 122. The distributed computing engine 121 can scan for the required data from the distributed storage system 111. Based on the data scanned by the distributed computing engine 121, the distributed caching system 122 can generate cache files. Next, the distributed computing engine 121 can read data from the distributed caching system 122.

[0025] Server 120 or Server 110 can be a server that provides various services.

[0026] It should be noted that the data processing method provided in the embodiments of this disclosure can generally be executed by the server 120. Accordingly, the data processing device provided in the embodiments of this disclosure can generally be located in the server 120.

[0027] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present disclosure.

[0028] In this embodiment of the disclosure, method 200 may be executed by a distributed computing engine.

[0029] like Figure 2 As shown, the method 200 may include operations S210 to S230.

[0030] In operation S210, multiple logical block data are determined based on the amount of initial data and the tasks to be processed corresponding to the initial data.

[0031] In this embodiment of the disclosure, the task to be processed can be various tasks. For example, the task to be processed can be an Optical Character Recognition (OCR) task. As another example, an image containing text can be used as initial data corresponding to an optical character recognition task.

[0032] In this embodiment of the disclosure, the task to be processed corresponds to the relevant operation performed based on the initial data. For example, an optical character recognition task may include the following operation: recognizing multiple characters from an image containing text.

[0033] In this embodiment of the disclosure, the initial data can be data from a distributed caching system. For example, the format of the initial data can be a sparse file. For example, the initial data can correspond to initial metadata. The initial metadata can indicate the position of a preset value in the initial data. In one example, the preset value can be 0. It is understood that on the disk storing the initial data, the non-zero data of the initial data occupies the physical space of the disk. The preset value in the initial data may not occupy physical space on the disk.

[0034] In this embodiment of the disclosure, taking an optical character recognition task as an example, the initial data size can be 28 megabytes. Based on this, 28 logical blocks of data can be determined. The data size of each logical block can be 1 megabyte. For example, multiple logical blocks of data can each correspond to multiple partial images of an image containing text. Each partial image can correspond to one character.

[0035] In operation S220, the initial metadata of the initial data is used to determine the respective logical block metadata of multiple logical block data.

[0036] In this embodiment of the disclosure, logical block metadata is used to indicate the storage location of logical block data in the storage unit and the location of preset values ​​within the logical block data. For example, the storage location of initial data in the storage unit (e.g., the disk mentioned above) is determinable. The logical block metadata is determined based on the initial metadata. The logical block metadata can indicate the storage location of logical block data in the storage unit.

[0037] In operation S230, at least one logical block data is read based on multiple logical block metadata.

[0038] For example, based on the logical block metadata, the distributed computing engine can determine the storage location of the logical block data. The distributed computing engine can then read the logical block data from that storage location.

[0039] Through the embodiments of this disclosure, multiple logical block data related to the task to be processed are determined based on initial data, and logical block metadata of the logical block data is also determined, enabling the distributed computing engine to read the logical block data, which helps to improve disk utilization. In scenarios such as interactive analysis, this can significantly improve the utilization of hardware resources and reduce hardware and time costs.

[0040] Furthermore, by determining the logic block data based on the task to be processed through the embodiments of this disclosure, the correlation between the logic block data and the task to be processed can be enhanced, which helps to improve the execution efficiency of the task to be processed.

[0041] Furthermore, through the embodiments of this disclosure, the distributed computing engine can read logical block data, which helps to improve the cache hit rate, especially when data is read frequently in a short period of time, it can efficiently improve the cache hit rate.

[0042] It is understood that the method flow of this disclosure has been described above, and the method of this disclosure will be further described below in conjunction with relevant embodiments.

[0043] Figure 3A This is a schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0044] like Figure 3A As shown, the distributed computing engine may include a scheduling node 301 and worker nodes 302. The distributed storage system 303 may be deployed on the first server. The worker node 302 may be deployed on the second server. The scheduling node 301 may be deployed on other servers. Server 320 may also deploy a distributed caching system 304. It can be understood that the distributed computing engine may include multiple worker nodes. Different worker nodes may be deployed on different servers. It can also be understood that the first server may be server 110 as described above. The second server may be server 120 as described above.

[0045] like Figure 3A As shown, worker node 302 may include worker thread 3021 and worker thread 3022. According to instructions issued by the scheduling node, worker thread 3022 can read data from the distributed storage system 303. For the data file read to worker node 302, the distributed caching system 304 can convert the data file into initial data 305 according to the sparse file format. Next, the distributed caching system 304 can write the initial data 305 to the disk of the second server. For example, the physical size of the initial data 305 may be 28 megabytes. The logical size of the initial data 305 may be 64 megabytes.

[0046] like Figure 3A As shown, initial data 305 may include: multiple preset values ​​(e.g., 0) and numerical values ​​other than the preset values. For example, initial data 305 may include initial sub-data 3051, initial sub-data 3052, and initial sub-data 3053. Initial sub-data 3053 may correspond to the initial metadata of initial data 305 and the footer of the initial data. Initial sub-data 3051, initial sub-data 3052, and initial sub-data 3053 may include numerical values ​​other than the preset values.

[0047] Figure 3B This is a schematic diagram of logic block data according to an embodiment of the present disclosure.

[0048] In some embodiments of operation S210 described above, taking an optical character recognition task as an example, multiple logical block data can be determined based on the data volume of the initial data 305 and the task to be processed corresponding to the initial data 305. The data volume of the logical block data can be 1 megabyte. For example, the multiple logical block data may include, for example, Figure 3B The first logical block data 3061 and the second logical block data 3062 are shown.

[0049] In some embodiments, in some implementations of operation S220 described above, the individual logical block metadata of multiple logical block data can be determined based on the initial metadata of initial data 305. For example, the initial metadata can be obtained from initial sub-data 3053. Next, the metadata of multiple logical blocks can be determined. In embodiments of this disclosure, the logical block metadata may include a starting address and an offset. The location of the logical block data can be determined based on the starting address and offset so that the distributed computing engine can read the data.

[0050] In some embodiments of operation S230 described above, reading at least one logical block data based on multiple logical block metadata may include: in response to receiving a data read instruction related to a task to be processed, generating multiple cache block data based on the multiple logical block metadata. At least one cache block data is then read. For example, after receiving a data read instruction related to a task to be processed, 28 cache block data can be generated based on 28 logical block metadata. As another example, for any given logical block data, the logical block metadata may also indicate the respective first positions of multiple preset values ​​within the logical block data. The cache block data can be obtained by filling the multiple preset values ​​into the multiple first positions respectively.

[0051] In this embodiment of the disclosure, there are N logical block data items and N logical block metadata items. For example, N can be 28.

[0052] In this embodiment of the disclosure, reading at least one logical block data based on multiple logical block metadata may further include: reading the nth logical block data based on the nth logical block metadata among N logical block metadata. For example, n is an integer greater than or equal to 1 and less than N. For example, as described above, 28 cache block data can be generated based on 28 logical block data. Next, taking n=1 as an example, the cache block data corresponding to the first logical block data 3061 can be read based on the first logical block metadata.

[0053] In this embodiment of the disclosure, reading at least one logical block data based on multiple logical block metadata may further include: in response to determining that the nth logical block data has been read completely, reading the (n+1)th logical block data based on the (n+1)th logical block metadata. For example, after determining that the cache block data corresponding to the first logical block data 3061 has been read completely, the cache block data corresponding to the second logical block data 3062 may be read based on the second logical block metadata.

[0054] In this embodiment of the disclosure, reading at least one logical block data based on multiple logical block metadata may further include: in response to determining that the (N-1)th logical block data has been read completely, reading the Nth logical block data based on the Nth logical block metadata. For example, after determining that the cache block data corresponding to the 27th logical block data has been read completely, the cache block data corresponding to the 28th logical block data can be read based on the 28th logical block metadata. Through this embodiment of the disclosure, when the distributed computing engine reads data, the logical block data is converted into cache block data based on the logical block metadata, enabling the distributed computing engine to accurately read the data and thus accurately perform data processing, which helps to improve the accuracy of data processing.

[0055] It is understood that the above has further explained the method flow of this disclosure. The following will describe some implementation methods for determining logical block data and logical block metadata in conjunction with relevant embodiments.

[0056] In another embodiment of this disclosure, in some implementations of the above-described operation S210, determining multiple logical block data of the initial data includes: determining multiple logical block data of the non-tail data of the initial data. For example, initial sub-data 3051 and initial sub-data 3052 of the initial data 305 can be used as the non-tail data of the initial data 305. Initial sub-data 3053 can be used as the tail data of the initial data 305. Multiple logical block data can be determined based on the initial sub-data 3051 and initial sub-data 3052.

[0057] Furthermore, in another embodiment of this disclosure, in some implementations of the above-described operation S220, determining the logical block metadata of each of the multiple logical block data may further include: updating the tail data of the initial data based on the multiple logical block metadata. For example, as described above, the initial sub-data 3053 may correspond to the initial metadata of the initial data 305 and the file tail of the initial data. The initial sub-data 3053 may serve as the tail data of the initial data 305. After obtaining the multiple logical block metadata, the initial sub-data 3053 may be updated so that the updated tail data corresponds to the multiple logical block metadata and the file tail of the initial data 305. Through the embodiments of this disclosure, the distributed computing engine can obtain logical block metadata from the tail data, which helps to read data accurately and quickly.

[0058] It is understandable that the logical block data is determined in order to facilitate the distributed computing engine to read the data. The physical storage form of the initial data 305 on the disk can be maintained, and the initial data 305 does not need to be physically split.

[0059] It is understood that the above description uses an optical character recognition task as an example to illustrate this disclosure. However, this disclosure is not limited to this. For example, unlike initial data 305, the data size of another initial data set could be 53 megabytes. The task to be processed corresponding to this initial data set could be various tasks. Based on this, according to the data size of the initial data set and the associated task to be processed, two logical blocks of data can be determined. The data sizes of the two logical blocks of data could be 32 megabytes and 13 megabytes, respectively.

[0060] Figure 4 This is a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0061] like Figure 4 As shown, the device 400 may include a first determining module 410, a second determining module 420, and a reading module 430.

[0062] The first determining module 410 is used to determine multiple logical block data based on the amount of initial data and the tasks to be processed corresponding to the initial data. For example, the tasks to be processed correspond to the relevant operations to be performed based on the initial data.

[0063] The second determining module 420 is used to determine the logical block metadata of each of the multiple logical block data based on the initial metadata of the initial data. For example, the initial metadata is used to indicate the location of a preset value in the initial data, and the logical block metadata is used to indicate the storage location of the logical block data in the storage unit and the location of the preset value in the logical block data.

[0064] The read module 430 is used to read at least one logical block data based on multiple logical block metadata.

[0065] In some embodiments, the reading module includes: a generation unit, configured to generate multiple cache block data based on multiple logical block metadata in response to receiving a data reading instruction related to a task to be processed; and a first reading unit, configured to read at least one cache block data.

[0066] In some embodiments, the second determining module includes: updating the tail data of the initial data based on the metadata of multiple logical blocks.

[0067] In some embodiments, the first determining module includes: a first determining unit, which includes a plurality of logical block data for determining non-tail data of the initial data.

[0068] In some embodiments, there are N logical block data sets and N logical block metadata sets. The reading module includes: a second reading unit, configured to read the nth logical block data based on the nth logical block metadata set among the N logical block metadata sets; and a third reading unit, configured to read the (n+1)th logical block data based on the (n+1)th logical block metadata set in response to determining that the nth logical block data has been read completely. For example, n is an integer greater than or equal to 1 and less than N.

[0069] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0070] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0071] In embodiments of this disclosure, an electronic device may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0072] In embodiments of this disclosure, a non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform a method provided according to this disclosure.

[0073] In embodiments of this disclosure, a computer program product may include a computer program that, when executed by a processor, implements the methods provided according to this disclosure. The following will be combined with... Figure 5 Please provide a detailed explanation.

[0074] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0075] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0076] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0077] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0078] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) monitor or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0083] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0084] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Based on the amount of initial data and the tasks to be processed corresponding to the initial data, multiple logical block data are determined, wherein the tasks to be processed correspond to the relevant operations performed based on the initial data; Based on the initial metadata of the initial data, the respective logical block metadata of the multiple logical block data is determined, wherein the initial metadata is used to indicate the position of a preset value in the initial data, and the logical block metadata is used to indicate the storage location of the logical block data in the storage unit and the position of the preset value in the logical block data; and Read at least one of the logical block data based on the metadata of the plurality of logical blocks.

2. The method according to claim 1, wherein, Reading at least one of the logical block data based on the metadata of the plurality of logical blocks includes: In response to receiving a data read instruction related to the task to be processed, multiple cache block data are generated based on the metadata of the multiple logical blocks; and Read at least one of the cached block data.

3. The method according to claim 1, wherein, The determination of the logical block metadata for each of the multiple logical block data includes: Update the tail data of the initial data based on the metadata of the multiple logical blocks.

4. The method according to claim 1, wherein, The determination of multiple logical block data includes: Determine multiple logical blocks of data that are not tail data of the initial data.

5. The method according to claim 1, wherein, The number of logical block data items is N, and the number of logical block metadata items is N. The step of reading at least one logical block data based on the metadata of the plurality of logical blocks includes: Based on the metadata of the nth logical block among the N logical block metadata, read the data of the nth logical block; In response to determining that the data of the nth logical block has been read, the data of the (n+1)th logical block is read according to the metadata of the (n+1)th logical block, where n is an integer greater than or equal to 1 and less than N.

6. A data processing apparatus, comprising: The first determining module is used to determine multiple logical block data based on the amount of initial data and the tasks to be processed corresponding to the initial data, wherein the tasks to be processed correspond to related operations performed based on the initial data; The second determining module is configured to determine the logical block metadata of each of the plurality of logical block data based on the initial metadata of the initial data, wherein the initial metadata is used to indicate the position of a preset value in the initial data, and the logical block metadata is used to indicate the storage location of the logical block data in the storage unit and the position of the preset value in the logical block data; and A reading module is used to read at least one of the logical block data based on the metadata of the plurality of logical blocks.

7. The apparatus according to claim 6, wherein, The reading module includes: A generation unit is configured to, in response to receiving a data read instruction related to the task to be processed, generate multiple cache block data based on the metadata of multiple logical blocks; and The first reading unit is used to read at least one of the cache block data.

8. The apparatus according to claim 6, wherein, The second determining module includes: Update the tail data of the initial data based on the metadata of the multiple logical blocks.

9. The apparatus according to claim 6, wherein, The first determining module includes: The first determining unit includes a plurality of logical block data that determine the non-tail data of the initial data.

10. The apparatus according to claim 6, wherein, The number of logical block data items is N, and the number of logical block metadata items is N. The reading module includes: The second reading unit is used to read the data of the nth logical block based on the metadata of the nth logical block among the N logical block metadata. The third reading unit is configured to read the (n+1)th logical block data based on the (n+1)th logical block metadata in response to determining that the nth logical block data has been read. Here, n is an integer greater than or equal to 1 and less than N.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images