Data processing method and apparatus

Abstract

The application relates to a data processing method and device. The method comprises the following steps: obtaining a to-be-processed task; the to-be-processed task comprises task data; determining an operation type corresponding to the to-be-processed task; the operation type at least comprises one of a data downloading type, a data storage type, a data access type and a data arrangement type; determining a space type to which a storage space related to the task data belongs; determining a target processing interface matched with the space type; the target processing interface is designed based on a communication protocol supported by the space type; calling the target processing interface, and operating the storage space based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task. The method can improve the diversity of data operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a data processing method and device. BACKGROUND

[0002] At present, intelligent learning technologies such as machine learning and deep learning have been widely applied in many fields such as smart retail, security, and education. However, with the development of science and technology, users have put forward more refined demands for intelligent learning technologies, and therefore, in order to quickly respond to user demands and promote efficient iteration of intelligent learning algorithms, it is particularly important to construct an intelligent learning platform for machine learning model training.

[0003] Among them, the data center is a key module in the intelligent learning platform, and the intelligent learning platform can process data through the data center, for example, the training data and label data can be stored through the data center. However, the existing data center is mainly designed for local import, and usually only supports operations on the same storage space, which greatly limits the diversity of data operations. SUMMARY

[0004] Therefore, it is necessary to provide a data processing method and device capable of improving the diversity of data operations, a computer device, and a storage medium in view of the above technical problems.

[0005] A data processing method, the method comprising:

[0006] obtaining a to-be-processed task; the to-be-processed task comprising task data;

[0007] determining an operation type corresponding to the to-be-processed task; the operation type comprising at least one of a data download type, a data storage type, a data access type, and a data arrangement type;

[0008] determining a space type to which a storage space related to the task data belongs;

[0009] determining a target processing interface matched with the space type; the target processing interface being designed based on a communication protocol supported by the space type;

[0010] calling the target processing interface, and operating the storage space based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0011] In an embodiment, the obtaining of the to-be-processed task comprises:

[0012] obtaining a current task queue and determining a first queue length of the task queue;

[0013] determining a second queue length of the cache queue when the first queue length is less than a preset queue threshold;

[0014] storing a difference number of task identifiers stored in the cache queue into the task queue according to a difference between the first queue length and the preset queue threshold when the second queue length is not zero;

[0015] respectively taking tasks corresponding to the task identifiers in the task queue as to-be-processed tasks.

[0016] In an embodiment, the method further comprises:

[0017] deleting a task identifier of a to-be-processed task that has been processed successfully from the task queue;

[0018] supplementing task identifiers in the cache queue to the task queue and performing corresponding tasks based on the supplemented task identifiers.

[0019] A data processing apparatus, the apparatus comprising:

[0020] a task acquisition module configured to acquire a to-be-processed task; the to-be-processed task comprising task data;

[0021] an interface determination module configured to determine an operation type corresponding to the to-be-processed task; the operation type comprising at least one of a data downloading type, a data storing type, a data accessing type, and a data sorting type; determine a space type to which a storage space related to the task data belongs; determine a target processing interface matched with the space type; the target processing interface being designed based on a communication protocol supported by the space type;

[0022] an execution module configured to call the target processing interface and perform an operation on the storage space based on the task data according to an operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0023] In an embodiment, the task acquisition module further comprises a task scheduling module configured to acquire a current task queue and determine a first queue length of the task queue; determine a second queue length of a cache queue when the first queue length is less than a preset queue threshold; store a difference number of task identifiers stored in the cache queue into the task queue according to a difference between the first queue length and the preset queue threshold when the second queue length is not zero; and respectively take tasks corresponding to the task identifiers in the task queue as to-be-processed tasks.

[0024] In an embodiment, the data processing apparatus is further configured to, for each to-be-processed task, invoke a target processing interface corresponding to the to-be-processed task, and perform the to-be-processed task in parallel by operating on a storage space according to an operation mode corresponding to an operation type to which the to-be-processed task belongs, based on task data in the to-be-processed task.

[0025] In an embodiment, the task scheduling module is further configured to receive a user operation request, and generate a corresponding task based on task data and a task identifier carried in the user operation request; store the task data in a cache space, and store the task identifier in the task queue when the first queue length is less than the preset queue threshold; and store the task identifier in the cache queue when the first queue length is equal to the preset queue threshold.

[0026] In an embodiment, the task obtaining module is further configured to, when it is determined that the to-be-processed task processing is abnormal, obtain a task identifier stored in the task queue, and re-read corresponding task data from the cache space according to the task identifier; and re-execute the to-be-processed task based on the task data.

[0027] In an embodiment, the data processing apparatus comprises a to-be-processed task, and the to-be-processed task comprises at least one of a data download task, a data import task, a data migration task, a data access task, and a data arrangement task; task data corresponding to the data download task comprises a to-be-downloaded data identifier; task data corresponding to the data import task comprises an import space identifier; task data corresponding to the data migration task comprises a source space identifier and a migration space identifier; task data corresponding to the data access task comprises an access data identifier; and task data corresponding to the data arrangement task comprises an arrangement space identifier.

[0028] In an embodiment, the interface determining module comprises a space type determining module configured to, when the to-be-processed task is a data download task, determine a space type of a storage space storing to-be-downloaded data according to the to-be-downloaded data identifier; when the to-be-processed task is a data import task, determine a space type of a storage space for storing to-be-imported data according to the import space identifier; when the to-be-processed task is a data migration task, determine a space type of a source storage space currently storing to-be-migrated data according to the source space identifier, and determine a space type of a migration space for migrating the to-be-migrated data according to the migration space identifier; when the to-be-processed task is a data access task, determine a space type of a storage space storing to-be-accessed data according to the access data identifier; and when the to-be-processed task is a data arrangement task, determine a space type of a to-be-arranged space according to the arrangement space identifier.

[0029] In an embodiment, the space types in the data processing apparatus include an object storage type, a file storage type, and a big data storage type; the target processing interface matched with the object storage type is an object processing interface designed based on a simple storage service communication protocol; the target processing interface matched with the file storage type is a file processing interface designed based on a basic read-write communication protocol; and the target processing interface matched with the big data storage type is a big data processing interface designed based on a distributed file system communication protocol.

[0030] In an embodiment, the data storage types include a data import type, and the execution module is further configured to, when the operation type corresponding to the to-be-processed task is the data import type, call the target processing interface corresponding to the to-be-processed task to obtain to-be-imported data, and perform decompression processing on the to-be-imported data to obtain at least one to-be-imported decompressed data; and store each to-be-imported decompressed data to a storage space related to task data in the to-be-processed task through the target processing interface.

[0031] In an embodiment, the execution module is further configured to, when it is determined based on a heartbeat packet sent by a user terminal that the user terminal has successfully uploaded to-be-imported data, call the target processing interface corresponding to the to-be-processed task to obtain the to-be-imported data; wherein the heartbeat packet is an information packet generated by the user terminal based on an upload progress of the to-be-imported data.

[0032] In an embodiment, the data storage types include a data migration type; the task data includes a to-be-migrated data identifier, a source space identifier, and a migration space identifier; the target processing interface includes a source processing interface corresponding to the source space identifier and a migration processing interface corresponding to the migration space identifier; and the execution module is further configured to, when the operation type corresponding to the to-be-processed task is the data migration type, call the source processing interface, and based on the to-be-migrated data identifier, obtain to-be-migrated data from a source storage space corresponding to the source space identifier; and call the migration processing interface to migrate the to-be-migrated data to a migration space corresponding to the migration space identifier.

[0033] In an embodiment, the task data includes a to-be-arranged space identifier; and the execution module is further configured to, when the operation type corresponding to the to-be-processed task is the data arrangement type, call the target processing interface corresponding to the to-be-processed task to access a to-be-arranged space corresponding to the to-be-arranged space identifier; and through the target processing interface, traverse non-compressed data stored in the to-be-arranged space to determine first data information of the non-compressed data; and store the first data information in a preset storage space.

[0034] In an embodiment, the execution module is further configured to, when the operation type corresponding to the to-be-processed task is a data arrangement type, invoke a target processing interface corresponding to the to-be-processed task, access a to-be-arranged space corresponding to the to-be-arranged space identifier, download compressed data stored in the to-be-arranged space through the target processing interface, and decompress the downloaded compressed data to obtain at least one to-be-decompressed data.

[0035] In an embodiment, the data processing apparatus is further configured to, when the task data includes label data, check the data format and data content of the label data to obtain a checking result, and when the checking result indicates that the checking is passed, perform an operation on a label database based on the label data according to an operation mode corresponding to the operation type to which the to-be-processed task belongs, to execute the to-be-processed task.

[0036] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] obtaining a to-be-processed task; the to-be-processed task includes task data;

[0038] determining an operation type corresponding to the to-be-processed task; the operation type includes at least one of a data download type, a data storage type, a data access type, and a data arrangement type;

[0039] determining a space type to which a storage space related to the task data belongs;

[0040] determining a target processing interface matched with the space type; the target processing interface is designed based on a communication protocol supported by the space type;

[0041] invoking the target processing interface, and performing an operation on the storage space based on the task data according to an operation mode corresponding to the operation type, to execute the to-be-processed task.

[0042] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0043] obtaining a to-be-processed task; the to-be-processed task includes task data;

[0044] determine an operation type corresponding to the to-be-processed task; the operation type at least includes one of a data download type, a data storage type, a data access type, and a data arrangement type;

[0045] determine a space type to which a storage space related to the task data belongs;

[0046] determine a target processing interface matched with the space type; the target processing interface is designed based on a communication protocol supported by the space type;

[0047] invoke the target processing interface, and perform an operation on the storage space based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0048] A computer program product or a computer program includes computer instructions stored in a computer readable storage medium, a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the following steps: obtaining a to-be-processed task; the to-be-processed task includes task data; determining an operation type corresponding to the to-be-processed task; the operation type at least includes one of a data download type, a data storage type, a data access type, and a data arrangement type; determining a space type to which a storage space related to the task data belongs; determining a target processing interface matched with the space type; the target processing interface is designed based on a communication protocol supported by the space type; invoking the target processing interface, and performing an operation on the storage space based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0049] The above data processing method, device, computer device, storage medium, and computer program can obtain a to-be-processed task, determine an operation type of the to-be-processed task, and obtain task data. By obtaining the task data, a space type associated with the task data can be determined, so that a target processing interface matched with the determined space type can be determined. By determining the target processing interface, an operation can be performed on a corresponding storage space based on the target processing interface and the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task. Since the target processing interface is determined based on the space type corresponding to the task data, the present application can perform operations on storage spaces of different space types through different types of processing interfaces. Compared with a traditional method of supporting storage of to-be-processed data in only one type of storage space, the present application greatly improves the diversity of data operations. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1Figure 1 is a diagram of an application environment for a data processing method according to one embodiment;

[0051] Figure 2 Figure 2 is a flowchart of a data processing method according to one embodiment;

[0052] Figure 3 Figure 3 is a diagram of a data center interface for a target application according to one embodiment;

[0053] Figure 4 Figure 4 is a diagram of an architecture for a target application according to one embodiment;

[0054] Figure 5 Figure 5 is a diagram of an asynchronous task scheduling according to one embodiment;

[0055] Figure 6 Figure 6 is a flowchart of performing a data import task according to one embodiment;

[0056] Figure 7 Figure 7 is a flowchart of performing a data migration task according to one embodiment;

[0057] Figure 8 Figure 8 is a flowchart of performing a data organization task according to one embodiment;

[0058] Figure 9 Figure 9 is a flowchart of processing label data according to one embodiment;

[0059] Figure 10 Figure 10 is a diagram of a data processing system according to one embodiment;

[0060] Figure 11 Figure 11 is a flowchart of a data processing method according to one embodiment;

[0061] Figure 12 Figure 12 is a block diagram of a data processing apparatus according to one embodiment;

[0062] Figure 13 Figure 13 is a block diagram of a data processing apparatus according to one embodiment;

[0063] Figure 14 Figure 14 is a diagram of an internal structure of a computer device according to one embodiment.DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] Figure 1 This is an application environment diagram illustrating a data processing method in one embodiment. (Refer to...) Figure 1 This data processing method is applied to a data processing system 100. The data processing system 100 includes a user terminal 102 and a server 104. The user terminal 102 can generate tasks to be processed based on user operations and send these tasks to the server 104. The server 104 then executes the tasks by operating on the corresponding storage space through a target processing interface based on the task data in the task. The user terminal 102 can be, but is not limited to, a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, or smart TV. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any limitations on this connection.

[0067] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0068] Step S202: Obtain the tasks to be processed; the tasks to be processed include task data.

[0069] Specifically, a target application, such as an intelligent learning application, runs on the user terminal and responds to user operations. This target application responds to the user's processing operations on the data to be processed, thereby obtaining task data and task identifiers. For example, an intelligent learning application can respond to a user's download operation, obtaining the data identifier of the data to be downloaded and the task identifier of the data download task. Further, the target application can send the task data and task identifiers to a server, so that the server can generate a corresponding task based on the task data and task identifiers. The target application can be either a parent application or a child application (MiniProgram, also known as a small program). The parent application can be a native app (Application) or a web page accessed via a web browser. Task data refers to the data carried by the task to be processed; specifically, task data can be the data input by the user through the target application when initiating the data processing task. For example, in the above example, the task data could be the data identifier of the data to be downloaded.

[0070] When the server needs to process the generated tasks, the server can select at least one task from the generated tasks, and make the selected tasks as to-be-processed tasks, so that each to-be-processed task can be processed in parallel subsequently. For example, the server can acquire at least one task generated within a preset time period, and make the at least one task generated within the preset time period as to-be-processed tasks.

[0071] In an embodiment, when the target application running in the user terminal acquires the task identifier and the task data, the target application can generate a task according to the task identifier and the task data, and send the task to the server.

[0072] In an embodiment, the to-be-processed task at least includes one of a data download task, a data import task, a data migration task, a data access task, and a data arrangement task; the task data corresponding to the data download task includes to-be-downloaded data identifier; the task data corresponding to the data import task includes import space identifier; the task data corresponding to the data migration task includes source space identifier and migration space identifier; the task data corresponding to the data access task includes access data identifier; and the task data corresponding to the data arrangement task includes arrangement space identifier.

[0073] Specifically, the to-be-processed task at least includes one of a data download task, a data import task, a data migration task, a data access task, and a data arrangement task. The data download task refers to a task of downloading data stored in a storage space. The task data corresponding to the data download task includes to-be-downloaded data identifier of to-be-downloaded data, so that when the data download task is executed, the server can determine corresponding to-be-downloaded data according to the to-be-downloaded data identifier, and perform download processing on the to-be-downloaded data.

[0074] The data storage task includes a data import task and a data migration task. The data import task refers to a task of storing to-be-imported data in a corresponding storage space. For example, a user terminal can upload to-be-imported data to a server, so that the server stores the to-be-imported data uploaded by the user in a corresponding storage space through the data import task. The task data corresponding to the data import task includes an import space identifier of a storage space used to store to-be-imported data, so that when the data import task is executed, the server can determine the corresponding storage space based on the import space identifier, and store the to-be-imported data uploaded by the user in the determined storage space. The data migration task refers to a task of migrating to-be-migrated data stored in a source storage space to a migration space. For example, the server can migrate data stored in an A storage space to a B storage space. The task data corresponding to the data migration task includes a source space identifier of a source storage space used to store to-be-migrated data, and a migration space identifier of a migration space used to migrate to-be-migrated data, so that when the data migration task is executed, the server can obtain to-be-migrated data from the source storage space corresponding to the source space identifier, and store the to-be-migrated data in the migration space corresponding to the migration space identifier.

[0075] The data access task refers to a task of accessing data stored in a storage space. The task data corresponding to the data access task includes an access data identifier, so that when the data access task is executed, the server can determine the corresponding to-be-accessed data based on the access data identifier, and read the to-be-accessed data. The data arrangement task refers to a task of arranging data stored in a corresponding storage space. The server can obtain related data information of data stored in a corresponding storage space through the data arrangement task. For example, the server can determine the data amount, data size, and data format of data stored in a corresponding storage space through the data arrangement task. The task data corresponding to the data arrangement task includes an arrangement space identifier, so that when the data arrangement task is executed, the server can determine a to-be-arranged to-be-arranged space according to the arrangement space identifier, and arrange the to-be-arranged space.

[0076] The storage space refers to a space for storing data. The storage space can be divided into a source storage space, a transfer storage space, a to-be-arranged storage space, and the like. The storage space can specifically be a space provided by a corresponding storage medium, for example, the storage space can specifically be a space provided by COS (China Object Storage), a space provided by CEPHFS (a file storage system), or a space provided by HDFS (Hadoop Distributed File System). It can be easily understood that when the storage medium providing the storage space is different, the type of the storage space is also different. For example, when the storage medium providing the storage space is COS, the type of the storage space is an object storage type, when the storage medium providing the storage space is CEPHFS, the type of the storage space is a file storage type, and when the storage medium providing the storage space is HDFS, the type of the storage space is a big data storage type.

[0077] The COS is a distributed object storage service without a directory hierarchy, without data format restrictions, capable of accommodating massive data, and supporting HTTP / HTTPS protocol access. The COS is compatible with the S3 (Amazon Simple Storage Service) communication protocol, facilitating users to directly use community tools and plug-ins. The CEPHFS is a distributed file system based on POSIX (Portable Operating System Interface) without a single point of failure, and through the CEPHFS, data can be fault-tolerant and seamlessly replicated. The HDFS is a distributed file system running on commodity hardware. The HDFS is a highly fault-tolerant system suitable for deployment on inexpensive machines. The HDFS can provide high-throughput data access and is suitable for large-scale data sets.

[0078] In one embodiment, with reference to Figure 3 , Figure 3 A schematic diagram of a data center interface of a target application in one embodiment is shown. As shown in Figure 3 , the target application can include a data center, and the data center interface can exhibit a data download control 302, a data import control 304, a data transfer control 306, a data access control 308, and a data arrangement control 310. The user terminal can obtain a corresponding task identifier and task data in response to a touch operation of a user on a corresponding control, and send the task identifier and the task data to the server, so that the server generates a corresponding task through the received task identifier and task data.

[0079] For example, when the user touches the data download control 302, the user terminal can display a corresponding data list according to the touch operation of the user, so that the user selects the to-be-downloaded data from the data list, and the user terminal can determine the to-be-downloaded data identifier of the to-be-downloaded data according to the selection operation of the user, and generate a task identifier of the data download task. The user terminal sends the to-be-downloaded data identifier and the task identifier to the server, so that the server receives the to-be-downloaded data identifier as task data, and generates a data download task according to the task data and the task identifier. For another example, when the user touches the data import control 304, the user terminal can display a data information input box according to the touch operation of the user, and the user can input an import space identifier of a storage space for storing to-be-imported data and upload the to-be-imported data through the data information input box. Further, the user terminal can send the import space identifier input by the user and the to-be-imported data uploaded by the user to the server, so that the server receives the import space identifier and the to-be-imported data as task data, and generates a corresponding data import task according to the task data. For another example, when the user touches the data transfer control 306, the terminal can display an information input box according to the touch operation of the user, and the user can input a source space identifier, a transfer space identifier, and a to-be-transferred data identifier of to-be-transferred data through the information data box, so that the user terminal can send the source space identifier, the transfer space identifier, and the to-be-transferred data identifier of the to-be-transferred data to the server. Further, the server receives the source space identifier, the transfer space identifier, and the to-be-transferred data identifier of the to-be-transferred data input by the user as task data, and generates a data transfer task according to the task data and a corresponding task identifier.

[0080] In one of the embodiments, the data center interface can further display a data publishing control and a data deletion control. The user can publish the data stored in the storage space through the data publishing control, and delete the data stored in the storage space through the data deletion control.

[0081] In the above embodiments, by generating a plurality of types of to-be-processed tasks, the to-be-processed data can be diversified processed based on the plurality of types of to-be-processed tasks, thereby greatly improving the diversity of data processing.

[0082] In step S204, the operation type corresponding to the to-be-processed task is determined. The operation type at least includes one of a data download type, a data storage type, a data access type, and a data arrangement type.

[0083] Specifically, the server can generate different types of to-be-processed tasks through different types of user operations performed in the target application, for example, can generate a data download task, a data storage task, a data access task, and the like. It is easy to understand that different types of to-be-processed tasks have different operation types, for example, the operation type corresponding to the data download task is a data download type, the operation type corresponding to the data storage task is a data storage type, the operation type corresponding to the data access task is a data access type, and the operation type corresponding to the data arrangement task is a data arrangement type.

[0084] The operation type refers to the type of operation mode used to execute the to-be-processed task, for example, when the to-be-processed task is a data storage task, the operation mode used to execute the data storage task is a data storage mode. That is, when the operation type corresponding to the to-be-processed task is a data storage type, the task processing thread can execute the code associated with the data storage in the data storage mode to implement the storage of the to-be-imported data.

[0085] In one embodiment, the to-be-processed task further includes an operation type identifier, and the server can determine the operation type corresponding to the corresponding to-be-processed task through the operation type identifier in the to-be-processed task. For example, when the user touches the data download control in the data center interface to expect to download the to-be-downloaded data, the user terminal can determine the corresponding operation type identifier based on the control touched by the user, and send the operation type identifier to the server, so that the server generates the to-be-processed task according to the operation type identifier.

[0086] In one embodiment, the to-be-processed task includes a task identifier, and the server can determine the task type through the task identifier, and determine the operation type according to the task type. For example, the server stores an association relationship between the task identifier and the task type, and an association relationship between the task type and the operation type. When the to-be-processed task is obtained, the server can determine the target task type corresponding to the to-be-processed task according to the association relationship between the task identifier and the task type, and determine the target operation type corresponding to the target task type according to the association relationship between the task type and the operation type.

[0087] Step S206, determining the space type to which the storage space related to the task data belongs.

[0088] The space type refers to a type corresponding to the storage space. When storage media providing the storage space are different, the space types of the storage spaces are also different. The space types of the storage spaces include an object storage type, a file storage type, and a big data storage type. When the space type of the storage space is the object storage type, the server can store data in the storage space in a distributed object manner. When the space type of the storage space is the file storage type, the server stores data in the storage space in a distributed file manner. When the space type of the storage space is the big data storage type, the server can store large-scale data in the storage space.

[0089] It is easily understood that the server in the embodiment of the present application is deployed with storage spaces of multiple space types, so that the server can subsequently perform corresponding operations on storage spaces of multiple types. For example, the server can store A data in a storage space of type A, download C data from a storage space of type B, and transfer D data from a storage space of type C to a storage space of type D.

[0090] Specifically, because the to-be-processed task includes corresponding task data, the server can determine the space type of the corresponding storage space according to the task data in the to-be-processed task.

[0091] In one embodiment, determining the space type to which the storage space related to the task data belongs includes: when the to-be-processed task is a data download task, determining the space type of the storage space storing the to-be-downloaded data according to the to-be-downloaded data identifier; when the to-be-processed task is a data import task, determining the space type of the storage space used to store the to-be-imported data according to the import space identifier; when the to-be-processed task is a data transfer task, determining the space type of the source storage space currently storing the to-be-transferred data according to the source space identifier, and determining the space type of the transfer storage space used to transfer the to-be-transferred data according to the transfer space identifier; when the to-be-processed task is a data access task, determining the space type of the storage space storing the to-be-accessed data according to the access data identifier; and when the to-be-processed task is a data arrangement task, determining the space type of the to-be-arranged space according to the arrangement space identifier.

[0092] Specifically, when the to-be-processed task is a data download task and the task data corresponding to the data download task includes a to-be-downloaded data identifier, the server can determine the storage space storing the to-be-downloaded data according to the to-be-downloaded data identifier, and determine the space type of the storage space. When the to-be-processed task is a data import task in a data storage task, and the task data corresponding to the data import task includes an import space identifier, the server can determine the storage space used to store the to-be-imported data according to the import space identifier, and determine the space type of the storage space.

[0093] When the to-be-processed task is a data migration task, and the task data corresponding to the data migration task includes a source space identifier and a migration space identifier, the server can determine a source storage space in which the to-be-migrated data is stored according to the source space identifier, and determine a space type of the source storage space. The server can also determine a migration space for receiving the to-be-migrated object according to the migration space identifier, and determine a space type of the migration space. When the to-be-processed task is a data access task, and the task data corresponding to the data access task includes an access data identifier, the server can determine a storage space in which the to-be-accessed data is stored according to the access data identifier, and determine a space type of the storage space. When the to-be-processed task is a data arrangement task, and the task data corresponding to the data arrangement task includes an arrangement space identifier, the server can determine a to-be-arranged space according to the arrangement space identifier, and determine a space type of the to-be-arranged space.

[0094] In one of the embodiments, the server stores an association relationship between storage space identifiers and space types of storage spaces. When determining a storage space, the server can determine a target storage space identifier of the storage space, and determine a target space type corresponding to the target storage space according to the association relationship between the storage space identifiers and the space types.

[0095] In the above embodiments, by obtaining the task data of the to-be-processed task, the space type of the corresponding storage space can be quickly determined based on the task data, thereby improving the confirmation efficiency of the space type.

[0096] In step S208, a target processing interface matching the space type is determined; the target processing interface is designed based on a communication protocol supported by the space type.

[0097] The processing interface refers to an interface designed according to a corresponding communication protocol. The interface generally refers to an abstract object (which can be another entity) provided by an entity to the outside world, which separates internal operations from external communication methods, so that it can be modified internally without affecting the way other entities interact with it.

[0098] Specifically, different types of storage spaces correspond to different processing interfaces, and the server can perform corresponding operations on the matching storage space through the target processing interface. For example, when the processing interface matching the A-type storage space is an A interface, the server can read the data in the A-type storage space through the A interface, or store the to-be-imported data into the A-type storage space through the A interface.

[0099] The server can determine task data included in the to-be-processed task, and determine a target space type to which a storage space related to the task data belongs. Further, the server stores a correspondence between space types and processing interfaces, and can determine a target processing interface matched with the target space type according to the correspondence between the space types and the processing interfaces. In this way, the to-be-processed task can be executed by invoking the target processing interface and based on the task data subsequently.

[0100] In one embodiment, the space types include an object storage type, a file storage type, and a big data storage type; the target processing interface matched with the object storage type is an object processing interface designed based on a simple storage service communication protocol; the target processing interface matched with the file storage type is a file processing interface designed based on a basic read-write communication protocol; and the target processing interface matched with the big data storage type is a big data processing interface designed based on a distributed file system communication protocol.

[0101] Specifically, when the space type of the storage space is the object storage type, the target processing interface matched with the object storage type is the object processing interface. The object processing interface refers to an interface designed based on a simple storage service communication protocol, which is also referred to as an S3 (Amazon Simple Storage Service) communication protocol. Through the object processing interface, the storage space of the object storage type can be operated.

[0102] When the space type of the storage space is the file storage type, the target processing interface matched with the file storage type is the file processing interface. The file processing interface refers to an interface designed based on a basic read-write communication protocol supported by a native computer system. Through the file processing interface, the storage space of the file storage type can be operated.

[0103] When the space type of the storage space is the big data storage type, the target processing interface matched with the big data storage type is the big data processing interface. The big data processing interface refers to an interface designed based on a distributed file system communication protocol, which is also referred to as an HDFS protocol. Through the big data processing interface, the storage space of the big data storage type can be operated.

[0104] In one embodiment, when the space type is an object storage type, the storage space can be a storage space provided by COS or CSP (Cloud Storage on Private). When the space type is a file storage type, the storage space can be a storage space provided by CEPHFS or CFS (Cloud File Storage). When the space type is a big data storage type, the storage space can be a storage space provided by HDFS.

[0105] In this embodiment, by designing various types of processing interfaces, the storage spaces of different space types can be operated through various types of processing interfaces, thereby improving the diversity of operations. In addition, by integrating various types of processing interfaces, the integrated multi-type processing interfaces can be compatible with multiple to-be-processed tasks being executed, thereby improving the compatibility of data processing.

[0106] In step S210, the target processing interface is called, and the storage space is operated based on the task data according to the operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0107] Specifically, the server calls the target processing interface corresponding to the to-be-processed task, and operates the storage space corresponding to the task data according to the operation mode of the operation type corresponding to the to-be-processed task based on the task data included in the to-be-processed task, so as to execute the to-be-processed task. For example, when the to-be-processed task is a data import task, the server can determine that the operation type corresponding to the data import task is a data import type, and determine the target space type to which the storage space used to store the to-be-imported data belongs through the import space identifier included in the data import task. Further, the server determines the target processing interface corresponding to the target space type, and acquires the to-be-imported data through the determined target processing interface, and stores the to-be-imported data into the storage space corresponding to the import space identifier.

[0108] In one embodiment, with reference to Figure 4 , Figure 4 An architecture schematic diagram of a target application in one embodiment is shown. With reference to Figure 4The target application can include a business layer, an interface layer, and a storage medium layer. Through the business layer, a corresponding to-be-processed task can be generated in response to a touch operation of a user. Through the interface layer, the to-be-processed task can be routed to a corresponding target processing interface, so that the target processing interface determines a corresponding storage space and executes the to-be-processed task based on the determined storage space. For example, when the user clicks a download control in the business layer, a download interface in the interface layer can generate a download task through the click operation of the user, and route the download interface to a corresponding target processing interface, so that the target processing interface downloads data in the corresponding storage space. Since the interface layer is encapsulated, the user only needs to call the corresponding processing interface to implement operations on different types of storage spaces without paying attention to the underlying implementation logic, thereby greatly improving the user experience. By designing a plurality of different types of processing interfaces, the user can operate different types of storage spaces according to their own needs, further improving the user experience.

[0109] In one embodiment, when the target application is an intelligent learning application used for machine learning model training, due to the diversity of training data, model training personnel can expect to store different types of training data in different space types of storage spaces. For example, the model training personnel can expect to store big data type training data in a big data storage type storage space, and expect to store file type training data in a file storage type storage space. Therefore, in order to facilitate the model training personnel to store different types of training data, a plurality of different types of processing interfaces can be integrated in the intelligent learning application, so that different types of processing interfaces can be used to operate the corresponding space type of storage space, thereby realizing the diversity of data storage.

[0110] In the above data processing method, by obtaining the to-be-processed task, the operation type of the to-be-processed task can be determined, and the task data can be obtained. By obtaining the task data, the space type associated with the task data can be determined, so that the matching target processing interface can be determined based on the determined space type. By determining the target processing interface, the corresponding storage space can be operated based on the task data according to the operation mode corresponding to the operation type based on the target processing interface, to execute the to-be-processed task. Since the target processing interface is determined based on the space type corresponding to the task data, the present application can operate the storage space of the corresponding space type through different types of processing interfaces, which greatly improves the diversity of data operations compared to the traditional method of only supporting storing the to-be-processed data into the same type of storage space.

[0111] In one embodiment, the tasks to be processed are obtained by: obtaining a current task queue and determining a first queue length of the task queue; when the first queue length is less than a preset queue threshold, determining a second queue length of a cache queue; when the second queue length is not zero, transferring a difference number of task identifiers stored in the cache queue to the task queue according to a difference between the first queue length and the preset queue threshold; and taking the tasks corresponding to the task identifiers in the task queue as the tasks to be processed respectively.

[0112] Specifically, in order to quickly respond to tasks and parallelize processing tasks, a task queue and a cache queue can be set. Both the task queue and the cache queue store task identifiers of tasks. When the server generates a task, the server can preferentially store the task identifier of the generated task in the task queue, and when the task queue is full, store the task identifier of the generated task in the cache queue.

[0113] The server can process the tasks corresponding to the task identifiers stored in the task queue in batches. When the server needs to process tasks in batches, the server can obtain a current task queue and determine a first queue length of the task queue, i.e., obtain a number value of the task identifiers stored in the task queue. The server compares the first queue length with a preset queue threshold to determine whether the first queue length is less than the preset queue length, and when it is determined that the first queue length is less than the preset queue length, it is considered that the task queue is not full. At this time, the server determines a second queue length of the cache queue, i.e., determines a number value of the task identifiers stored in the cache queue. The preset queue threshold refers to a number value of the task identifiers that the task queue can store at most, i.e., a number value of the task identifiers that the task queue can store when it is full.

[0114] When the second queue length is not zero, it can be considered that the cache queue stores task identifiers. At this time, the server subtracts the first queue length from the preset queue threshold to obtain a difference between the first queue length and the preset queue threshold, extracts a difference number of task identifiers from the cache queue, and stores the extracted difference number of task identifiers in the task queue to make the task queue full. For each task identifier in the task queue, the server takes the task corresponding to the current task identifier as a task to be processed, i.e., the server takes the tasks corresponding to the task identifiers in the task queue as tasks to be processed respectively.

[0115] In one embodiment, when the number of task identifiers stored in the cache queue is less than the difference number, the server extracts all the task identifiers in the cache queue and stores the extracted task identifiers in the task queue. When the number of task identifiers stored in the cache queue is greater than the difference number, the server extracts a difference number of task identifiers from the cache queue in the order of task generation and stores the extracted task identifiers in the task queue.

[0116] In the above embodiments, by taking the tasks corresponding to the task identifiers in the task queue as processing tasks, the subsequent processing of the to-be-processed tasks can be performed in parallel, thus greatly improving the processing efficiency of the tasks. By supplementing the task identifiers in the cache queue to the task queue, the task queue can always be in a full state, so that the subsequent to-be-processed tasks can always be processed in parallel in a preset number, thus further improving the processing efficiency of the tasks.

[0117] In one embodiment, the target processing interface is called, and the storage space is operated based on the task data according to the operation mode corresponding to the operation type, to execute the to-be-processed tasks, including: for each to-be-processed task, the target processing interface corresponding to the to-be-processed task is called, and the related storage space is operated based on the task data in the to-be-processed task according to the operation mode corresponding to the operation type to which the to-be-processed task belongs, to execute the to-be-processed tasks in parallel.

[0118] Specifically, for each to-be-processed task in the plurality of to-be-processed tasks, the server calls the target processing interface corresponding to the current to-be-processed task, and operates the related storage space based on the task data in the current to-be-processed task according to the operation mode corresponding to the operation type to which the current to-be-processed task belongs, to execute the current to-be-processed task. That is, the server can execute the to-be-processed tasks corresponding to the task identifiers in the task queue in parallel.

[0119] In one of the embodiments, after executing the to-be-processed tasks corresponding to the task identifiers in the task queue in parallel, the above data processing method further includes: deleting the task identifier of the to-be-processed task that has been successfully processed from the task queue; supplementing the task identifiers in the cache queue to the task queue, and executing the corresponding tasks based on the supplemented task identifiers.

[0120] When it is determined that the to-be-processed task is processed, the server can correspondingly delete the task identifier of the to-be-processed task that has been processed from the task queue. Further, when the server correspondingly deletes the task identifier of the to-be-processed task that has been processed from the task queue, the server can supplement the task identifiers in the cache queue to the task queue, so that the server continues to process the tasks corresponding to the supplemented task identifiers. For the convenience of description, the tasks corresponding to the supplemented task identifiers are referred to as supplemented tasks. It is easy to understand that the supplemented tasks can also be considered as to-be-processed tasks, so that the server can perform corresponding processing on the supplemented tasks according to the above steps S204 to S210.

[0121] In the embodiment, the task identifiers stored in the task queue can be quickly updated and iterated by deleting and supplementing the task identifiers in the task queue, so that the processing efficiency of the task processing is improved.

[0122] In one embodiment, the above data processing method further includes a task identifier storage step, which includes: receiving a user operation request, and generating a corresponding task based on the task data and the task identifier carried in the user operation request; storing the task data in a cache space, and storing the task identifier in the task queue when the first queue length is less than the preset queue threshold; and storing the task identifier in the cache queue when the first queue length is equal to the preset queue threshold.

[0123] Specifically, the target application can generate an operation request based on the user's operation and send the operation request to the server, so that the server generates a task based on the task data and the task identifier in the operation request. For example, when the user touches the data download control, the target application can generate an operation request according to the user's touch operation and send the operation request to the server, so that the server generates a data download task based on the task identifier and the to-be-downloaded data identifier. Further, in order to reduce the storage space occupied by the task queue, the server can store the task data and the task identifier in the generated task respectively. The server can store the task data in a cache space and store the task identifier in a task queue or a cache queue. The cache space refers to a storage space for storing task data, which can be a cache database. The cache space and the storage space can be the same storage space or can not be the same storage space.

[0124] When the server stores the task identifier in the task queue or the cache queue, the server determines whether the first queue length of the task queue is less than the preset queue threshold, that is, the server determines whether the task queue is full. If the first queue length is less than the preset queue threshold, it can be considered that the task queue is not full, and the server directly stores the task identifier in the task queue at this time. If the first queue length is equal to the preset queue threshold, it can be considered that the task queue is full, and the server stores the task identifier in the cache queue at this time.

[0125] In one embodiment, the server can store the task identifiers in the task queue or the cache queue according to the order of the generation time of the tasks. For example, when obtaining the task identifiers of multiple tasks, the server can store the task identifiers of the multiple tasks in the task queue or the cache queue according to the order of the generation time of the tasks.

[0126] In the above embodiments, in order to reduce the probability that the to-be-processed task cannot be normally executed due to insufficient computer resources, the queue length of the task queue can be controlled, so as to avoid the case that the to-be-processed task is constructed unlimitedly and the computer resources are exhausted based on the set queue length. In addition, by constructing the task queue and the cache queue, when the progress of executing the to-be-processed task is slow, the subsequent generated task can be cached in the cache queue, so as to reduce the probability of task loss.

[0127] In one embodiment, the above data processing method further includes a task restart recovery step, which includes: when it is determined that the to-be-processed task processing is abnormal, obtaining the task identifier stored in the task queue, and re-reading the corresponding task data from the cache space according to the task identifier; and re-executing the to-be-processed task based on the task data.

[0128] Specifically, during the execution of the task, network failure or device restart may occur, which causes the to-be-processed task corresponding to the task identifier in the task queue to be abnormal. For example, when the device is restarted, a plurality of to-be-processed tasks corresponding to the task identifier in the task queue may be lost, and for example, when the network fails, a plurality of to-be-processed tasks corresponding to the task identifier in the task queue may fail. Therefore, in order to reduce the probability of data processing failure caused by processing abnormality, the server can re-execute the to-be-processed task with processing abnormality through a task restart recovery mechanism.

[0129] When the to-be-processed task processing is abnormal, the server can read the task identifier of the to-be-processed task from the task queue, and read the task data corresponding to the task identifier from the cache space, and re-execute the to-be-processed task based on the task data.

[0130] In one of the embodiments, when there is a to-be-processed task with processing abnormality, it can be considered that a plurality of to-be-processed tasks corresponding to the task identifier in the task queue will fail, at this time, the server reads all the task identifiers stored in the task queue, and reads the corresponding task data from the cache space according to each task identifier, and re-executes the corresponding to-be-processed task based on each task data. For example, when the device is restarted, a plurality of to-be-processed tasks corresponding to the task identifier in the task queue may be lost, at this time, the server can re-execute a plurality of to-be-executed tasks through the task identifiers in the task queue.

[0131] In one of the embodiments, with reference to Figure 5 , Figure 5A schematic diagram of asynchronous task scheduling in an embodiment is shown. The asynchronous task scheduling takes a cache queue and a task queue as the core, and includes several modules such as queue initialization, task insertion, task acquisition, task execution, and task recovery. The queue initialization is mainly used to initialize the cache queue and the task queue, including determination of the queue length, generation of the queue, and the like. When the system consumes the task queue at a slow speed, the subsequent tasks can be buffered in the cache queue to avoid task loss, so the queue length of the cache queue is generally greater than that of the task queue. The task insertion is used to store the task identifier to the cache queue or the task queue according to the generation time of the task, and store the task data in the cache space. The expiration time of the task data can be set, so that when the expiration time is reached, the task data can be automatically deleted from the cache space, thereby reducing the waste of storage space. The task acquisition is mainly performed when the task queue is not empty, and a corresponding number of task identifiers can be acquired from the cache queue when the task queue is not full, and the task identifiers are inserted into the task queue in the order of the generation time of the tasks. The task execution is mainly used to execute the to-be-executed task, and when the to-be-executed task is executed, the task identifier of the executed to-be-processed task is deleted from the task queue. The task recovery is used to re-execute the to-be-executed task when the to-be-executed task is executed abnormally, by using the task identifier in the task queue and the task data in the cache space.

[0132] In the above embodiment, by re-executing the to-be-executed task with an abnormal processing, the execution success rate of the to-be-executed task can be improved, thereby improving the processing success rate of the data processing.

[0133] In an embodiment, the data storage type includes a data import type, the target processing interface is called, and the storage space is operated based on the task data according to the operation mode corresponding to the operation type, to execute the to-be-processed task, including: when the operation type corresponding to the to-be-processed task is the data import type, the target processing interface corresponding to the to-be-processed task is called to acquire the to-be-imported data, and the to-be-imported data is decompressed to obtain at least one to-be-imported decompressed data; and each to-be-imported decompressed data is stored in the storage space related to the task data in the to-be-processed task through the target processing interface corresponding to the to-be-processed task.

[0134] Specifically, when the to-be-processed task is a data import task, and correspondingly, the operation type corresponding to the to-be-processed task is a data import type, the server invokes a target processing interface corresponding to the data import task to obtain the to-be-imported data. That is, the server invokes a target processing interface matching the space type of the storage space used to store the to-be-imported data to obtain the to-be-imported data. For the convenience of description, the storage space used to store the to-be-imported data corresponding to the import space identifier is referred to as a to-be-imported storage space below. For example, when the user specifies to store the to-be-imported data in an object type storage space, that is, when the space type of the storage space corresponding to the import space identifier included in the data import task is an object storage type, the server obtains the to-be-imported data through an object processing interface matching the object type. When the user specifies to store the to-be-imported data in a file storage type storage space, the server obtains the to-be-imported data through a file processing interface corresponding to the file storage type.

[0135] Further, when the to-be-imported data is a compressed file, the server can perform decompression processing on the to-be-imported data and decompress the to-be-imported data into a preset decompression path to obtain at least one to-be-imported decompressed data. The server traverses the at least one to-be-imported decompressed data in the decompression file by invoking the target processing interface corresponding to the data import task, and stores each to-be-imported decompressed data into the to-be-imported storage space. For example, in the above example, when the to-be-imported data is obtained through the object processing interface, the server can store the to-be-imported decompressed data into the storage space of the object storage type through the object processing interface. When the to-be-imported data is obtained through the file processing interface, the server can store the to-be-imported decompressed data into the storage space of the file storage type through the file processing interface.

[0136] In one embodiment, when the to-be-imported data is decompressed into the decompression path, the server can traverse the to-be-imported decompressed data in the decompression path, determine the data information of the to-be-imported decompressed data, and store the data information of the to-be-imported decompressed data into a preset storage space. The data information of the to-be-imported decompressed data refers to information associated with the to-be-imported decompressed data, which can be, for example, the data amount of the to-be-imported decompressed data included in the decompression path, the data size and data format of the to-be-imported decompressed data, and the like.

[0137] In the above embodiments, by determining the target processing interface corresponding to the to-be-processed task, the to-be-imported data can be stored into the storage space specified based on the task data based on the target processing interface. In this way, the to-be-imported data can be stored into the corresponding storage space according to the user demand, thereby greatly improving the user experience.

[0138] In one embodiment, when the operation type corresponding to the to-be-processed task is a data import type, the target processing interface corresponding to the to-be-processed task is called to obtain the to-be-imported data, including: when it is determined based on the heartbeat packet sent by the user terminal that the user terminal has successfully uploaded the to-be-imported data, the target processing interface is called to obtain the to-be-imported data; wherein the heartbeat packet is an information packet generated by the user terminal based on the upload progress of the to-be-imported data.

[0139] Specifically, the user can upload the to-be-imported data to the server through the target application running in the user terminal. When the user expects to store the to-be-imported data through the server, the user can touch the data import control in the target application to make the target application display a data information input box, so that the user can upload the locally stored to-be-imported data through the data information input box, and then the user terminal can send the to-be-imported data uploaded by the user to the server.

[0140] In the process of sending the to-be-imported data to the server by the user terminal, the user terminal can monitor the sending situation of the to-be-imported data in real time through the heartbeat packet. The user terminal can generate a corresponding heartbeat packet according to the sending progress of the to-be-imported data at regular intervals, and send the heartbeat packet to the server, so that the server can determine the sending progress of the to-be-imported data sent by the user terminal according to the received heartbeat packet. When it is determined based on the heartbeat packet that the user terminal has successfully sent the to-be-imported data, the server can call the target processing interface corresponding to the to-be-processed task to download the to-be-imported data to the local server, and perform decompression processing on the to-be-imported data downloaded to the local to store the decompressed to-be-imported data in the corresponding storage space.

[0141] In one embodiment, the data import task is mainly to store the to-be-imported data stored in the local user terminal into an object storage type storage space, a file storage type storage space or a big data storage type storage space. The format of the to-be-imported data is mainly a compressed package, so the entire data import process includes uploading the compressed package locally by the user terminal, decompressing the compressed package by the server background, and storing the decompressed file. Referring to Figure 6S602 when the user terminal obtains the to-be-imported data, the user terminal can send the to-be-imported data to the server. S604 the user terminal can generate a heartbeat packet periodically and send the heartbeat packet to the server. S606 the server receives and parses the heartbeat packet, S608 determines whether the to-be-imported data has been successfully sent through the received heartbeat packet. S610 when it is determined that the user terminal has not successfully sent the to-be-imported data, the server extracts the current sending progress of the to-be-imported data from the heartbeat packet received at the current time, extracts the historical sending progress of the to-be-imported data from the historical heartbeat packet received at the previous time, and compares the current sending progress with the historical sending progress, S612 determines whether the sending process of the to-be-imported data has stopped according to the comparison result. S614 when it is determined that the sending process of the to-be-imported data has not stopped due to failure based on the comparison progress, the server continues to receive subsequent heartbeat packets. S616 when it is determined that the sending process of the to-be-imported data has stopped due to failure based on the comparison progress, the server determines that the to-be-imported data fails to be sent. S618 when it is determined that the user terminal successfully sends the to-be-imported data, the server calls a corresponding target processing interface to obtain the to-be-imported data, and downloads the to-be-imported data to the local server. S620 the server decompresses the to-be-imported data downloaded to the local server to obtain at least one to-be-imported decompressed data, S622 traverses the to-be-imported decompressed data to obtain data information of the to-be-imported decompressed data, and stores the data information of the to-be-imported decompressed data in a preset storage space. S624 the server stores the to-be-imported decompressed data in a to-be-imported storage space through a corresponding target processing interface. The preset storage space and the to-be-imported storage space can be the same storage space or different storage spaces. The to-be-imported storage space can be an object storage type storage space, a file storage type storage space or a big data storage type storage space. The preset storage space can be a general type storage space, for example, a MySQL database. Figure 6 A flowchart of performing a data import task in an embodiment is shown.

[0142] In the above embodiment, by introducing the heartbeat mechanism, the sending progress of the to-be-imported data can be reported through the heartbeat mechanism, and the probability that the user terminal is stuck and unresponsive due to network, browser and other reasons while the server is still in a waiting state can be reduced, that is, the occurrence of invalid waiting of the server is reduced.

[0143] In an embodiment, the data storage type includes a data migration type; the task data includes a to-be-migrated data identifier, a source space identifier, and a migration space identifier; the target processing interface includes a source processing interface corresponding to the source space identifier and a migration processing interface corresponding to the migration space identifier; and the calling the target processing interface and performing the to-be-processed task on the storage space according to the operation mode corresponding to the operation type based on the task data includes: when the operation type corresponding to the to-be-processed task is the data migration type, calling the source processing interface and obtaining the to-be-migrated data from the source storage space corresponding to the source space identifier based on the to-be-migrated data identifier; and calling the migration processing interface and migrating the to-be-migrated data to the migration space corresponding to the migration space identifier.

[0144] Specifically, when the to-be-processed task is a data migration task, the task data of the to-be-processed task can include a migration data identifier, a source space identifier, and a migration space identifier. The server can determine the space type of the source storage space storing the to-be-migrated data according to the source space identifier, and determine the corresponding source processing interface based on the space type of the source storage space. The server can determine the space type of the migration space used to migrate the to-be-migrated data according to the migration space identifier, and determine the corresponding migration processing interface based on the space type of the migration space.

[0145] Further, when the data migration task needs to be executed, the server extracts the to-be-migrated data from the source storage space corresponding to the source space identifier according to the migration data identifier by calling the source processing interface, and stores the extracted to-be-migrated data in the migration space corresponding to the migration space identifier by calling the migration processing interface. The space types of the migration space and the source storage space can be the same or different. For example, a user can migrate data in an object storage type storage space to a big data storage type storage space according to the user's own needs, or migrate data in an object storage type A storage space to an object storage type B storage space. That is, the server can be deployed with multiple storage spaces belonging to the same space type.

[0146] In one of the embodiments, reference is made to Figure 7 , Figure 7A flowchart of performing a data migration task in an embodiment is shown. S702 When a data migration task needs to be performed, the server determines the data to be migrated by calling the source processing interface, S704 and judges the data type of the data to be migrated. S706 When it is determined based on the data type of the data to be migrated that the data to be migrated is a data table, the server determines a local non-compressed data storage path for storing non-compressed data, and downloads the data table to the non-compressed data storage path through the source processing interface. S708 When it is determined based on the data type of the data to be migrated that the data to be migrated is not a data table, the server stores the compressed type of the data to be migrated to the local compressed data storage path, and stores the non-compressed type of the data to be migrated to the non-compressed data storage path. S710 The server traverses the non-compressed data storage path and determines the data information of the data to be migrated stored in the non-compressed path, S712 stores the data information of the data to be migrated in the preset storage space. S714 The server stores the data to be migrated stored in the non-compressed path to the migration space by calling the migration processing interface. S716 The server traverses the compressed data storage path and decompresses the data to be migrated stored in the compressed data storage path to obtain at least one decompressed data to be migrated. S718 The server traverses the decompressed data to be migrated and determines the data information of the decompressed data to be migrated, and stores the data information of the decompressed data to be migrated in the preset storage space. S720 The server calls the migration processing interface to migrate the decompressed data to be migrated to the migration space.

[0147] In the above embodiment, the existing system only provides the most basic data import function, and cannot realize the data migration function. The embodiment of the application integrates multiple types of processing interfaces, so that users can migrate data in multiple different storage spaces according to their own needs, which not only improves the user experience, but also meets the increasingly expanding data migration needs of current machine learning training.

[0148] In one embodiment, the task data includes a space to be arranged identifier; a target processing interface is called, and the storage space is operated based on the task data in accordance with the operation mode corresponding to the operation type, to execute the task to be processed, including: when the operation type corresponding to the task to be processed is a data arrangement type, a target processing interface corresponding to the task to be processed is called, and a space to be arranged corresponding to the space to be arranged identifier is accessed; the non-compressed data stored in the space to be arranged is traversed through the target processing interface to read the first data information of the non-compressed data; and the first data information is stored in a preset storage space.

[0149] Specifically, when the to-be-processed task is a data arrangement task and the operation type of the to-be-processed task is a data arrangement type, the server can extract the to-be-arranged space identifier from the task data of the to-be-processed task, and determine a to-be-arranged space corresponding to the to-be-arranged space identifier. Further, the server can determine a target processing interface corresponding to the space type of the to-be-arranged space, and access the to-be-arranged space through the target processing interface to traverse the uncompressed data stored in the to-be-arranged space. The uncompressed data refers to data in a non-compressed state. For each uncompressed data stored in the to-be-arranged space, the server can determine data-related information of the current uncompressed data, such as data size, data format, and data storage time, and after the traversal is completed, the data-related information of each uncompressed data is summarized to obtain first data information. The first data information refers to information associated with the uncompressed data, and the first data information can specifically include the total data amount, the total data size, and the data-related information of each uncompressed data. Further, when the first data information is obtained, the server can store the first data information in a preset storage space.

[0150] In one of the embodiments, when the user expects to visually display the data-related information in the to-be-arranged space, the server can read the first data information from the preset storage space, and send the first data information to the user terminal, so that the user terminal visually displays the first data information.

[0151] In the above embodiments, by obtaining the first data information and storing the first data information in the preset storage space, the first data information can be visually displayed subsequently.

[0152] In one of the embodiments, the task data includes the to-be-arranged space identifier; the target processing interface is called, and the storage space is operated based on the task data in the operation mode corresponding to the operation type, to execute the to-be-processed task, including: when the operation type corresponding to the to-be-processed task is a data arrangement type, the target processing interface corresponding to the to-be-processed task is called to access the to-be-arranged space corresponding to the to-be-arranged space identifier; the compressed data stored in the to-be-arranged space is downloaded through the target processing interface, and the downloaded compressed data is decompressed to obtain at least one to-be-arranged decompressed data; the to-be-arranged decompressed data is traversed to determine second data information of the to-be-arranged decompressed data, and the second data information is stored in a preset storage space.

[0153] Specifically, when the task to be processed is a data processing task, and the operation type of the task is data processing, the server can extract the space identifier to be processed from the task data of the task to be processed, and determine the space to be processed corresponding to the space identifier. Further, the server can determine the target processing interface corresponding to the space type of the space to be processed, and access the space to be processed through the target processing interface to traverse the compressed data stored in the space to be processed. Here, compressed data refers to data in a compressed state. For each compressed data stored in the space to be processed, the server can download the current compressed data to the server's local machine through the target processing interface, and decompress the downloaded compressed data to obtain at least one decompressed data to be processed.

[0154] For each of the multiple data sets to be consolidated and decompressed, the server can determine its data size, data format, and data storage time, among other data-related information. After determining this information, the server summarizes and processes the relevant data information of each data set to obtain second data information. This second data information refers to information associated with the data to be consolidated and decompressed, and may specifically include the total amount of data to be consolidated and decompressed, the total data size, and relevant data information for each data set. Furthermore, upon obtaining the second data information, the server can store it in a preset storage space.

[0155] In one embodiment, reference Figure 8 , Figure 8 A flowchart illustrating the data processing task in one embodiment is shown. S802 When data processing is required in the space to be processed, the server can call the target processing interface. S804 The server iterates through the uncompressed data in the space to be processed via the target processing interface to obtain first data information. S806 The server stores the first data information in a preset storage space. S808 The server can also download compressed data stored in the space to be processed via the target processing interface. S810 The server decompresses the downloaded compressed data to obtain decompressed data to be processed. S812 The server iterates through the decompressed data to obtain second data information. S814 The server stores the second data information in the preset storage space.

[0156] In the above embodiments, by acquiring the second data information, the second data information can be stored in a preset storage space, thereby facilitating the subsequent visualization of the second data information.

[0157] In one embodiment, the above data processing method further comprises a label data processing step, which comprises: when the task data comprises label data, checking the data format and data content of the label data to obtain a checking result; and when the checking result indicates that the checking passes, performing an operation on the label database based on the label data in accordance with an operation mode corresponding to the operation type to which the task to be processed belongs, so as to execute the task to be processed.

[0158] Specifically, when the task data is obtained, the server can determine whether the task data comprises label data, and when it is determined that the task data comprises label data, the server performs data format checking and data content checking on the label data by using a preset checking rule. The label data refers to label information annotated by a model training personnel in training data during model training. Further, when it is determined that the data format checking and the data content checking both pass, the server performs an operation on the label database based on the label data in accordance with an operation mode corresponding to the operation type to which the task to be processed belongs, so as to execute the task to be processed. For example, when the operation type is a data import type, the server can store the label data in the label database. By storing the label data in the label database separately, the model training personnel can modify and update the label data subsequently.

[0159] In one embodiment, the server can determine whether the task data comprises label data, and when it is determined that the task data comprises label data, the server performs data format checking and data content checking on the label data by using a preset checking rule. Figure 9 Figure 9 A processing flow of label data in one embodiment is shown. When it is necessary to process label data, the server can obtain task data S902, and determine whether the task data comprises label data S904. When the task data comprises label data, it can be considered that the task to be processed is a task of importing data of the label data, and the server checks the label format of the label data S908 and checks the label content of the label data S910. The server determines whether the checking passes S912, ends the execution of the task to be executed when the checking does not pass S914, and determines the task type of the task to be processed when the checking passes S916. When the task data does not comprise label data, the server can directly determine the task type of the task to be processed. When it is determined based on the task type that the task to be processed is a task of importing data of the label data, the server stores the label data in the label database S922. When it is determined based on the task type that the task to be processed is a task of transferring data of the label data, the server transfers the label data from the current label database to the target label database S918. When it is determined based on the task type that the task to be processed is a task of arranging data of the label data, the server arranges the label database and caches the arrangement result in the label database S920. When it is determined that the task to be processed is completed, the server updates the task state of the task to be processed from an unprocessed state to a completed state S924.​

[0160] In the above embodiment, by storing the label data in the label database separately, subsequent model training personnel can conveniently arrange, modify and update the label data.

[0161] The application further provides an application scenario applying the above data processing method. Specifically, the data processing method is applied in the application scenario as follows:

[0162] Reference Figure 10 , Figure 10 A schematic diagram of a data processing system in an embodiment is shown, wherein the data processing system comprises a data center for storing training data, label data, retail data or operation and maintenance data, and the training data, label data, retail data or operation and maintenance data can be pictures, texts, compressed packages or structure files. The data center integrates storage spaces provided by COS, CEPHFS, HDFS or local disks, and the training data, label data, retail data or operation and maintenance data can be stored through the storage spaces. The data center can execute the pending tasks through the task queue cache to realize the download, import, arrangement, release and deletion of data. When executing the pending tasks, the server can operate the corresponding storage spaces through the target processing interface, and clean up the executed pending tasks when the execution of the pending tasks is completed, so as to reduce the storage space occupied by the entire data processing system. It is easy to understand that each application platform can execute tasks through the data processing center, for example, the training platform can perform model training through the training data stored in the data center, and the operation and maintenance platform can perform operation and maintenance processing through the operation and maintenance data stored in the data center.

[0163] Since the data can be imported, stored, downloaded or arranged through the data processing system, the processing types of data processing are fully widened, the data pollution is avoided, and the diversified requirements of algorithms, training, labeling and the like are met. In addition, since the system integrates various types of storage spaces and processing interfaces, the user can conveniently and quickly process data based on the processing interface without paying attention to the underlying processing logic.

[0164] The application further provides an application scenario applying the above data processing method.

[0165] Specifically, the data processing method is applied in the application scenario as follows:

[0166] In the field of intelligent retail, when the customer flow in the shopping mall needs to be counted, the customer flow counting platform can extract training data from the data center, and train the face recognition model based on the training data, so that the customer flow counting platform can perform face recognition on the customers entering and leaving the shopping mall based on the trained face recognition model, and determine the customer flow in a preset time period based on the face recognition result.

[0167] The above application scenarios are only illustrative, and it can be understood that the application of the data processing method provided by the embodiments of the present application is not limited to the above scenarios.

[0168] In one specific embodiment, with reference to Figure 11 The above data processing method comprises:

[0169] S1102, obtaining a current task queue and determining a first queue length of the task queue.

[0170] S1104, when the first queue length is less than a preset queue threshold, determining a second queue length of the cache queue; when the second queue length is not zero, according to the difference between the first queue length and the preset queue threshold, transferring the difference number of task identifiers stored in the cache queue to the task queue.

[0171] S1106, the tasks corresponding to the task identifiers in the task queue are respectively and uniformly regarded as to-be-processed tasks; the to-be-processed tasks include task data; the to-be-processed tasks at least include one of a data download task, a data import task, a data transfer task, a data access task, and a data arrangement task.

[0172] S1108, determining an operation type corresponding to the to-be-processed task; the operation type at least includes one of a data download type, a data storage type, a data access type, and a data arrangement type;

[0173] S1110, when the to-be-processed task is a data download task, according to the to-be-downloaded data identifier, determining a space type of a storage space storing the to-be-downloaded data.

[0174] S1112, when the to-be-processed task is a data import task, according to the import space identifier, determining a space type of a storage space for storing the to-be-imported data.

[0175] S1114, when the to-be-processed task is a data transfer task, according to the source space identifier, determining a space type of a source storage space currently storing the to-be-transferred data, and according to the transfer space identifier, determining a space type of a transfer space for transferring the to-be-transferred data.

[0176] S1116, when the to-be-processed task is a data access task, a storage space type of a storage space storing to-be-accessed data is determined according to the access data identifier.

[0177] S1118, when the to-be-processed task is a data arrangement task, a storage space type of a to-be-arranged space is determined according to the arrangement space identifier; the storage space type includes an object storage type, a file storage type and a big data storage type.

[0178] S1120, a target processing interface matched with the storage space type is determined; the target processing interface is designed based on a communication protocol supported by the storage space type.

[0179] S1122, when an operation type corresponding to the to-be-processed task is a data import type, the target processing interface corresponding to the to-be-processed task is called to obtain to-be-imported data, and the to-be-imported data is decompressed to obtain at least one to-be-imported decompressed data; each to-be-imported decompressed data is stored in a storage space related to task data in the to-be-processed task through the target processing interface.

[0180] S1124, when an operation type corresponding to the to-be-processed task is a data transfer type, the source processing interface is called, and to-be-transferred data is obtained from a source storage space corresponding to the source space identifier based on the to-be-transferred data identifier; the transfer processing interface is called to transfer the to-be-transferred data to a transfer storage space corresponding to the transfer space identifier.

[0181] S1126, when an operation type corresponding to the to-be-processed task is a data arrangement type, the target processing interface corresponding to the to-be-processed task is called to access a to-be-arranged space corresponding to the to-be-arranged space identifier; the target processing interface is used to traverse non-compressed data and compressed data stored in the to-be-arranged space to determine first data information of the non-compressed data and second data information of the compressed data; the first data information and the second data information are stored in a preset storage space.

[0182] The above data processing method can determine an operation type of a to-be-processed task and obtain task data by obtaining the to-be-processed task. The task data is obtained, and a storage space type associated with the task data is determined, so that a target processing interface matched with the determined storage space type is determined. The target processing interface is determined, and the corresponding storage space is operated based on the task data according to an operation mode corresponding to the operation type based on the target processing interface, so as to execute the to-be-processed task. Since the target processing interface is determined based on the storage space type corresponding to the task data, the application can operate the storage space of the corresponding storage space type through different types of processing interfaces. Compared with the traditional method of supporting storage of to-be-processed data in the same type of storage space, the application greatly improves the diversity of data operation.

[0183] It should be understood that although Figure 2 the steps in the flowcharts of Figure 11 are shown in sequential order, such that one step necessarily occurs before another, the steps do not have to be carried out in the order shown. Unless explicitly stated, the steps can be carried out in any order, and need not be carried out sequentially. Furthermore, Figure 2 at least some of the steps in Figure 11 may include multiple steps or stages, which need not necessarily be carried out at the same time, and which need not necessarily be carried out sequentially, but can be carried out in rotation or alternation with other steps or stages in other steps.

[0184] In one embodiment, as shown in Figure 12 a data processing apparatus 1200 is provided, which can be a part of a computer device, and can be implemented by software modules or hardware modules, or a combination of both, and specifically includes a task obtaining module 1202, an interface determining module 1204, and an executing module 1206, wherein:

[0185] The task obtaining module 1202 is configured to obtain a to-be-processed task; the to-be-processed task includes task data.

[0186] The interface determining module 1204 is configured to determine an operation type corresponding to the to-be-processed task; the operation type includes at least one of a data downloading type, a data storage type, a data access type, and a data arrangement type; determine a space type to which a storage space related to the task data belongs; determine a target processing interface matched with the space type; and the target processing interface is designed based on a communication protocol supported by the space type.

[0187] The executing module 1206 is configured to call the target processing interface, and perform an operation on the storage space based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task.

[0188] In one embodiment, referring to Figure 13 the task obtaining module 1202 further includes a task scheduling module 1221, configured to obtain a current task queue, and determine a first queue length of the task queue; when the first queue length is less than a preset queue threshold, determine a second queue length of a cache queue; when the second queue length is not zero, according to a difference between the first queue length and the preset queue threshold, transfer a difference number of task identifiers stored in the cache queue to the task queue; and the tasks corresponding to the task identifiers in the task queue are respectively taken as to-be-processed tasks.

[0189] In an embodiment, the data processing apparatus 1200 is further configured to, for each to-be-processed task, invoke a target processing interface corresponding to the to-be-processed task, and perform the to-be-processed task in parallel according to an operation mode corresponding to an operation type to which the to-be-processed task belongs, based on task data in the to-be-processed task.

[0190] In an embodiment, the data processing apparatus 1200 is further configured to delete, from the task queue, a task identifier of a to-be-processed task whose processing is successful, and supplement the task identifier in the cache queue to the task queue, and perform a corresponding task based on the supplemented task identifier.

[0191] In an embodiment, the task scheduling module 1221 is further configured to receive a user operation request, and generate a corresponding task based on task data and a task identifier carried in the user operation request; store the task data in the cache space, and when the first queue length is less than the preset queue threshold, store the task identifier in the task queue; and when the first queue length is equal to the preset queue threshold, store the task identifier in the cache queue.

[0192] In an embodiment, the task obtaining module 1202 is further configured to, when determining that the to-be-processed task processing is abnormal, obtain a task identifier stored in the task queue, and re-read corresponding task data from the cache space according to the task identifier; and perform the to-be-processed task again based on the task data.

[0193] In an embodiment, the data processing apparatus 1200 includes a to-be-processed task, and the to-be-processed task includes at least one of a data download task, a data import task, a data transfer task, a data access task, and a data arrangement task; task data corresponding to the data download task includes to-be-downloaded data identifier; task data corresponding to the data import task includes import space identifier; task data corresponding to the data transfer task includes source space identifier and transfer space identifier; task data corresponding to the data access task includes access data identifier; and task data corresponding to the data arrangement task includes arrangement space identifier.

[0194] In an embodiment, the interface determining module 1204 comprises a space type determining module 1241 configured to, when the task to be processed is a data download task, determine the space type of the storage space storing the data to be downloaded according to the data to be downloaded identifier; when the task to be processed is a data import task, determine the space type of the storage space for storing the data to be imported according to the import space identifier; when the task to be processed is a data transfer task, determine the space type of the source storage space currently storing the data to be transferred according to the source space identifier, and determine the space type of the transfer space for transferring the data to be transferred according to the transfer space identifier; when the task to be processed is a data access task, determine the space type of the storage space storing the data to be accessed according to the access data identifier; and when the task to be processed is a data arrangement task, determine the space type of the arrangement space according to the arrangement space identifier.

[0195] In an embodiment, the space types in the data processing apparatus 1200 comprise an object storage type, a file storage type and a big data storage type; the target processing interface matching the object storage type is an object processing interface designed based on a simple storage service communication protocol; the target processing interface matching the file storage type is a file processing interface designed based on a basic read-write communication protocol; and the target processing interface matching the big data storage type is a big data processing interface designed based on a distributed file system communication protocol.

[0196] In an embodiment, the data storage types in the data processing apparatus 1200 comprise a data import type, and the execution module 1206 is further configured to, when the operation type corresponding to the task to be processed is the data import type, call the target processing interface corresponding to the task to be processed to obtain the data to be imported, and perform decompression processing on the data to be imported to obtain at least one piece of decompressed data to be imported; and store each piece of decompressed data to be imported to the storage space related to the task data in the task to be processed through the target processing interface.

[0197] In an embodiment, the execution module 1206 is further configured to, when it is determined that the user terminal has successfully uploaded the data to be imported based on the heartbeat packet sent by the user terminal, call the target processing interface corresponding to the task to be processed to obtain the data to be imported; wherein the heartbeat packet is an information packet generated by the user terminal based on the uploading progress of the data to be imported.

[0198] In an embodiment, the data storage type comprises a data migration type; the task data comprises a data to be migrated identifier, a source space identifier, and a migration space identifier; the target processing interface comprises a source processing interface corresponding to the source space identifier, and a migration processing interface corresponding to the migration space identifier; the execution module 1206 is further configured to, when the operation type corresponding to the task to be processed is the data migration type, invoke the source processing interface, and acquire the data to be migrated from a source storage space corresponding to the source space identifier based on the data to be migrated identifier; and invoke the migration processing interface to migrate the data to be migrated to a migration space corresponding to the migration space identifier.

[0199] In an embodiment, the task data comprises a space to be organized identifier; the execution module 1206 is further configured to, when the operation type corresponding to the task to be processed is the data organization type, invoke the target processing interface corresponding to the task to be processed, access a space to be organized corresponding to the space to be organized identifier, and traverse the uncompressed data stored in the space to be organized through the target processing interface to determine first data information of the uncompressed data; and store the first data information in a preset storage space.

[0200] In an embodiment, the execution module 1206 is further configured to, when the operation type corresponding to the task to be processed is the data organization type, invoke the target processing interface corresponding to the task to be processed, access a space to be organized corresponding to the space to be organized identifier, download the compressed data stored in the space to be organized through the target processing interface, and decompress the downloaded compressed data to obtain at least one decompressed data to be organized; traverse the decompressed data to be organized to determine second data information of the decompressed data to be organized, and store the second data information in a preset storage space.

[0201] In an embodiment, the data processing apparatus 1200 is further configured to, when the task data comprises label data, verify a data format and a data content of the label data to obtain a verification result; and when the verification result indicates that the verification is passed, perform an operation on a label database based on the label data according to an operation mode corresponding to the operation type to which the task to be processed belongs, to execute the task to be processed.

[0202] The specific limitations of the data processing apparatus can be referred to the limitations of the data processing method in the above, which will not be repeated here. Each module in the above data processing apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0203] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as follows:Figure 14 The computer device shown in the figure includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data processing data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a data processing method.

[0204] Those skilled in the art can understand that, Figure 14 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0205] In one embodiment, a computer device is also provided, including a memory and a processor, the memory having different types of storage spaces disposed therein, and the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0206] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0207] In one embodiment, a computer readable storage medium is provided, storing a computer program, the computer program being executed by a processor to implement the steps in the above method embodiments.

[0208] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps in the above method embodiments.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0210] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0211] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a to-be-processed task; the to-be-processed task is a task generated according to a task identifier and task data; determining a task type according to the task identifier in the to-be-processed task, and determining a corresponding operation type according to the task type; the operation type comprises a data download type, a data storage type, a data access type, and a data arrangement type; determining a space type to which a storage space related to the task data belongs according to the task data; the storage space comprises an object storage type space provided by a cloud object storage system, a file storage type space provided by a file storage system, and a big data storage type space provided by a distributed file system; determining a target processing interface matched with the space type; the target processing interface matched with the object storage type is an object processing interface designed based on a simple storage service communication protocol; the target processing interface matched with the file storage type is a file processing interface designed based on a basic read-write communication protocol; and the target processing interface matched with the big data storage type is a big data processing interface designed based on a distributed file system communication protocol; calling the target processing interface, and performing operation on the storage space based on the task data in a manner corresponding to the operation type, so as to execute the to-be-processed task; the data storage type comprises a data transfer type; the task data comprises a to-be-transferred data identifier, a source space identifier, and a transfer space identifier; the target processing interface comprises a source processing interface corresponding to the source space identifier and a transfer processing interface corresponding to the transfer space identifier; the calling the target processing interface and performing operation on the storage space based on the task data in a manner corresponding to the operation type so as to execute the to-be-processed task comprises: when the data operation type corresponding to the to-be-processed task is the data transfer type and the to-be-processed task is a data transfer task, when it is determined based on the data type of the to-be-transferred data in the source space that the to-be-transferred data is a data table, performing download processing on the data table through the source processing interface to download the data table into a non-compressed data storage path; when it is determined that the to-be-transferred data is not a data table, storing the to-be-transferred data of a compressed type into a compressed data storage path and storing the to-be-transferred data of a non-compressed type into a non-compressed data storage path through the source processing interface; traversing the non-compressed data storage path, calling the transfer processing interface, and transferring the to-be-transferred data stored in the non-compressed data storage path to a transfer space; traversing the compressed data storage path, decompressing the to-be-transferred data stored in the compressed data storage path to obtain at least one to-be-transferred decompressed data, traversing the to-be-transferred decompressed data, and calling the transfer processing interface to transfer the to-be-transferred decompressed data to the transfer space.

2. The method of claim 1, wherein, the acquiring the to-be-processed task comprises: acquiring a current task queue and determining a first queue length of the task queue; when the first queue length is less than a preset queue threshold, determining a second queue length of a cache queue; When the second queue length is not zero, a difference number of task identifiers stored in the cache queue is stored in the task queue according to a difference between the first queue length and a preset queue threshold; The tasks corresponding to the task identifiers in the task queue are respectively taken as to-be-processed tasks.

3. The method of claim 2, wherein, The calling the target processing interface and performing operation on the storage space according to the operation mode corresponding to the operation type based on the task data to execute the to-be-processed task includes: For each to-be-processed task, the target processing interface corresponding to the to-be-processed task is called, and the related storage space is operated based on the task data in the to-be-processed task according to the operation mode corresponding to the operation type to which the to-be-processed task belongs, so that the to-be-processed task is executed in parallel.

4. The method of claim 2, wherein, The method further includes: Receiving a user operation request and generating a corresponding task based on task data and a task identifier carried in the user operation request; Storing the task data in a cache space and storing the task identifier in the task queue when the first queue length is less than the preset queue threshold; When the first queue length is equal to the preset queue threshold, the task identifier is stored in the cache queue.

5. The method of claim 4, wherein, The method further includes: When it is determined that the to-be-processed task processing is abnormal, the task identifier stored in the task queue is obtained, and the corresponding task data is re-read from the cache space according to the task identifier; Re-executing the to-be-processed task based on the task data.

6. The method of claim 1, wherein, The to-be-processed task further includes a data download task, a data import task, a data access task, and a data arrangement task; the task data corresponding to the data download task includes a to-be-downloaded data identifier; the task data corresponding to the data import task includes an import space identifier; the task data corresponding to the data access task includes an access data identifier; and the task data corresponding to the data arrangement task includes an arrangement space identifier.

7. The method of claim 6, wherein, The determining the space type of the storage space related to the task data according to the task data includes: When the to-be-processed task is a data download task, the space type of the storage space storing the to-be-downloaded data is determined according to the to-be-downloaded data identifier; When the to-be-processed task is a data import task, the space type of the storage space for storing the to-be-imported data is determined according to the import space identifier; When the to-be-processed task is a data transfer task, the space type of the source storage space currently storing the to-be-transferred data is determined according to the source space identifier, and the space type of the transfer space for transferring the to-be-transferred data is determined according to the transfer space identifier; When the to-be-processed task is a data access task, the space type of the storage space storing the to-be-accessed data is determined according to the access data identifier; When the to-be-processed task is a data arrangement task, the space type of the to-be-arranged space is determined according to the arrangement space identifier.

8. The method of claim 1, wherein, The data storage type includes a data import type, the target processing interface is called, and the storage space is operated based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task, and the method comprises the following steps of: When the operation type corresponding to the to-be-processed task is a data import type, a target processing interface corresponding to the to-be-processed task is called to obtain to-be-imported data, and the to-be-imported data is decompressed to obtain at least one to-be-imported decompressed data; Each to-be-imported decompressed data is stored in a storage space related to the task data in the to-be-processed task through the target processing interface.

9. The method of claim 8, wherein, When the operation type corresponding to the to-be-processed task is a data import type, a target processing interface corresponding to the to-be-processed task is called to obtain to-be-imported data, and the to-be-imported data is decompressed to obtain at least one to-be-imported decompressed data; When it is determined based on a heartbeat packet sent by the user terminal that the user terminal has successfully uploaded the to-be-imported data, the target processing interface corresponding to the to-be-processed task is called to obtain the to-be-imported data; wherein the heartbeat packet is an information packet generated by the user terminal based on the uploading progress of the to-be-imported data.

10. The method of claim 1, wherein, The task data comprises a to-be-arranged space identifier; the target processing interface is called, and the storage space is operated based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task, and the method comprises the following steps of: When the operation type corresponding to the to-be-processed task is a data arrangement type, a target processing interface corresponding to the to-be-processed task is called to access a to-be-arranged space corresponding to the to-be-arranged space identifier; The first data information of the non-compressed data stored in the to-be-arranged space is determined through the target processing interface; The first data information is stored in a preset storage space.

11. The method of claim 1, wherein, The task data comprises a to-be-arranged space identifier; the target processing interface is called, and the storage space is operated based on the task data in an operation mode corresponding to the operation type, so as to execute the to-be-processed task, and the method comprises the following steps of: When the operation type corresponding to the to-be-processed task is a data arrangement type, a target processing interface corresponding to the to-be-processed task is called to access a to-be-arranged space corresponding to the to-be-arranged space identifier; The first data information of the non-compressed data stored in the to-be-arranged space is determined through the target processing interface; The second data information of the to-be-arranged decompressed data is determined by traversing the to-be-arranged decompressed data, and the second data information is stored in a preset storage space.

12. The method according to any one of claims 1 to 11, characterized in that, The method further comprises the following steps of: When the task data comprises label data, the data format and the data content of the label data are verified to obtain a verification result; When the verification result indicates that the verification is passed, a label database is operated based on the label data in an operation mode corresponding to the operation type to which the to-be-processed task belongs, so as to execute the to-be-processed task.

13. A data processing apparatus, characterized by: The device comprises: The task acquisition module is configured to acquire a to-be-processed task; the to-be-processed task is a task generated according to a task identifier and task data; The interface determination module is configured to determine a task type according to the task identifier in the to-be-processed task, and determine a corresponding operation type according to the task type; the operation type includes a data download type, a data storage type, a data access type, and a data arrangement type; determine a space type to which a storage space related to the task data belongs according to the task data; the storage space includes an object storage type space provided by a cloud object storage system, a file storage type space provided by a file storage system, and a big data storage type space provided by a distributed file system; determine a target processing interface matched with the space type; the target processing interface matched with the object storage type is an object processing interface designed based on a simple storage service communication protocol; the target processing interface matched with the file storage type is a file processing interface designed based on a basic read-write communication protocol; the target processing interface matched with the big data storage type is a big data processing interface designed based on a distributed file system communication protocol; The execution module is configured to call the target processing interface, and perform an operation on the storage space based on the task data in a manner corresponding to the operation type, so as to execute the to-be-processed task; the data storage type includes a data transfer type; the task data includes a to-be-transferred data identifier, a source space identifier, and a transfer space identifier; the target processing interface includes a source processing interface corresponding to the source space identifier and a transfer processing interface corresponding to the transfer space identifier; when the data operation type corresponding to the to-be-processed task is the data transfer type and the to-be-processed task is a data transfer task, if it is determined that the to-be-transferred data is a data table based on the data type of the to-be-transferred data in the source space, the source processing interface is used to perform download processing on the data table to download it to a non-compressed data storage path; if it is determined that the to-be-transferred data is not a data table, the source processing interface is called to store compressed to-be-transferred data in a compressed data storage path and store non-compressed to-be-transferred data in a non-compressed data storage path; the non-compressed data storage path is traversed, the transfer processing interface is called to transfer the to-be-transferred data stored in the non-compressed data storage path to a transfer space; the compressed data storage path is traversed, and the to-be-transferred data stored in the compressed data storage path is decompressed to obtain at least one to-be-transferred decompressed data; the to-be-transferred decompressed data is traversed, and the transfer processing interface is called to transfer the to-be-transferred decompressed data to the transfer space.

14. The apparatus of claim 13, wherein, The task acquisition module is further configured to acquire a current task queue and determine a first queue length of the task queue; when the first queue length is less than a preset queue threshold, determine a second queue length of a cache queue; When the second queue length is not zero, a difference number of task identifiers stored in the cache queue is stored in the task queue according to a difference between the first queue length and a preset queue threshold; and tasks corresponding to the task identifiers in the task queue are all taken as to-be-processed tasks.

15. The apparatus of claim 14, wherein, The execution module is further configured to invoke a target processing interface corresponding to each to-be-processed task, and perform the to-be-processed task in parallel by operating a relevant storage space according to an operation mode corresponding to an operation type to which the to-be-processed task belongs based on task data in the to-be-processed task.

16. The apparatus of claim 14, wherein, The task obtaining module is further configured to receive a user operation request, and generate a corresponding task based on task data and a task identifier carried in the user operation request; store the task data in a cache space, and store the task identifier in the task queue when the first queue length is less than the preset queue threshold; and store the task identifier in the cache queue when the first queue length is equal to the preset queue threshold.

17. The apparatus of claim 16, wherein, The task obtaining module is further configured to obtain a task identifier stored in the task queue when it is determined that the to-be-processed task processing is abnormal, and re-read corresponding task data from the cache space according to the task identifier; re-execute the to-be-processed task based on the task data.

18. The apparatus of claim 13, wherein, The to-be-processed task further includes one of a data download task, a data import task, a data access task, and a data arrangement task; the task data corresponding to the data download task includes a to-be-downloaded data identifier; the task data corresponding to the data import task includes an import space identifier; the task data corresponding to the data access task includes an access data identifier; and the task data corresponding to the data arrangement task includes an arrangement space identifier.

19. The apparatus of claim 18, wherein, The interface determining module is further configured to determine a space type of a storage space storing to-be-downloaded data according to the to-be-downloaded data identifier when the to-be-processed task is the data download task; and determine a space type of a storage space storing to-be-imported data according to the import space identifier when the to-be-processed task is the data import task. When the to-be-processed task is the data transfer task, determine a space type of a source storage space currently storing to-be-transferred data according to the source space identifier, and determine a space type of a transfer storage space used for transferring the to-be-transferred data according to the transfer space identifier; When the to-be-processed task is the data access task, determine a space type of a storage space storing to-be-accessed data according to the access data identifier; and when the to-be-processed task is the data arrangement task, determine a space type of a to-be-arranged space according to the arrangement space identifier. When the to-be-processed task is the data access task, determine a space type of a storage space storing to-be-accessed data according to the access data identifier; and when the to-be-processed task is the data arrangement task, determine a space type of a to-be-arranged space according to the arrangement space identifier.

20. The apparatus of claim 13, wherein, The data storage type includes a data import type, and the execution module is further configured to, when the operation type corresponding to the to-be-processed task is the data import type, call a target processing interface corresponding to the to-be-processed task to obtain to-be-imported data, and perform decompression processing on the to-be-imported data to obtain at least one to-be-imported decompressed data; and store each to-be-imported decompressed data in a storage space related to task data in the to-be-processed task through the target processing interface.

21. The apparatus of claim 20, wherein, The execution module is further configured to, when it is determined based on a heartbeat packet sent by the user terminal that the user terminal has successfully uploaded the to-be-imported data, call the target processing interface corresponding to the to-be-processed task to obtain the to-be-imported data; wherein the heartbeat packet is an information packet generated by the user terminal based on the uploading progress of the to-be-imported data.

22. The apparatus of claim 13, wherein, The task data includes a to-be-arranged space identifier; the execution module is further configured to, when the operation type corresponding to the to-be-processed task is a data arrangement type, call a target processing interface corresponding to the to-be-processed task to access a to-be-arranged space corresponding to the to-be-arranged space identifier; and perform iteration on non-compressed data stored in the to-be-arranged space through the target processing interface to determine first data information of the non-compressed data. The first data information is stored in a preset storage space.

23. The apparatus of claim 13, wherein, The task data includes a to-be-arranged space identifier; the execution module is further configured to, when the operation type corresponding to the to-be-processed task is a data arrangement type, call a target processing interface corresponding to the to-be-processed task to access a to-be-arranged space corresponding to the to-be-arranged space identifier; and download compressed data stored in the to-be-arranged space through the target processing interface, and perform decompression processing on the downloaded compressed data to obtain at least one to-be-arranged decompressed data. The to-be-arranged decompressed data is iterated to determine second data information of the to-be-arranged decompressed data, and the second data information is stored in a preset storage space.

24. The apparatus of claim 13, wherein, The execution module is further configured to, when the task data includes label data, perform verification on a data format and data content of the label data to obtain a verification result; and when the verification result indicates that the verification is passed, perform an operation on a label database based on the label data according to an operation mode corresponding to the operation type to which the to-be-processed task belongs, to execute the to-be-processed task. 25.A computer device, comprising a memory and a processor, wherein the memory stores a computer program. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 12.

26. A computer readable storage medium storing a computer program, wherein the computer program comprises instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-25. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 12.

27. A computer program product comprising computer instructions, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 12.

Images