Data processing method, device, system and equipment and storage medium

Abstract

The invention provides a data processing method, device, system and equipment and a storage medium, the method is applied to a server side, and the server side is used for managing at least one piece of user data and maintaining a corresponding relation between the user data and a data fragment to which the user data belongs; wherein the data fragment to which the user data belongs is determined in at least one preset data fragment based on a preset data fragment rule; the method comprises the steps that in response to a received data calculation request, the task number of at least one calculation task contained in the data calculation request is obtained, and at least one data fragment is distributed to the calculation task based on the task number; and providing the user data corresponding to the data fragment allocated by the calculation task for the calculation task for calculation based on the allocation result and the maintained corresponding relationship between the user data and the data fragment to which the user data belongs. According to the embodiment, the response speed of the data calculation request can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular, to data processing methods, devices, systems, equipment, and storage media. Background Art

[0002] In scenarios such as cloud computing, a service provider can provide cloud storage services for users, and users can store data in the cloud. In some scenarios, the service provider also supports the computing requirements for the data stored by users. For example, in the scenario of big data processing, users need to start multiple parallel computing tasks to compute the data sets they hold. In related technologies, usually after a user starts a computing task, they submit the sharding rules for the data set they have specified. After the service provider receives it, based on the user's sharding rules, the data set is evenly sharded, and then each computing task calculates the shards separately. It can be seen that in related technologies, after the service provider receives the user's sharding rules, it takes a long time to shard the user's data set, and the computing tasks need to wait for a long time to start computing. Summary of the Invention

[0003] To overcome the problems existing in related technologies, the present disclosure provides data processing methods, devices, systems, equipment, and storage media.

[0004] According to a first aspect of the embodiments of the present specification, a data processing method is provided. The method is applied to a server, and the server is used to manage at least one user's data and to maintain the correspondence between the user's data and the data shards to which it belongs; wherein, the data shards to which the user's data belongs are determined based on a preset data sharding rule among at least one preset data shards; the method includes:

[0005] In response to receiving a data calculation request, obtain the number of tasks of at least one computing task included in the data calculation request, and allocate at least one of the data shards to the computing task based on the number of tasks;

[0006] Based on the allocation result and the maintained correspondence between the user's data and the data shards to which it belongs, provide the user's data corresponding to the data shards allocated to the computing task to the computing task for calculation.

[0007] According to a second aspect of the embodiments of the present specification, a data processing device is provided. The device is applied to a server, and the server is used to manage at least one user's data and to maintain the correspondence between the user's data and the data shards to which it belongs; wherein, the data shards to which the user's data belongs are determined based on a preset data sharding rule among at least one preset data shards; the device includes:

[0008] In response to receiving a data calculation request, obtain the number of tasks of at least one calculation task included in the data calculation request, and allocate at least one of the data shards to the calculation task based on the number of tasks;

[0009] Based on the allocation result and the maintained correspondence between the user data and the data shards to which it belongs, provide the user data corresponding to the data shards allocated to the calculation task for the calculation task to perform the calculation.

[0010] According to the third aspect of the embodiments of the present specification, a data processing system is provided. The data processing system includes at least one data service node, at least one calculation node, and at least one storage node. The at least one calculation node is used to run at least one calculation task, and the at least one storage node is used to store user data; wherein, the data service node is used to run the steps of the method embodiment of the first aspect.

[0011] According to the fourth aspect of the embodiments of the present specification, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method embodiment of the first aspect described above are implemented.

[0012] According to the fifth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method embodiment of the first aspect described above are implemented.

[0013] The technical solutions provided by the embodiments of the present specification may include the following beneficial effects:

[0014] In the embodiments of the present specification, the server can manage user data, and among the preset at least one data shards, the data shards to which the user data belongs are determined in advance based on the preset data sharding rules. Therefore, the server can maintain the correspondence between the user data and the data shards to which it belongs; based on this, if a data calculation request is received, the number of tasks of at least one calculation task included in the data calculation request can be obtained, and one or more data shards are allocated to each calculation task based on the number of tasks; based on the allocation result and the maintained correspondence between the user data and the data shards to which it belongs, the user data corresponding to the data shards allocated to the calculation task is provided for the calculation task to perform the calculation. It can be seen that the server pre-shards the user data. When calculation is required, the allocation is based on multiple data shards, rather than sharding based on the full amount of data. Therefore, data can be quickly provided to the calculation task, significantly improving the response speed of the data calculation request.

[0015] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present specification, and are used together with the specification to explain the principles of the present disclosure.

[0017] Figure 1A is a flowchart of a data processing method shown in accordance with an exemplary embodiment of the present specification.

[0018] Figure 1B is a schematic diagram of data processing shown in accordance with an exemplary embodiment of the present specification.

[0019] Figure 2 is an application scenario diagram of a data processing method shown in accordance with an exemplary embodiment of the present specification.

[0020] Figure 3 is a hardware structure diagram of a computer device where a data processing device is located shown in accordance with an exemplary embodiment of the present specification.

[0021] Figure 4 is a block diagram of a data processing device shown in accordance with an exemplary embodiment of the present specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present specification as detailed in the appended claims.

[0023] The terms used in this specification are for the purpose of describing particular embodiments only and are not intended to limit the present specification. The singular forms "a", "the", and "said" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0025] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0026] In some cloud service scenarios, the service provider can provide data storage services, which can include different types of data storage services such as file storage, block storage, or object storage (OSS, Object Storage Service). In some application scenarios, users may need to perform calculations on the data sets stored in their storage spaces. For example, users need to start multiple parallel computing tasks. It can be to evenly slice the data set, and then each computing task calculates the corresponding data slice respectively. Some service providers also provide a data resource platform to support the above parallel computing requirements.

[0027] The data resource platform is a full-process platform for data asset definition, processing, management, and service, and can provide functions such as data synchronization, data query, data standard, data modeling, data processing, quality assessment, model construction, asset management, and data service. The data resource platform can manage the data sets in the user's storage space. In the related technology, the user needs to determine the corresponding data slice for each computing task. The user can submit the slicing rule to the data resource platform, and then the platform determines the slice to which each piece of data belongs based on the user's slicing rule, and finally provides the data contained in each slice to the corresponding computing task. It can be seen that after the user submits the slicing rule, the data resource platform starts to calculate the data slice to which each piece of data in the data set belongs based on the slicing rule. Therefore, it takes a long time to respond to the user's computing task. In addition, the user also needs to configure the data slicing rule by himself, which has relatively high requirements for the user and the user operation is rather cumbersome.

[0028] An embodiment of this specification provides a data processing method. The server can manage user data, and in at least one preset data shard, the data shard to which the user data belongs is determined in advance based on a preset data sharding rule. Therefore, the server can maintain the correspondence between the user data and the data shard to which it belongs. Based on this, if a data calculation request is received, the number of tasks of at least one calculation task included in the data calculation request can be obtained, and based on the number of tasks, the at least one data shard is allocated to the at least one calculation task. Based on the allocation result and the maintained correspondence between the user data and the data shard to which it belongs, the user data corresponding to the data shard allocated to the calculation task is provided to the calculation task for calculation. It can be seen that the server has pre-sharded the user data. When calculation is required, the allocation is based on multiple data shards instead of sharding based on the full amount of data. Therefore, data can be quickly provided to the calculation task, significantly improving the response speed of the data calculation request. Next, the embodiments of this specification will be described in detail.

[0029] As Figure 1A shown, Figure 1A FIG. is a flowchart of a data processing method shown in accordance with an exemplary embodiment of this specification. The method is applied to a server, and the server is used to manage at least one user data and to maintain the correspondence between the user data and the data shard to which it belongs. Among them, the data shard to which the user data belongs is determined based on a preset data sharding rule in at least one preset data shard. The method may include the following steps:

[0030] In step 102, in response to receiving a data calculation request, obtain the number of tasks of at least one calculation task included in the data calculation request, and allocate at least one of the data shards to the calculation task based on the number of tasks.

[0031] In step 104, based on the allocation result and the maintained correspondence between the user data and the data shard to which it belongs, provide the user data corresponding to the data shard allocated to the calculation task to the calculation task for calculation.

[0032] The method of this embodiment can be applied to a server, such as a cloud computing service platform, etc. The cloud computing service platform can be used to provide one or more cloud computing services to users, such as a data warehouse, object storage, container service, or virtual machine service, etc.

[0033] Optionally, the data in the dataset can be unstructured data, structured data, semi-structured data, etc., which are not limited in this book. Taking unstructured data as an example, it can include text documents, images, audio, video, etc.; taking structured data as an example, it can include data tables, etc. Semi-structured data can include data stored in Extensible Markup Language documents, JSON (JavaScript Object Notation, a lightweight data interchange format name) documents or similar formats.

[0034] Optionally, it can be the server that provides cloud storage services, and users can submit any user data at any time through the cloud storage services.

[0035] The data storage method in this embodiment can include different types such as file storage, block storage, or object storage, which are not limited in this embodiment. Among them, file storage organizes and presents data in a hierarchical structure of files and folders; block storage can split data into volumes of arbitrary partitions and the same size.

[0036] Object storage is a storage service for storing and managing large-scale unstructured data. It is achieved by storing data in the form of objects. That is, various user data can be stored in the storage space (Bucket) in the form of objects (Object). Object storage can manage data and link it to associated metadata.

[0037] As an example, the user data submitted by the user can be stored in the user's cloud storage space using the object storage method. Optionally, the user can also delete the user data in the storage space.

[0038] This embodiment can be applied to any data calculation scenario. For example, the data producer can produce and store a large number of images. To support the picture search function, the data calculator needs to calculate the vectors of the pictures and save the vector results for data retrieval. Since the number of pictures involved in the calculation reaches the scale of hundreds of millions, a large number of calculation tasks are required to ensure the timely completion of the calculation work. The server in this embodiment can be used as the provider of basic data. When the data calculator starts multiple calculation tasks, it can quickly provide each calculation task with the images it needs to calculate, and the data provided to each calculation task can also be non-repetitive. When the calculation scale increases, it should also be able to provide good support without additional configuration or operation.

[0039] In this embodiment, the server can pre-determine the data shards to which the user data belongs. In some examples, the data shards to which the user data belongs can be determined by the following methods:

[0040] In response to receiving an update message of user data, if the update message indicates to store at least one target user data, based on a preset data sharding rule, determine the data shard to which the at least one target user data stored this time belongs in at least one preset data shard.

[0041] The user data update message of this embodiment may include the case where the user writes the user data in the storage space for the first time, or may include the case where the new data is updated to the storage space after the data is written for the first time. The aforementioned server may manage all user data, and the target user data stored here may be all or part of all user data, that is, the data storage here may occur once or multiple times.

[0042] As an example, the server may obtain an update message sent by the cloud storage service. If the update message indicates to store one or more target user data, the server may specify a data shard for each target user data stored this time in at least one preset data shard based on a preset data shard rule.

[0043] As an example, when a user writes a batch of user data to the storage space for the first time, the execution of the solution of this embodiment can be triggered, and by obtaining the data update message of this time, data sharding and other processing can be performed for the batch of data written this time. After a period of time, the user updates another batch of user data in the storage space, which can also trigger the execution of the solution of this embodiment. By obtaining the data update message of this time, data sharding and other processing can be performed for the batch of data updated this time.

[0044] In this embodiment, the number of data shards can be preset. Optionally, the number of data shards can be designed to be a value as large as possible as the number of tasks of the user's computing tasks based on actual application scenarios. The number of tasks of the user's historical computing tasks can also be counted based on historical data, and the design can be combined with the size of actual computing resources or the amount of user data.

[0045] Data sharding rules can be implemented in many ways. For example, it can be a rule for evenly sharding user data sets, or other sharding rules. Taking the rule for even sharding as an example, the key fields of the data (such as identification, date, etc.) can be used as input to calculate a hash value through a hash function. The hash value is modulo the number of shards to obtain the shard number to which the data should be assigned. Alternatively, for multiple preset data shards, each data is assigned to each data shard in turn.

[0046] As an example, in this embodiment, through the data sharding rule, a data set containing multiple user data can be evenly sharded; among them, for the sharding result, it is not limited that each data shard contains the same number of user data. The data volume of the user data set may vary, and the data volume is not necessarily an integer multiple of the data shard. Therefore, each data shard may contain different numbers of user data. The data sharding rule is used to avoid the problem that some data shards contain too much or too little user data, so that the user data in the data set can be evenly distributed in each data shard, that is, there is a certain difference allowed between any two data shards among the data shards in this embodiment. The difference can be within a preset range and can be configured according to needs in actual applications, as long as the number of user data in each data shard is basically close.

[0047] In some examples, the data sharding rule includes a rule for indicating the correspondence between the data identifier and the data shard; based on the preset data sharding rule, determining the data shard to which at least one target user data to be stored this time belongs in at least one preset data shard may include:

[0048] Assign a data identifier to each target user data to be stored this time, and based on the data sharding rule, determine the data shard corresponding to the data identifier of each target user data, so as to obtain the data shard to which each target user data belongs.

[0049] The rule for indicating the correspondence between the data identifier and the data shard in this embodiment can be customized according to actual needs. For example, it can be a rule set based on a hash function, range partitioning, round-robin allocation, etc.

[0050] Optionally, the data identifier can be a unique identifier, which can be a continuous number or other forms of identifiers, etc. This embodiment does not limit this. Based on this, according to the data sharding rule, the data identifier of each target user data can be corresponded to the corresponding data shard, so as to determine the data shard to which the target user data belongs.

[0051] In some examples, the assigning a data identifier to each target user data to be stored this time may include:

[0052] Access each target user data to be stored this time in sequence, and assign a self-increasing number as the data identifier to each user data when accessing each target user data.

[0053] In this embodiment, by using self-incrementing numbers as data identifiers, unique data identifiers can be quickly assigned to user data during the data storage phase, which also facilitates subsequent data sharding. Among them, there are various ways to implement the self-incrementing numbers of user data. For example, before starting to access the target user data, you can initialize a counter or a self-incrementing sequence and set the initial value to 1 or other appropriate values. Access each target user data in turn. When accessing each user data, obtain the current value from the counter or the self-incrementing sequence as the data identifier and assign this value to the current target user data. Update the counter or the self-incrementing sequence: After assigning the data identifier to the target user data, update the value of the counter or the self-incrementing sequence to increment it by one, so as to assign a new data identifier to the next target user data.

[0054] In some examples, determining the data shard corresponding to the data identifier of each target user data based on the data sharding rule may include:

[0055] Perform a modulo operation on the data identifier of each target user data and the preset number of data shards, and determine the data shard corresponding to the data identifier of each target user data based on the result of the modulo operation. In this embodiment, by using the modulo operation, the data shard corresponding to the data identifier of each target user data can be quickly and accurately determined.

[0056] Exemplarily, assuming that the preset number of data shards is 1000, the following steps can be performed:

[0057] Step 1: Preset the number of data shards to 1000.

[0058] Step 2: Create a self-incrementing counter, and the initial value can be configured as needed, for example, 0.

[0059] Step 3: Traverse each data:

[0060] For each data, take the modulo of the value of the counter by 1000 to obtain a remainder.

[0061] The remainder represents the data shard number to which the data belongs.

[0062] Assign the data to the corresponding data shard.

[0063] Increment the value of the counter by 1.

[0064] Step 4: Repeat Step 3 until all data are assigned to the corresponding data shards.

[0065] Among them, the value of the counter does not need to be cleared. After the user updates and adds new user data again next time, the value of the counter increments again. In this way, each piece of user data of the user will have a self-incrementing number, and all user data can be evenly sharded based on the self-incrementing number.

[0066] In this embodiment, each data shard can have a unique identifier, which is called a shard identifier in this embodiment. Each piece of user data can have corresponding metadata. Based on the sharding result of the user data, the correspondence between the user data and the sharding information of the data shard to which it belongs can be recorded. As an example, the server maintains metadata of user data, and the metadata records the sharding information of the data shard to which the user data belongs.

[0067] The metadata in this embodiment refers to data that describes data. Metadata is usually used to describe information such as the attributes, characteristics, structure, and format of data, and can help understand and manage data. As an example, it can include but is not limited to the following information:

[0068] Data structure description: Describes information such as the field name, data type, and length of the data.

[0069] Data source: Information such as the source of the data, collection time, and collection method.

[0070] Data relationship: Information such as the association and dependency relationships between the data and other data.

[0071] Data usage rules: Rules such as the access rights, usage methods, and confidentiality levels of the data.

[0072] In this embodiment, when recording the data shard to which the data belongs, the sharding information of the data shard, such as the shard identifier, can be represented as part of the metadata, so that the shard where the data is located can be quickly located when querying, analyzing, or managing the data later. Optionally, the metadata can be structured data, semi-structured data, or unstructured data, and this embodiment does not limit this.

[0073] The data calculation request in step 104 can have various receiving methods in practical applications. For example, it can be receiving a calculation request initiated by the user, or the user can set rules for starting tasks, and in response to meeting the preset rules for starting tasks, it is determined that a calculation request is received. In other examples, it can also be receiving calculation requests sent by each calculation task.

[0074] In practical applications, the calculation tasks that the user can start can be one or more. Optionally, the calculation request can include the number of calculation tasks required to be started for this calculation. If each calculation task initiates a calculation request, each calculation task can also send information such as its own sequence number among all calculation tasks.

[0075] Optionally, the number of tasks of the computing task may be less than or equal to the number of data shards.

[0076] Among them, there are various implementation manners for allocating the at least one data shard to the at least one computing task based on the number of tasks, which can be set according to the actual scenario, and this embodiment does not limit this. For example, the computing resources of each computing task are different, and the number of data shards of each computing task can be determined based on the computing resources.

[0077] In some other examples, such as in scenarios like parallel computing, it may be to evenly allocate multiple data shards to each computing task. Here, it is not limited that each computing task must be allocated the same number of data shards. The number of tasks of the computing tasks for parallel computing by the user each time may be different, and the number of data shards is not necessarily an integer multiple of the number of tasks. Therefore, each computing task may correspond to a different number of data shards. This embodiment can evenly allocate multiple data shards to each computing task, which can avoid too many or too few data shards corresponding to some computing tasks, so that multiple data shards can be evenly allocated to each computing task, that is, a certain difference is allowed between the number of data shards corresponding to any two computing tasks among the computing tasks in this embodiment. The difference can be within a preset range and can be configured according to needs during actual application, as long as the number of data shards corresponding to each computing task is basically close.

[0078] There are various implementation manners for evenly allocating multiple data to each computing task among all computing tasks in actual application. As an example, it can be an even allocation based on polling. For example, all data shards can be numbered in sequence, and then each shard is allocated to the computing task in turn. For example, as Figure 1B shown, there are 10 data shards and 5 computing tasks, and they can be allocated in the following manner:

[0079] Computing task 1: Shard 1, Shard 2

[0080] Computing task 2: Shard 3, Shard 4

[0081] Computing task 3: Shard 5, Shard 6

[0082] Computing task 4: Shard 7, Shard 8

[0083] Computing task 5: Shard 9, Shard 10.

[0084] In some examples, allocating at least one of the data shards to the computing task based on the number of tasks may include:

[0085] Obtain the number of data shards, divide the number of data shards by the number of tasks to obtain a quotient value m and a remainder n;

[0086] After allocating m data shards to each computing task from the at least one data shard, allocate the remaining n data shards to any n data shards among the at least one data shard; where m is an integer greater than or equal to 1, and n is an integer greater than or equal to 0.

[0087] As an example, assume there are 15 data shards and 6 computing tasks that need to be evenly allocated, and the following operations can be performed:

[0088] 1. Obtain the quotient value m and the remainder n of the preset number of shards divided by the number of tasks:

[0089] Number of data shards: 15

[0090] Number of tasks: 6

[0091] Quotient value m: 15 / 6 = 2

[0092] Remainder n: 15 % 6 = 3

[0093] 2. Allocate m data shards to each computing task:

[0094] Computing task 1: Shard 1, Shard 2

[0095] Computing task 2: Shard 3, Shard 4

[0096] Computing task 3: Shard 5, Shard 6

[0097] Computing task 4: Shard 7, Shard 8

[0098] Computing task 5: Shard 9, Shard 10

[0099] Computing task 6: Shard 11, Shard 12

[0100] Among them, in the above allocation, each computing task is first allocated two data shards, and these two data shards are two adjacent shards. It can be understood that this is just an example here. In actual applications, the two data shards corresponding to each computing task can be arbitrary, as long as the data shards corresponding to each computing task are different.

[0101] 3. Randomly allocate the remaining n shards to any n data shards:

[0102] The remaining 3 shards: Shard 13, Shard 14, Shard 15

[0103] Randomly allocate to data shards: Computing task 1: Shard 13, Computing task 2: Shard 14, Computing task 3: Shard 15.

[0104] Based on this, through this embodiment, it can be ensured that each computing task obtains at least m data shards, and the remaining n shards are evenly distributed to the tasks, realizing the even distribution of data shards. As an example, the distribution of the last remaining shards can be random, as long as each task obtains one. Of course, in practical applications, it can be adjusted and extended according to specific situations. Through this embodiment, each computing task can obtain fair data shards, thereby ensuring load balancing among tasks.

[0105] In some examples, the server maintains metadata of user data, and the metadata records the shard information of the data shards to which the user data belongs and the storage path of the user data;

[0106] Providing the user data corresponding to the data shards allocated to the computing task to the computing task for calculation may include:

[0107] For each computing task, query from the maintained metadata the metadata that contains the shard information of the data shards allocated to the computing task, and provide the storage paths recorded in each piece of the queried metadata to the computing task, so that the computing task can access the user data based on the storage path and perform calculations.

[0108] Exemplarily, each user data has corresponding metadata, and all the metadata can be stored. The metadata may include fields such as shard identifiers and storage paths; after determining the data shards allocated to each computing task, it is possible to know the shard identifiers of one or more data shards corresponding to each computing task, and each metadata records the shard identifier of the corresponding user data. Therefore, those metadata corresponding to the shard identifiers of each computing task can be queried from all the stored metadata. At the same time, since the metadata also contains the storage path of the user data, the storage paths can also be retrieved from the queried metadata and provided to the computing task, so that the computing task can access the user data based on the storage path and perform calculations.

[0109] Through the above design of metadata in this embodiment, after the data shard allocation is completed, the storage paths of the user data corresponding to the computing tasks can be quickly queried based on the metadata, and then these storage paths are provided to the computing tasks for data access and calculation operations, which can realize the flexible management of data and the efficient response to data calculation requests.

[0110] In some examples, the data calculation request further includes calculation conditions;

[0111] Providing the user data corresponding to the data shards allocated to the computing task to the computing task for calculation includes:

[0112] After determining the user data corresponding to the data shards assigned to the computing task, query the user data that meets the computing conditions from the determined user data and provide it to the computing task for calculation.

[0113] In an actual computing scenario, a single calculation can be performed on the entire dataset of a user, or it may be a calculation on specific data within the entire dataset. In this embodiment, the data calculation request may further include calculation conditions; as mentioned above, the metadata corresponding to the user data may include one or more description information of the user data, and the calculation conditions may include one or more specific description information.

[0114] For each computing task, after this embodiment assigns the corresponding data shards to each computing task, it can be understood that the data range to be calculated by the computing task is determined. For example, if the entire dataset is 10,000 pieces of data and 10 computing tasks are each assigned 1,000 pieces of data, for each computing task, query the data that meets the computing conditions from the 1,000 pieces of data assigned to this computing task. For example, it can be queried from the metadata of each piece of data corresponding to the computing task. The calculation conditions include one or more specific description information, and it can be queried which metadata records these specific description information. Compared with the prior art, where it is necessary to query in the entire dataset for each computing task, this embodiment can significantly improve the response speed.

[0115] In some examples, before the step of, in response to receiving a data calculation request, obtaining the number of tasks of at least one computing task included in the data calculation request, the method further includes:

[0116] Obtain the number of data shards, and output a prompt message to the user indicating that the number of tasks needs to be less than or equal to the number of data shards.

[0117] In this embodiment, a prompt message can be output to the user to prompt the user that the number of tasks set needs to be less than or equal to the number of data shards, and the prompt message can include the specific number of data shards. In this way, the user can set a number of tasks according to the prompt, such that the number should be less than or equal to the number of data shards.

[0118] Optionally, according to the need, further verify the number of tasks set by the user, verify whether the number of tasks set by the user is less than or equal to the number of data shards. If the number of tasks set by the user is greater than the number of data shards, an error prompt can be returned and the user can be required to reset a suitable number of tasks. If the number of tasks set by the user meets the requirements, the calculation tasks can be allocated according to this number. Through the above steps, the system can ensure that the number of tasks set by the user meets the requirements and perform subsequent calculation task allocation and processing based on this number.

[0119] In some examples, the method may further include: obtaining the number of data shards, and in response to the number of tasks being greater than the number of data shards, outputting a prompt message for adjusting the number of tasks to the user. Based on this, the user can be allowed to readjust the number of computing tasks.

[0120] In some examples, the method may further include: when the received update message indicates deleting one or more user data, the corresponding relationship between each piece of user data to be deleted this time and the data shard to which it belongs can be deleted, for example, the metadata of each piece of user data to be deleted this time can be deleted. In this embodiment, for the deletion of user data, the corresponding relationship between the user data and the data shard to which it belongs can be deleted accordingly, so that the deleted data will not be provided to the computing task for calculation, preventing redundant calculation of the computing task.

[0121] Next, it will be described through another embodiment. As Figure 2 shown, it is an application scenario diagram of a data processing method shown in this specification according to an exemplary embodiment, in which a server, a computing task system, and storage nodes are shown; the server can execute the data processing method in any of the foregoing embodiments, the computing task system can be used to provide computing services to users, and the storage system can include one or more storage nodes for storing user data.

[0122] Among them, the server may specifically be a data resource platform. Optionally, a metadata management function may be configured in the data resource platform. The metadata center can create and manage the metadata of each piece of data for the user's data set. Here, the metadata of the data includes one or more description information of the data, such as data name, format, write time, type, or storage path, etc. As an example, the method of the foregoing embodiment can be applied to the metadata management function of the data resource platform and can execute the following process:

[0123] Receiving a data update message; the data update here includes: update types such as data writing, information update of data, or data deletion; it is possible to determine which data update types are involved in this update based on the data update message;

[0124] If it is data writing, it means that new data needs to be added to the user's storage space. The data to be written this time can be obtained, and a unique identifier is assigned to each piece of data. The unique identifier can be a self-increasing number generated by a counter, and the unique identifier can be used as one of the fields of the metadata of the data. The data and the corresponding identifier can be written into the storage node together.

[0125] After the data writing is completed, a data synchronization process can be triggered, so as to read the metadata of the data from the storage node to obtain the identifier recorded in the metadata;

[0126] Based on the pre-set number of data shards and the identifiers recorded in each metadata, it is possible to calculate the data shard to which each piece of data is assigned.

[0127] In the scenario of massive data, the computing tasks initiated by the user may also be for specific data in the massive data held by the user. To meet the user's needs, the data that meets the user's specific needs is queried from the user's data and provided to the computing tasks for calculation. The server in this embodiment may further include a search engine. This embodiment does not limit the search engine. For example, it may be Elasticsearch (the name of a search engine), etc. Elasticsearch is a distributed text search and data analysis engine and is widely used in building applications such as real-time search, log analysis, and data visualization. It can quickly retrieve, analyze, and process massive amounts of data and has good scalability and reliability. In Elasticsearch, all data is stored in the form of documents, and these documents can be indexed, searched, and analyzed. Users can use the interfaces provided by Elasticsearch to perform various operations, such as creating, updating, deleting documents, searching for documents, aggregating data, etc.

[0128] Optionally, the search engine can build an index. The specific way of building the index can be implemented by the native index function of the search engine. This embodiment does not limit this. The metadata can be provided to the search engine to build an index, and the metadata contains the data shard identifier corresponding to each piece of data. Therefore, the search engine can build an index for the shard identifiers of the data shards, and subsequently, the search engine can quickly query the data under the data shards corresponding to each computing task based on the index.

[0129] After starting the task, in the parallel computing scenario, each computing task can be the same computing process. Each computing task can initiate a computing request separately. The server can obtain the computing requests of each computing task. The computing request can carry the total number of tasks and the task identifier of the current task (for example, it can be the serial number of this batch of computing tasks started this time, etc.). Based on the total number of tasks and the number of data shards, the data shards covered by the current task are determined. Optionally, this calculation can be the user's full amount of data or the data that meets specific calculation conditions. In this case, the computing request can also include calculation conditions.

[0130] The processing of the computing requests for each computing task is the same. Taking the current computing task as an example, the calculation conditions and the shard identifiers of the data shards corresponding to the current computing task can be passed to the search engine, so that the search engine performs data retrieval based on the calculation conditions and the shard identifiers;

[0131] The search engine can first obtain all user data covered by the current computing task based on the shard identifier of data sharding, and then query the data that meets the computing conditions from it to obtain the query result and return it to the current computing task; among them, the query result can be the storage path of each piece of data, and after the computing task obtains the storage path of each piece of data, it can access each piece of data based on the storage path in the storage node for computing.

[0132] Exemplarily, taking user data as images for example, each computing task can calculate and store the corresponding image features of the respective images assigned to it to support subsequent image search tasks; for example, the image search task can carry the image features of the image to be searched, and compare the image features carried in the image search task with the stored respective image features, so as to search for the same or highly similar image features, and the images corresponding to the searched image features can be returned to the image search task.

[0133] As can be seen from the above embodiments, in the data storage stage of this embodiment, the dataset is pre-sharded. When querying, by using the number of tasks passed by the client and the current task serial number being executed, the data shard range that needs to be queried for the current query request can be calculated, ensuring reasonable data sharding and query performance. And by setting a relatively large number of shards, the expansion of the number of tasks can be well supported, and the scalability of the overall computing can be improved.

[0134] Since the relationship between the data shards corresponding to each piece of user data is maintained, for example, each piece of data can have a corresponding shard identifier, in the case of a configured search engine, the search engine can create an index for the shard identifier. In this way, since the computing part of data sharding is placed in the data storage stage and calculations are performed first during data storage, when the computing task needs to read data, it can directly perform index query without additional calculations, reducing the amount of calculation, and the search engine can accelerate data query through the index.

[0135] The data sharding rule of this embodiment is transparent to the data computing party. The data computing party only needs to set the total number of tasks. The computing task provides the number of tasks and its own task serial number. The server calculates the data shard range corresponding to the current task according to the number of tasks, and then returns the content within the corresponding data shard. In this way, the client does not need to care about the underlying data structure and logic at all, and only needs to care about the setting of its own tasks, greatly reducing the usage cost.

[0136] Corresponding to the foregoing embodiments of the data processing method, this specification also provides embodiments of a data processing apparatus and a computer device to which it is applied.

[0137] Embodiments of the data processing device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, or through hardware, or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by the processor in the data processing where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for operation. From a hardware perspective, as Figure 3 shown, it is a hardware structure diagram of the computer device where the data processing device described in this specification is located, and it can include Figure 3 the processor 310, the memory 330, the network interface 320, and the non-volatile memory 340 as shown. Among them, the memory 330 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing device in the embodiments of this specification; the processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 330. In the embodiments, the computer device where the data processing device 331 is located usually includes other hardware according to the actual functions of the computer device, which will not be elaborated here.

[0138] As Figure 4 shown, Figure 4 is a block diagram of a data processing device shown according to an exemplary embodiment of this specification. The device is applied to the server side, and the server side is used to manage at least one user data and to maintain the corresponding relationship between the user data and the data shards to which it belongs; among them, the data shards to which the user data belongs are determined based on a preset data sharding rule among at least one preset data shards; the device includes:

[0139] An allocation module 41, configured to, in response to receiving a data calculation request, obtain the number of tasks of at least one calculation task included in the data calculation request, and allocate at least one of the data shards for the calculation task based on the number of tasks;

[0140] A data providing module 42, configured to provide the user data corresponding to the data shard allocated to the calculation task for the calculation task to perform calculation based on the allocation result and the maintained corresponding relationship between the user data and the data shards to which it belongs.

[0141] In some examples, the device further includes an allocation module, configured to:

[0142] In response to receiving an update message of user data, if the update message indicates storing at least one target user data, determine the data shards to which the at least one target user data stored this time belongs among at least one preset data shards based on a preset data sharding rule.

[0143] In some examples, the data sharding rule includes a rule for indicating the correspondence between data identifiers and data shards; the allocation module, based on a preset data sharding rule, determines the data shard to which at least one target user data to be stored this time belongs among at least one preset data shards, including:

[0144] Assign a data identifier to each target user data to be stored this time, and based on the data sharding rule, determine the data shard corresponding to the data identifier of each target user data, so as to obtain the data shard to which each target user data belongs.

[0145] In some examples, the allocation module assigns a data identifier to each target user data to be stored this time, including:

[0146] Access each target user data to be stored this time in sequence, and when accessing each target user data, assign a self-increasing number as the data identifier to each user data.

[0147] In some examples, the allocation module determines the data shard corresponding to the data identifier of each target user data based on the data sharding rule, including:

[0148] Perform a modulo operation on the data identifier of each target user data and the preset number of data shards, and based on the result of the modulo operation, determine the data shard corresponding to the data identifier of each target user data.

[0149] In some examples, the server maintains metadata of user data, and the metadata records the shard information of the data shard to which the user data belongs and the storage path of the user data;

[0150] The providing module provides the user data corresponding to the data shard allocated to the computing task to the computing task for computing, including:

[0151] For each computing task, query from the maintained metadata the metadata containing the shard information of the data shard allocated to the computing task, and provide the storage paths recorded in each piece of queried metadata to the computing task, so that the computing task accesses the user data based on the storage path and performs computing.

[0152] In some examples, the data calculation request further includes calculation conditions;

[0153] The data providing module provides the user data corresponding to the data shard allocated to the computing task to the computing task for computing, including:

[0154] After determining the user data corresponding to the data shards allocated for the computing task, query the user data that meets the computing conditions from the determined user data and provide it to the computing task for computing.

[0155] In some examples, the allocation module allocates at least one of the data shards for the computing task based on the number of tasks, including:

[0156] Obtain the number of data shards, divide the number of data shards by the number of tasks to obtain a quotient value m and a remainder n;

[0157] After allocating m data shards for each computing task from the at least one data shard, allocate the remaining n data shards to any n data shards among the at least one data shard; where m is an integer greater than or equal to 1, and n is an integer greater than or equal to 0.

[0158] In some examples, the device further includes a prompt module for: before obtaining the number of tasks of at least one computing task included in the data computing request in response to receiving the data computing request, obtaining the number of data shards and outputting a prompt message to the user that the number of tasks needs to be less than or equal to the number of data shards.

[0159] In some examples, the device further includes a prompt module for:

[0160] Obtain the number of data shards, and in response to the number of tasks being greater than the number of data shards, output a prompt message to the user to adjust the number of tasks.

[0161] The implementation processes of the functions and roles of each module in the above data processing device are specifically described in the implementation processes of the corresponding steps in the above data processing method, and will not be elaborated here.

[0162] Correspondingly, an embodiment of this specification further provides a data processing system, which includes at least one data service node, at least one computing node, and at least one storage node. The at least one computing node is used to run at least one computing task, and the at least one storage node is used to store user data; wherein, the data service node is used to execute the steps of the foregoing data processing method embodiment.

[0163] Correspondingly, an embodiment of this specification further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the foregoing data processing method embodiment.

[0164] Correspondingly, an embodiment of this specification further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the data processing method embodiment are implemented.

[0165] Correspondingly, an embodiment of this specification further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method embodiment are implemented.

[0166] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this specification. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0167] The above embodiments can be applied to one or more computer devices. The computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0168] The computer device can be any electronic product that can perform human-computer interaction with users. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0169] The computer device may further include a network device and / or a user device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing (Cloud Computing).

[0170] The network where the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, Virtual Private Network (VPN), etc.

[0171] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this patent; adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of its algorithm and process, are all within the protection scope of this application.

[0173] Among them, the description of "specific examples" or "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of this specification. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0174] Those skilled in the art will readily think of other embodiments of this specification after considering the specification and practicing the invention claimed herein. This specification is intended to cover any variations, uses or adaptations of this specification, which follow the general principles of this specification and include the common general knowledge or conventional technical means in the technical field not claimed in this application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of this specification are pointed out by the following claims.

[0175] It should be understood that this specification is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is only limited by the appended claims.

[0176] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included within the protection scope of this specification.

Claims

1. A data processing method, which is applied to a server. The server is used to manage at least one user data and maintain the correspondence between the user data and the data shard to which it belongs; wherein, The data shard to which the user data belongs is determined based on a preset data sharding rule among at least one preset data shards; the method includes: In response to receiving a data calculation request, obtain the number of tasks of at least one calculation task included in the data calculation request, and allocate at least one of the data shards to the calculation task based on the number of tasks; Based on the allocation result and the maintained correspondence between the user data and the data shard to which it belongs, provide the user data corresponding to the data shard allocated to the calculation task to the calculation task for calculation.

2. The method according to claim 1, wherein the data shard to which the user data belongs is determined in the following manner: In response to receiving an update message of user data, if the update message indicates storing at least one target user data, determine the data shard to which the at least one target user data stored this time belongs among at least one preset data shards based on a preset data sharding rule.

3. The method according to claim 2, wherein the data sharding rule includes a rule for indicating the correspondence between a data identifier and a data shard; determining the data shard to which the at least one target user data stored this time belongs among at least one preset data shards based on the preset data sharding rule includes: Allocate a data identifier to each target user data stored this time, and determine the data shard corresponding to the data identifier of each target user data based on the data sharding rule to obtain the data shard to which each target user data belongs.

4. The method according to claim 3, wherein allocating a data identifier to each target user data stored this time includes: Access each target user data stored this time in sequence, and allocate an auto-incrementing number as the data identifier to each user data when accessing each target user data.

5. The method according to claim 4, wherein determining the data shard corresponding to the data identifier of each target user data based on the data sharding rule includes: Perform a modulo operation on the data identifier of each target user data and the preset number of data shards, and determine the data shard corresponding to the data identifier of each target user data based on the result of the modulo operation.

6. The method according to claim 1, wherein the server maintains metadata of user data, and the metadata records the shard information of the data shard to which the user data belongs and the storage path of the user data; The providing the user data corresponding to the data shard allocated to the calculation task to the calculation task for calculation includes: For each calculation task, query from the maintained metadata the metadata containing the shard information of the data shard allocated to the calculation task, and provide the storage paths recorded in each piece of queried metadata to the calculation task, so that the calculation task accesses the user data based on the storage path and performs calculations.

7. The method according to claim 1, wherein the data calculation request further includes calculation conditions; The providing the user data corresponding to the data shard allocated to the calculation task to the calculation task for calculation includes: After determining the user data corresponding to the data shards assigned to the computing task, query the user data that meets the computing conditions from the determined user data and provide it to the computing task for computing.

8. The method according to claim 1, wherein the step of allocating at least one of the data shards to the computing task based on the number of tasks includes: Obtain the number of data shards, divide the number of data shards by the number of tasks to obtain a quotient value m and a remainder n; After allocating m data shards to each computing task from the at least one data shard, allocate the remaining n data shards to any n data shards among the at least one data shard; wherein, m is an integer greater than or equal to 1, and n is an integer greater than or equal to 0.

9. The method according to claim 1, before the step of, in response to receiving a data computing request, obtaining the number of tasks of at least one computing task included in the data computing request, the method further includes: Obtain the number of data shards, and output a prompt message to the user that the number of tasks needs to be less than or equal to the number of data shards.

10. The method according to claim 1, the method further includes: Obtain the number of data shards, and in response to the number of tasks being greater than the number of data shards, output a prompt message to the user to adjust the number of tasks.

11. A data processing device, which is applied to a server. The server is used to manage at least one user data and maintain the corresponding relationship between the user data and the data shard to which it belongs; wherein, The data shard to which the user data belongs is determined based on a preset data shard rule among at least one preset data shard; the apparatus includes: An allocation module, configured to, in response to receiving a data computing request, obtain the number of tasks of at least one computing task included in the data computing request, and allocate at least one of the data shards to the computing task based on the number of tasks; A data providing module, configured to, based on the allocation result and the correspondence between the maintained user data and the data shards to which they belong, provide the user data corresponding to the data shards allocated to the computing task to the computing task for computing.

12. A data processing system, the data processing system includes at least one data service node, at least one computing node, and at least one storage node, the at least one computing node is used to run at least one computing task, and the at least one storage node is used to store user data; wherein, The data service node is used to execute the steps of any one of claims 1 to 10.

13. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor, when executing the computer program, implements the steps of any one of claims 1 to 10.

14. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of claims 1 to 10 are implemented.

Images