New task processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for processing a newly added task. The method comprises: determining the task type of the newly added task; when the newly added task is determined to be a valid task according to the task type, obtaining the first task array data corresponding to the newly added task; comparing the first task array data with the second task array data already existing in the database, and when it is determined that there is duplicate data between the first task array data and the second task array data, removing the duplicate data in the first task array data. The above method can be used to deduplicate the duplicate data in the newly added task, release the occupied system resources, and thus reduce the pressure on the system.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for processing a newly added task. Background Art

[0002] With the continuous development of computer technology, the functions of system platforms are becoming more and more perfect. In order to support the normal operation of the system platform, it is usually necessary to add new tasks to the background server.

[0003] For the newly added tasks of the system platform, the inventor found that some of the newly added tasks have duplicate data with the data stored in the original database. However, since it is impossible to locate the newly added tasks with duplicate data in the original database, there is a lot of duplicate data in the system, which in turn occupies more system resources and leads to resource waste. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device, computer-readable storage medium, and computer program product for processing a newly added task, so as to at least solve the problem in the related art that there is a lot of duplicate data in the system, which occupies a lot of system resources and causes waste of resources. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing a newly added task is provided, including:

[0006] Determine the task type of the newly added task;

[0007] When the newly added task is determined to be a valid task according to the task type, obtaining first task array data corresponding to the newly added task;

[0008] The first task array data is compared with the second task array data already existing in the database, and when it is determined that duplicate data exists between the first task array data and the second task array data, the duplicate data in the first task array data is removed.

[0009] In one of the embodiments, when the newly added task is determined to be a valid task according to the task type, before obtaining the first task array data corresponding to the newly added task, the method further includes:

[0010] According to the task type, query the valid task type table;

[0011] When it is determined that there is a task type matching the task type of the newly added task in the valid task type table, the newly added task is determined to be a valid task; wherein the valid task type table stores valid task types to be processed.

[0012] In one of the embodiments, when the newly added task is determined to be a valid task according to the task type, before obtaining the first task array data corresponding to the newly added task, the method further includes:

[0013] Determine, according to the task type of the newly added task, the number of hosts and the host memory usage required to process the newly added task;

[0014] If the current number of idle hosts meets the host number condition, and the memory usage of the current idle hosts meets the host memory usage condition, then the newly added task is determined to be a valid task.

[0015] In one of the embodiments, a data deduplication function is built into the first task array data;

[0016] The comparing the first task array data with the second task array data already existing in the database, and when it is determined that duplicate data exists between the first task array data and the second task array data, removing the duplicate data from the first task array data, comprises:

[0017] The data in the first task array data is compared with the data in the second task array data one by one through the data deduplication function. When it is determined that there is duplicate data between the first task array data and the second task array data, the duplicate data in the first task array data is removed.

[0018] In one embodiment, determining the task type of the newly added task includes:

[0019] Determining a task type of the newly added task according to the type of the file included in the newly added task;

[0020] or,

[0021] Determine the task type of the newly added task according to the task name of the newly added task.

[0022] In one embodiment, before determining the task type of the newly added task, the method further includes:

[0023] receiving a processing request for the newly added task;

[0024] In response to the processing request, when querying whether there is an idle channel resource for processing the newly added task, select an idle channel resource corresponding to the emergency level priority from a resource pool according to the emergency level priority of the newly added task;

[0025] Allocate the idle channel resources to the newly added task.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a device for processing a newly added task, including:

[0027] A task type determination module is configured to perform task type determination to determine the task type of a newly added task;

[0028] A data acquisition module is configured to acquire first task array data corresponding to the newly added task when the newly added task is determined to be a valid task according to the task type;

[0029] The data deduplication module is configured to compare the first task array data with the second task array data already existing in the database, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data.

[0030] In one embodiment, the device further comprises:

[0031] A query module, configured to query a valid task type table according to the task type;

[0032] The first task processing module is configured to determine that the newly added task is a valid task when it is determined that there is a task type matching the task type of the newly added task in the valid task type table; wherein the valid task type table stores valid task types to be processed.

[0033] In one embodiment, the device further comprises:

[0034] A processing resource determination module is configured to determine the number of hosts and the host memory usage rate required to process the newly added task according to the task type of the newly added task;

[0035] The second task processing module is configured to determine that the newly added task is a valid task if the current number of idle hosts meets the host number condition and the memory usage of the current idle hosts meets the host memory usage condition.

[0036] In one of the embodiments, a data deduplication function is built into the first task array data;

[0037] The data deduplication module is configured to execute the data deduplication function to compare the data in the first task array data with the data in the second task array data one by one, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data.

[0038] In one embodiment, the device further comprises:

[0039] A receiving module, configured to receive a processing request for the newly added task;

[0040] A resource allocation module is configured to execute in response to the processing request, when querying whether there is an idle channel resource for processing the newly added task, select an idle channel resource corresponding to the emergency level priority from a resource pool according to the emergency level priority of the newly added task;

[0041] Allocate the idle channel resources to the newly added task.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0043] processor;

[0044] a memory for storing instructions executable by the processor;

[0045] The processor is configured to execute the instructions to implement the method for processing a new task as described in any one of the embodiments of the first aspect above.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a processing method for a new task as described in any one of the embodiments of the first aspect above.

[0047] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform a processing method for a new task as described in any one of the embodiments of the first aspect above.

[0048] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0049] Determine the task type of the newly added task; when the newly added task is determined to be a valid task according to the task type, treat the newly added task as a task that requires data deduplication, and obtain the first task array data corresponding to the newly added task; compare the first task array data with the second task array data already existing in the database, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data, so that on the basis of accurately locating the new task that requires data deduplication, the duplicate data in the new task can be deduplicated, thereby releasing occupied system resources and reducing the pressure on the system.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0052] Figure 1 The figure is a flowchart of a method for processing a newly added task according to an exemplary embodiment.

[0053] Figure 2 The figure is a flowchart of a method for processing a newly added task according to an exemplary embodiment.

[0054] Figure 3 The figure is a block diagram of a device for processing a newly added task according to an exemplary embodiment.

[0055] Figure 4 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0059] The processing method of the newly added task provided by the present disclosure can be applied to a server. Among them, a task processing system is deployed in the server, and the task processing system can be, but is not limited to, used to generate new tasks, perform data deduplication processing on the new tasks, and process the new tasks after deduplication processing. The server generates new tasks through the task processing system and determines the task type of the new tasks. In the case where the new task is determined to be a valid task according to the task type, the first task array data corresponding to the new task is obtained. The server compares the first task array data with the second task array data already existing in the database. When it is determined that there is duplicate data in the first task array data and the second task array data, the duplicate data in the first task array data is removed, and the first task array data after the duplicate data is removed is stored in the database. Among them, the server can be implemented with an independent server or a server cluster composed of multiple servers.

[0060] Figure 1 is a flowchart of a method for processing a newly added task according to an exemplary embodiment. Figure 1 As shown, the method for processing the newly added task is used in the server and includes the following steps.

[0061] In step S110 , the task type of the newly added task is determined.

[0062] The task type can be determined based on the task association information of the newly added task. The task association information may include, but is not limited to, the data type in the newly added task, the task name of the newly added task, the business scenario to which the newly added task belongs, etc. In one example, if the business scenario to which the newly added task belongs is a face special effect generation scenario, the task type is determined to be an image processing type.

[0063] In step S120 , when the newly added task is determined to be a valid task according to the task type, first task array data corresponding to the newly added task is acquired.

[0064] Among them, the valid task can be any one or more of the tasks that the server can process, the tasks that the server is capable of processing, etc. In one example, the server determines that the new task is a valid task when it determines that other tasks with which the new task has dependencies have been processed according to the task type of the new task. In another example, the server determines that the new task is a valid task when it determines that the current remaining resources of the system (such as free threads) meet the resource requirements of the new task according to the task type of the new task.

[0065] Array data is a kind of collection data that organizes several elements of the same type in an ordered form. An array can be decomposed into multiple array elements, which can be basic data types or constructed types. According to the type of array elements, array data can be divided into various categories such as numeric array, character array, pointer array, structure array, etc.

[0066] Specifically, the first task array data is pre-configured array data that supports the normal operation of the newly added task. When the server determines that the newly added task is a valid task based on the task type, the first task array data corresponding to the newly added task is obtained. When the server determines that the newly added task is an invalid task based on the task type, it can stop processing the newly added task and reprocess the newly added task while waiting for the next processing opportunity to arrive. In one example, the next processing opportunity may be after a preset time length. That is, the server waits for the preset time length before reprocessing the newly added task.

[0067] In step S130 , the first task array data is compared with the second task array data already existing in the database, and when it is determined that duplicate data exists between the first task array data and the second task array data, the duplicate data in the first task array data is removed.

[0068] Among them, the second task array data may be data belonging to the same newly added task as the first task array data, but obtained through different channels. For example, the newly added task is a video recommendation task, and the first task array data is data obtained through the short video platform channel: [(Zhang San, female, nickname, browsing preference, related friends), (Li Si, male, nickname, browsing preference, related friends), (Wang Wu, male, nickname, browsing preference, related friends)]; the second task array data is data obtained through the live broadcast channel: [(Zhao Mou, female, nickname, browsing preference, related friends), (Li Si, male, nickname, browsing preference, related friends), (Wang Mou, female, nickname, browsing preference, related friends)].

[0069] Specifically, the server obtains the second task array data corresponding to the newly added task that already exists in the database, compares the array elements in the first task array data with those in the second task array data, and when it is determined that duplicate data exists between the first task array data and the second task array data, deletes the duplicate data in the first task array data, and saves the first task array data after deleting the duplicate data to the database.

[0070] In the above-mentioned method for processing newly added tasks, the task type of the newly added task is determined; when the newly added task is determined to be a valid task according to the task type, the newly added task is regarded as a task requiring data deduplication, and the first task array data corresponding to the newly added task is obtained; the first task array data is compared with the second task array data already existing in the database, and when it is determined that there is duplicate data between the first task array data and the second task array data, the duplicate data in the first task array data is removed, so that on the basis of accurately locating the new tasks that require data deduplication, the duplicate data in the newly added tasks can be deduplicated, thereby releasing occupied system resources and reducing the pressure on the system.

[0071] In an exemplary embodiment, in step S120, when a newly added task is determined to be a valid task based on the task type, before obtaining the first task array data corresponding to the newly added task, it also includes: querying a valid task type table based on the task type; when it is determined that there is a task type in the valid task type table that matches the task type of the newly added task, determining that the newly added task is a valid task.

[0072] The valid task type table stores the valid task types to be processed. The valid task types to be processed may be types that the system is capable of processing. In one example, if the system detects the ability to perform target detection on images in the current tasks to be processed, the image processing task type may be included in the valid task list. If the system has the ability to perform speech recognition on speech, the speech processing task type may be included in the valid task list. The valid task type table may be updated regularly or in real time according to the processing tasks of the system to ensure the accuracy and consistency of the data in the system.

[0073] In one embodiment, the matching task types may be the same task types. In this case, after determining the task type of the newly added task, the server obtains the current valid task type list. The newly added task type is compared with the valid task types included in the valid task type list. If it is determined that the valid task list contains the same task type as the newly added task, the newly added task is determined to be a valid task.

[0074] In another embodiment, the matched task type can be an associated task type. For example, for a video creation scene, an image processing task and an audio processing task are associated tasks. In this case, after determining the task type of the newly added task, the server obtains a current valid task type list. The newly added task type is compared with the valid task types included in the valid task type list. If it is determined that there is a valid task type associated with the task type of the newly added task in the valid task list, the newly added task is determined to be a valid task.

[0075] In this embodiment, whether the newly added task is a valid task is determined according to the task type of the newly added task. After ensuring that the newly added task is a task to be processed by the system, the array data in the newly added task is processed, which can avoid high concurrency in task processing and speed up task processing efficiency.

[0076] In an exemplary embodiment, in step S120, when a newly added task is determined to be a valid task based on the task type, before obtaining the first task array data corresponding to the newly added task, it also includes: determining the host quantity condition and the host memory usage condition required to process the newly added task based on the task type of the newly added task; if the current number of idle hosts meets the host quantity condition, and the memory usage of the current idle hosts meets the host memory usage condition, then the newly added task is determined to be a valid task.

[0077] Specifically, a resource requirement table may be preset in the server, and the resource requirement table includes a mapping relationship between the task type and the host quantity condition and the host utilization rate condition. The mapping relationship may be obtained by analyzing the host quantity condition and the host utilization rate condition actually required by a number of processed tasks, or may be predicted based on the host quantity condition and the host utilization rate condition actually required by the processed tasks.

[0078] The server queries the host resource requirement table according to the task type of the newly added task to obtain the host quantity condition and host memory usage condition required to process the newly added task. The host quantity condition is compared with the current number of idle hosts, and the host memory usage condition is compared with the memory usage of the current idle hosts. If the host quantity condition is less than or equal to the current host quantity condition, and the host usage condition is less than or equal to the memory usage of the current idle hosts, the newly added task is determined to be a valid task.

[0079] In this embodiment, whether the newly added task is a valid task is determined based on the host resources required by the newly added task, and the newly added task is processed when the current host condition meets the new task condition, which can ensure normal execution of the processing and reduce the error rate.

[0080] In an exemplary embodiment, a data deduplication function is built into the first task array data; step S130, the first task array data is compared with the second task array data already existing in the database, and when it is determined that duplicate data exists between the first task array data and the second task array data, the duplicate data in the first task array data is removed, including: using the data deduplication function, the data in the first task array data is compared one by one with the data in the second task array data, and when it is determined that duplicate data exists between the first task array data and the second task array data, the duplicate data in the first task array data is removed.

[0081] Among them, the data deduplication function (unique function) at least defines the array element comparison method, for example, comparing in order from left to right; the basis for judging duplicate data, for example, confirming array elements with the same content as duplicate data, confirming array elements with the same attributes as duplicate data, etc.

[0082] Specifically, a data deduplication function is pre-built into the array prototype of the first task array data. After the server obtains the first task array data, array elements are obtained one by one from the first task array data according to the array element comparison method defined in the array deduplication function, and the obtained array elements are compared with the array elements in the second task array data. Whether the array element is duplicate data is determined according to the first duplicate data in the array deduplication function, and if so, the duplicate data is removed from the first task array data.

[0083] In this embodiment, a data deduplication method is built into the array prototype, and data deduplication can be achieved by executing native built-in functions. For large data arrays, optimization can be performed at the bottom of the engine to reduce the method execution time. In addition, the standardized native method can meet the vast majority of array element deduplication requirements, increasing code readability and maintainability.

[0084] In an exemplary embodiment, step S110, determining the task type of the newly added task, can be implemented in any of the following ways:

[0085] (1) Determine the task type of the new task based on the type of files included in the new task.

[0086] The file type may be, but is not limited to, a file type, an audio and video type, an image type, and an executable type. Specifically, if the server obtains that the file type included in the newly added task is a file type such as doc (document), excel (spreadsheet), pdf (Portable Document Format), etc., then the task type of the newly added task is determined to be a file task type. If the types of files included in the newly added task are audio and video types such as MP3 / MP3Pro (a music file), WAV (waveform sound file), MPEG (Moving Picture Experts Group), AVI (Audio Video Interleaved), nAVI (a new video format), ASF (Advanced Streaming Format), MOV (movie format), 3GP (a video encoding container format for 3G streaming media), WMV (a media video format), DivX (digital multimedia compression format), XviD (a video codec), RM (video container file format), RMVB (variable bit rate extension version of multimedia digital container format), FLV / F4V (a video editor), etc., then the task type of the newly added task is determined to be an audio and video task type. If the types of files included in the newly added task are image types such as JPEG (an image file format), TIFF (standard image file format), RAW (an image file format), etc., then the task type of the newly added task is determined to be an image task type.

[0087] (2) Determine the task type of the new task based on the task name of the new task.

[0088] If the task name of the newly added task contains keywords related to the file type (such as document, text, etc.), the task type of the newly added task is determined to be a file task type; if the task name of the newly added task contains keywords related to the audio and video type (such as resource, video, audio, voice, etc.), the task type of the newly added task is determined to be an audio and video task type. If the task name of the newly added task contains keywords related to the image type (such as image, picture, etc.), the task type of the newly added task is determined to be an image task type.

[0089] In this embodiment, it supports processing of multiple new task types, so that the processing method of new tasks can support the processing of multiple application scenarios and multiple data types, expanding the scope of application and flexibility of the processing method of new tasks, so that this method can meet more comprehensive array deduplication needs.

[0090] In an exemplary embodiment, before determining the task type of the newly added task in step S110, it also includes: receiving a processing request for the newly added task; in response to the processing request, when querying whether there are idle channel resources for processing the newly added task, selecting idle channel resources corresponding to the emergency level priority from the resource pool according to the emergency level priority of the newly added task; and allocating the idle channel resources to the newly added task.

[0091] The emergency level priority indicates the processing time of the newly added task. The corresponding emergency level priority can be assigned to the newly added task according to the latency requirements and processing timing of the newly added task. For example, if the newly added task is the current real-time recommendation task, a higher emergency level priority is configured; if the newly added task is an offline update task, a lower emergency level priority is configured.

[0092] The mapping relationship between the emergency level priority and the channel resources can be preconfigured. In the mapping relationship, the higher the emergency level priority, the more corresponding channel resources (eg, the number of channels).

[0093] Specifically, the processing request for the newly added task can be a request received in response to a user's trigger operation; it can also be a request triggered autonomously by the system, for example, when the system obtains a newly added task from the task queue, it autonomously triggers the processing request for the newly added task. In response to the processing request, the server queries whether there are idle channel resources. If so, it selects an idle channel resource that matches the emergency level priority of the newly added task from the resource pool, and processes the newly added task through the idle channel resource.

[0094] In this embodiment, an emergency level priority is configured for the newly added tasks, and corresponding channel resources are allocated to the newly added tasks according to the urgency of the tasks, so that the newly added tasks with higher urgency can be processed first, thereby optimizing the processing time efficiency of the newly added tasks.

[0095] Figure 2 is a flowchart of a method for processing a newly added task according to an exemplary embodiment. Figure 2 As shown, the method for processing the newly added task is used in the server and includes the following steps.

[0096] In step S202, a processing request for a newly added task is received.

[0097] In step S204, in response to the processing request, it is checked whether there are idle channel resources for processing the newly added task. If so, proceed to step 206; otherwise, execute step S220.

[0098] In step S206, according to the emergency level priority of the newly added task, idle channel resources corresponding to the emergency level priority are selected from the resource pool, and the idle channel resources are allocated to the newly added task.

[0099] In step S208, the task type of the newly added task is determined according to the type of the file included in the newly added task or the task name of the newly added task. The specific method of determining the task type can refer to the above embodiment and will not be elaborated here.

[0100] In step S210, according to the task type, the host quantity condition and the host memory usage condition required for the newly added task are determined.

[0101] In step S212, if the current number of idle hosts meets the host number condition and the memory usage of the current idle hosts meets the host memory usage condition, the newly added task is determined to be a valid task, and step S214 is continued; otherwise, step S220 is executed.

[0102] In step S214, the first task array data corresponding to the newly added task is obtained, and the first task array data is compared with the second task array data already existing in the database through the data deduplication function built into the first task array data. If it is determined that there is duplicate data between the first task array data and the second task array data, step S216 is executed; if it is determined that there is no duplicate data between the first task array data and the second task array data, step S218 is executed.

[0103] In step S216 , duplicate data in the first task array data is removed, and the first task array data after the duplicate data is removed is stored in a database.

[0104] In step S218 , the first task array data is stored in a database.

[0105] In step S220, the processing of the newly added task is stopped, and the next processing time of the newly added task is waited for to arrive.

[0106] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0108] Figure 3 FIG. 3 is a block diagram of a processing device 300 for a newly added task according to an exemplary embodiment. Figure 3 The device includes a task type determination module 302 , a data acquisition module 304 , and a data deduplication module 306 .

[0109] The task type determination module 302 is configured to determine the task type of the newly added task by executing the task type determination; the data acquisition module 304 is configured to obtain the first task array data corresponding to the newly added task when the newly added task is determined to be a valid task according to the task type; the data deduplication module 306 is configured to compare the first task array data with the second task array data already existing in the database, and when it is determined that the first task array data and the second task array data have duplicate data, remove the duplicate data in the first task array data.

[0110] In an exemplary embodiment, the device 400 also includes: a query module, configured to query a valid task type table according to the task type; a first task processing module, configured to determine that the newly added task is a valid task when it is determined that there is a task type in the valid task type table that matches the task type of the newly added task; wherein the valid task type table stores valid task types to be processed.

[0111] In an exemplary embodiment, the device 400 also includes: a processing resource determination module, which is configured to execute a condition on the number of hosts and a condition on the host memory usage rate required to process the newly added task according to the task type of the newly added task; a second task processing module, which is configured to execute a condition on the number of hosts and the host memory usage rate required to process the newly added task if the current number of idle hosts meets the host number condition and the memory usage rate of the current idle hosts meets the host memory usage rate condition.

[0112] In an exemplary embodiment, a data deduplication function is built into the first task array data; the data deduplication module 306 is configured to execute the data deduplication function to compare the data in the first task array data with the data in the second task array data one by one, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data.

[0113] In an exemplary embodiment, the task type determination module 302 is configured to determine the task type of the new task according to the type of files included in the new task; or determine the task type of the new task according to the task name of the new task.

[0114] In an exemplary embodiment, the device 400 also includes: a receiving module, configured to execute receiving a processing request for a newly added task; a resource allocation module, configured to execute in response to the processing request, when querying whether there are idle channel resources for processing the newly added task, selecting idle channel resources corresponding to the emergency level priority from the resource pool according to the emergency level priority of the newly added task; and allocating the idle channel resources to the newly added task.

[0115] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0116] Figure 4 is a block diagram of an electronic device S00 for processing a newly added task according to an exemplary embodiment. For example, the electronic device S00 may be a server. Figure 4 The electronic device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22 for storing instructions executable by the processing component S20, such as an application. The application stored in the memory S22 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0117] The electronic device S00 may further include: a power supply component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input / output (I / O) interface S28. The electronic device S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD or the like.

[0118] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory S22 including instructions, and the above instructions can be executed by a processor of the electronic device S00 to complete the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0119] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by a processor of the electronic device S00 to complete the above method.

[0120] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for processing a newly added task, characterized in that: include: Determine the task type of the newly added task; When the newly added task is determined to be a valid task according to the task type, obtaining first task array data corresponding to the newly added task, wherein the first task array data has a built-in data deduplication function; Through the data deduplication function, the data in the first task array data is compared with the data in the second task array data already existing in the database. When it is determined that there is duplicate data between the first task array data and the second task array data, the duplicate data in the first task array data is removed.

2. The method for processing a newly added task according to claim 1, characterized in that: When the newly added task is determined to be a valid task according to the task type, before obtaining the first task array data corresponding to the newly added task, the method further includes: According to the task type, query the valid task type table; When it is determined that there is a task type matching the task type of the newly added task in the valid task type table, the newly added task is determined to be a valid task; wherein the valid task type table stores valid task types to be processed.

3. The method for processing a newly added task according to claim 1, characterized in that: When the newly added task is determined to be a valid task according to the task type, before obtaining the first task array data corresponding to the newly added task, the method further includes: Determine, according to the task type of the newly added task, the number of hosts and the host memory usage required to process the newly added task; If the current number of idle hosts meets the host number condition, and the memory usage of the current idle hosts meets the host memory usage condition, then the newly added task is determined to be a valid task.

4. The method for processing a newly added task according to any one of claims 1 to 3, characterized in that: The step of determining the task type of the newly added task includes: Determining a task type of the newly added task according to the type of the file included in the newly added task; or, Determine the task type of the newly added task according to the task name of the newly added task.

5. The method for processing a newly added task according to any one of claims 1 to 3, characterized in that: Before determining the task type of the newly added task, the method further includes: receiving a processing request for the newly added task; In response to the processing request, when querying whether there is an idle channel resource for processing the newly added task, select an idle channel resource corresponding to the emergency level priority from a resource pool according to the emergency level priority of the newly added task; Allocate the idle channel resources to the newly added task.

6. A device for processing a newly added task, characterized in that: include: A task type determination module is configured to perform task type determination to determine the task type of a newly added task; A data acquisition module is configured to acquire first task array data corresponding to the newly added task when the newly added task is determined to be a valid task according to the task type, wherein the first task array data has a built-in data deduplication function; The data deduplication module is configured to execute the data deduplication function to compare the data in the first task array data with the data in the second task array data already existing in the database, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data.

7. The device for processing a newly added task according to claim 6, characterized in that: The device also includes: A query module is configured to query a valid task type table according to the task type; The first task processing module is configured to determine that the newly added task is a valid task when it is determined that there is a task type matching the task type of the newly added task in the valid task type table; wherein the valid task type table stores valid task types to be processed.

8. The device for processing a newly added task according to claim 6, characterized in that: The device also includes: A processing resource determination module is configured to determine the number of hosts and the host memory usage rate required to process the newly added task according to the task type of the newly added task; The second task processing module is configured to determine that the newly added task is a valid task if the current number of idle hosts meets the host number condition and the memory usage of the current idle hosts meets the host memory usage condition.

9. The device for processing a newly added task according to claim 6, characterized in that: The first task array data has a built-in data deduplication function; The data deduplication module is configured to execute the data deduplication function to compare the data in the first task array data with the data in the second task array data one by one, and when it is determined that there is duplicate data between the first task array data and the second task array data, remove the duplicate data in the first task array data.

10. The device for processing a newly added task according to any one of claims 6 to 9, characterized in that: The task type determination module is configured to determine the task type of the newly added task according to the type of the file included in the newly added task; or determine the task type of the newly added task according to the task name of the newly added task.

11. The device for processing a newly added task according to any one of claims 6 to 9, characterized in that: The device also includes: A receiving module, configured to receive a processing request for the newly added task; A resource allocation module is configured to execute in response to the processing request, when querying whether there is an idle channel resource for processing the newly added task, select an idle channel resource corresponding to the emergency level priority from a resource pool according to the emergency level priority of the newly added task; Allocate the idle channel resources to the newly added task.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for processing a newly added task as described in any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the processing method for the newly added task as described in any one of claims 1 to 5.

14. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the method for processing a newly added task as described in any one of claims 1 to 5.

Images