Data processing task AI distribution and processing method based on deep learning model

Abstract

The invention provides a data processing task AI allocation and processing method based on a deep learning model, and aims to perform deep learning model training and construct a task AI allocation model through the self-adaption and learning ability of the deep learning model and based on big data formed by historical task allocation logs. And the task AI distribution model is utilized to identify the task characteristics of the real-time tasks, so that the system can conveniently perform corresponding node distribution according to the task characteristics of the data processing tasks, intelligent task distribution is realized by utilizing the task AI distribution model, manual distribution is replaced, and the data processing task distribution efficiency is greatly improved. And the task features of the corresponding tasks can be synchronized to the corresponding task nodes, so that the task nodes can perform deep task processing based on the task features, corresponding data analysis is realized according to the task features, and intelligent distribution and efficient processing of the data processing tasks are realized.

Description

Technical Field

[0001] The present invention relates to the field of data intelligent processing technology, and in particular to a method for AI allocation and processing of data processing tasks based on a deep learning model, an apparatus for AI allocation and processing of data processing tasks based on a deep learning model, an electronic device and a computer-readable storage medium. Background Art

[0002] Data processing tasks, such as data cleaning, data analysis, data conversion, data integration, data storage, data security and privacy protection, data visualization and data mining, are important components of data intelligent processing. With the increasing amount of data, how to efficiently process this data, especially how to reasonably allocate and process AI tasks, has become an urgent problem to be solved.

[0003] Traditional data processing methods often rely on fixed rules and preset processes, and are difficult to adapt to complex and changeable data processing needs. For example, the invention patent with publication number CN109947556A provides a task allocation method: sending a computing task command to the node chip of the data processing device; each node chip of the data processing device performs calculations according to their respective tasks; the data processing device returns the calculation results to the external control device. This technology requires sending corresponding task allocation instructions to the corresponding node chips in series, so as to achieve the allocation of computing tasks of the serial node chips through less instruction interaction. This kind of task allocation is only suitable for task allocation with fixed processes (that is, the mode of interaction process is clearly defined between its node chips).

[0004] However, the existing big data contains data processing tasks with various needs, such as big data cleaning, on-demand analysis, and data processing of various permissions or professions, which often involve several computing nodes (even hundreds or thousands). If the mode of sending computing task commands to the corresponding computing nodes according to task attributes is adopted, it will undoubtedly take up a lot of time for task allocation calculations in the early stage of task allocation, which reduces the efficiency of data processing.

[0005] In addition, during the task assignment process, administrators are required to monitor the progress and correctness of task assignments and to confirm the assignments. Administrators need to review the assigned tasks and the results of task assignments, which will undoubtedly still take up a lot of manual review time, leading to delayed task issuance and assignment, resulting in delayed execution of data processing tasks and low efficiency. Summary of the invention

[0006] In order to solve the technical problems existing in the prior art, the present invention provides the following technical solutions: On the one hand, a method for AI allocation and processing of data processing tasks based on a deep learning model is provided, the method being implemented by an electronic device, the method comprising: S1. Build and deploy a task AI allocation model based on historical task allocation big data; S2, receiving and parsing task data packets to obtain several data processing tasks; S3. Through the preset task AI allocation model, traverse each of the data processing tasks, identify and output the task allocation features of the data processing tasks; S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes; S5. Synchronously sending the task allocation characteristics of the data processing task to the allocated task nodes.

[0007] Preferably, S1, based on historical task allocation big data, build and deploy a task AI allocation model, including: Collecting the allocation log data of several historical tasks from the task log library, and obtaining the historical task allocation big data after preprocessing, wherein the allocation log data includes at least one of the following task information: task attributes and the assigned task nodes, wherein the task attributes include task type, parent-child task relationship, task authority, task size and / or task text keywords; Performing feature extraction on the historical task allocation big data to obtain a task allocation feature set of the historical task allocation big data includes: Traversing the allocation log data of each historical task in the historical task allocation big data; By using a text mining algorithm, task allocation features in the allocation log data of each historical task are mined and extracted, and after feature integration processing, the task allocation feature set consisting of the task allocation features of each historical task is obtained; Dividing the task allocation feature set into a training set and a validation set according to a preset ratio; Input the training set into a preset deep learning model, use the task allocation features in the training set to train the model, and when a preset optimization iteration stop condition is reached, stop the training to generate the task AI allocation model; Use the validation set to perform performance validation on the task AI allocation model: If the verification is successful, the task AI allocation model is deployed to the task manager, and the model application parameters are configured; Otherwise, repeat the above model training steps.

[0008] Preferably, when the allocation log data of several historical tasks are collected from the task log library, the following is also included: constructing search terms and search conditions for collecting the distribution log data according to the task attributes of the distribution log data; Based on the search term and the search condition, a large language model prompt word is generated, and the large language model prompt word is input into a preset LLM large language model; Through the LLM large language model, the task log library is searched based on the large language model prompt words, and the allocation log data of the corresponding historical tasks are retrieved and collected from the task log library and output.

[0009] Preferably, S2, receiving and parsing a task data packet to obtain a number of data processing tasks, including: Receive the task data packet sent by the administrator through the task receiver; After parsing the task data packet to obtain a plurality of the data processing tasks, each of the data processing tasks is sent to the task manager in sequence.

[0010] Preferably, S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes, includes: The task manager sequentially reads and identifies the task allocation characteristics of the data processing tasks, identifies the task attributes from the task allocation characteristics, and sends the current data processing tasks and their task attributes to the router; The router receives the current data processing task and forwards the current data processing task to the corresponding task node according to the task type, parent-child task relationship, task authority, task size and / or task text keywords in the task attributes.

[0011] Preferably, the router is pre-configured with task attributes of the data processing tasks adapted by each task node.

[0012] On the other hand, a data processing task AI allocation and processing device based on a deep learning model is provided, and the data processing task AI allocation and processing device based on a deep learning model is used to implement the data processing task AI allocation and processing method based on a deep learning model, and the device includes: A task receiver is used to receive and parse task data packets to obtain a number of data processing tasks; A task manager, used to traverse each of the data processing tasks through the preset task AI allocation model, identify and output the task allocation characteristics of the data processing tasks; A router is used to read the task attributes in the task allocation characteristics of the data processing task, and allocate the data processing task to the corresponding task node according to the task attributes; and synchronously send the task allocation characteristics of the data processing task to the allocated task node.

[0013] On the other hand, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for AI allocation and processing of data processing tasks based on deep learning models is implemented.

[0014] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for AI allocation and processing of data processing tasks based on deep learning models.

[0015] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: The present invention proposes a data processing task AI allocation and processing method based on a deep learning model, which aims to train a deep learning model based on the big data formed by the historical task allocation log through the self-adaptation and learning ability of the deep learning model, build a task AI allocation model, and use the task AI allocation model to identify the task characteristics of real-time tasks, so that the system can allocate corresponding nodes according to the task characteristics of each data processing task, thereby using the task AI allocation model to realize intelligent task allocation, replace manual allocation and greatly improve the efficiency of data processing task allocation. And the present invention can synchronize the task characteristics of the corresponding task to the corresponding task node, so that the task node can perform task depth processing based on the task characteristics, realize the corresponding data analysis according to the task characteristics, and realize the intelligent allocation and efficient processing of data processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a flow chart of a method for AI allocation and processing of data processing tasks based on a deep learning model provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of an application flow of task AI allocation provided by an embodiment of the present invention; Figure 3 It is a flow chart of data retrieval using the LLM large language model provided by an embodiment of the present invention; Figure 4 It is a block diagram of a data processing task AI allocation and processing device based on a deep learning model provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0019] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0020] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0021] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0022] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0023] The embodiment of the present invention provides a method for allocating and processing data processing tasks based on deep learning models. The method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of the AI ​​allocation and processing method for data processing tasks based on a deep learning model is shown. The processing flow of the method may include the following steps: S1. Build and deploy a task AI allocation model based on historical task allocation big data; S2, receiving and parsing task data packets to obtain several data processing tasks; S3. Through the preset task AI allocation model, traverse each of the data processing tasks, identify and output the task allocation features of the data processing tasks; S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes; S5. Synchronously sending the task allocation characteristics of the data processing task to the allocated task nodes.

[0024] The following is combined with Figure 2 The implementation principle of the present invention will be described in detail.

[0025] Each time the system performs task assignment, the system can record the assignment records of each task, that is, the task assignment log and save it to the task log library (assignment log). Therefore, the present invention can use the historical task assignment logs in the library to perform big data feature learning, train and build the corresponding task AI assignment model, so as to identify the attributes of each task and output the adapted task node for executing the current data processing task, so as to realize task AI assignment. It is also possible to let the task node perform deep data processing based on the task features identified by the model, including data feature mining, association identification and keyword utilization.

[0026] The data processing tasks involved in the present invention, such as data cleaning, data analysis, data conversion, data integration, data storage, data security and privacy protection, data visualization and data mining, can be constructed according to specific task requirements and needs.

[0027] The present invention utilizes the task AI allocation model to identify the task characteristics of real-time tasks, so that the system can perform corresponding node allocation according to the task characteristics of each data processing task.

[0028] The task AI allocation model is built based on historical task allocation big data. It can learn the task characteristics of data processing tasks corresponding to different task attributes and the task nodes they are adapted to, so that the model can quickly identify the attributes of the current task and deliver it to the router for task forwarding, allowing the router to issue the current task to the corresponding task node based on the task attributes.

[0029] Therefore, the present invention uses the task AI allocation model to realize intelligent task allocation, replacing manual allocation and greatly improving the efficiency of data processing task allocation. In addition, the present invention can synchronize the task features of the corresponding tasks to the corresponding task nodes, so that the task nodes can perform task depth processing based on the task features, realize corresponding data analysis according to the task features, and realize intelligent allocation and efficient processing of data processing tasks.

[0030] Preferably, S1, based on historical task allocation big data, build and deploy a task AI allocation model, including: Collecting the allocation log data of several historical tasks from the task log library, and obtaining the historical task allocation big data after preprocessing, wherein the allocation log data includes at least one of the following task information: task attributes and the assigned task nodes, wherein the task attributes include task type, parent-child task relationship, task authority, task size and / or task text keywords; Performing feature extraction on the historical task allocation big data to obtain a task allocation feature set of the historical task allocation big data includes: Traversing the allocation log data of each historical task in the historical task allocation big data; By using a text mining algorithm, task allocation features in the allocation log data of each historical task are mined and extracted, and after feature integration processing, the task allocation feature set consisting of the task allocation features of each historical task is obtained; Dividing the task allocation feature set into a training set and a validation set according to a preset ratio; Input the training set into a preset deep learning model, use the task allocation features in the training set to train the model, and when a preset optimization iteration stop condition is reached, stop the training to generate the task AI allocation model; Use the validation set to perform performance validation on the task AI allocation model: If the verification is successful, the task AI allocation model is deployed to the task manager, and the model application parameters are configured; Otherwise, repeat the above model training steps.

[0031] In the task log library, a large amount of historical task allocation logs are usually collected and stored. Therefore, big data identification and training learning can be performed based on the allocation log data of several historical tasks stored and collected in the library to construct the AI ​​allocation model of the present invention. Therefore, the historical task allocation big data can be collected from the library, and the collected big data is cleaned and screened. After preprocessing, the historical task allocation big data can be obtained. The allocation log data needs to include the following task information: for each historical task allocation log, the task attributes of the task and the task nodes assigned to the task need to be recorded. The task attributes include the task type of the task, whether the task has a parent-child task relationship, such as the task is a subtask or a parent task of a task, and the execution of the subtask needs to rely on the parent task, so these task relationships need to be attributed. The task authority, task size or task text keywords also involve the relevant attributes of historical tasks, and they need to be adaptively allocated according to the authority size of the current task, etc.

[0032] After collecting the allocation history log data of several historical tasks, feature engineering is required to extract data features from the allocation log big data in order to extract the task allocation features of each historical task.

[0033] The present invention mines and extracts the task allocation features in the allocation log data of each historical task through a text mining algorithm, and obtains the task allocation feature set composed of the task allocation features of each historical task after feature integration processing.

[0034] The task allocation features in the allocation log data of each historical task can be mined and extracted using a text mining algorithm. The following steps can be followed: 1. Text segmentation: Perform word segmentation on the log data to cut the continuous text into independent words or phrases to facilitate subsequent feature extraction.

[0035] 2. Feature Extraction ‌Keyword-based feature extraction‌: Identify keywords related to task assignment in the logs, such as "assignment", "task", "user", etc.

[0036] Count the frequency of these keywords appearing in the log as one of the features.

[0037] ‌Feature extraction based on template matching‌: Based on common patterns of task assignments, templates are built to match relevant information in the logs.

[0038] For example, a regular expression may be used to match a sentence pattern such as "task X is assigned to user Y" and extract the task ID and user ID as features.

[0039] ‌Feature extraction based on natural language processing (NLP): Use named entity recognition (NER) technology to identify entity information in the log, such as task name, user name, allocation time, etc.

[0040] Use dependency syntactic analysis to understand sentence structure and extract key information such as subject, predicate and object of task assignment.

[0041] ‌Feature extraction based on machine learning‌: Use unsupervised learning methods, such as cluster analysis, to group log data and discover underlying task allocation patterns.

[0042] Use supervised learning methods, such as classification algorithms, to train on known types of task assignments and then classify new logs and extract features.

[0043] In this embodiment, the features extracted from the assignment log of each task include the attribute features in the task attributes and the information feature values ​​(or feature information, etc.) of the assigned task nodes, including the task type, parent-child task relationship, task authority, task size and / or feature relationships and feature values ​​in the task text keywords, etc. Specifically: Characteristic data range: ‌Task Type‌: can be the parameter value of the task type, for example, a type A task represents a data cleaning task, etc.; ‌Parent-child task relationship‌: Features include: parent-child relationship (one task is a subtask of another task), peer relationship (there is no direct parent-child relationship between tasks), dependency relationship (the completion of one task depends on the completion of another task), etc.

[0044] ‌Task permissions‌: Features include: high permissions (allowing sensitive operations), medium permissions (allowing general operations), low permissions (allowing only basic operations), no permissions (not allowing any operations), etc.

[0045] ‌Task size‌: Features are measured by indicators such as data volume, computational complexity, and execution time, such as: small tasks (small data volume, simple computation, and short execution time), medium tasks (moderate data volume, average computational complexity, and medium execution time), and large tasks (large data volume, complex computation, and long execution time).

[0046] ‌Task text keywords‌: Features depend on the specific content of the task, and may include task-related professional terms, operating instructions, target descriptions, etc. For example, in a network security task, keywords may include "network attack", "defense strategy", "vulnerability scanning", etc.

[0047] The allocation features corresponding to different historical tasks include their task attribute features and allocation relationship features, which facilitate the subsequent model to match the corresponding allocation relationship according to the identified task attribute features, and automatically allocate the current task to the corresponding task node based on the allocation relationship.

[0048] 3. Feature Selection and Optimization ‌Feature Selection‌: The most relevant features for task assignment are selected from the extracted features, and redundant and irrelevant features are removed.

[0049] Feature selection can be performed using statistical methods, machine learning methods, or domain knowledge.

[0050] ‌Feature Optimization‌: The selected features are further processed and optimized, such as normalization and standardization, to improve the usability and accuracy of the features.

[0051] Through the above steps, text mining algorithms can be used to effectively mine and extract features from historical task allocation log data, providing strong support and guidance for task allocation.

[0052] The feature integration process includes: Feature encoding: Convert the extracted features into quantifiable values ​​or classification labels.

[0053] Data normalization: standardize the feature values ​​to facilitate subsequent analysis; Feature selection: Select the most influential features based on task requirements; Model training: Use the extracted features to train classification or regression models; Model evaluation: Verify model performance and ensure the effectiveness of feature extraction.

[0054] For deep learning models, you can use random forests or decision trees; or choose a suitable deep learning framework such as TensorFlow or PyTorch based on the data scale and feature complexity. This is up to the user to choose.

[0055] The task allocation feature set extracted by feature engineering is divided into a training set and a validation set; the training set is input into a preset deep learning model, and the model is trained using the task allocation features in the training set. When the preset optimization iteration stop condition is reached, the training is stopped to generate the task AI allocation model.

[0056] Optimize the iteration stop condition, such as when the recognition performance of the verification model reaches the preset verification condition (determined according to the verification mode, such as calculating the accuracy, recall rate, F1 score and other indicators to evaluate the performance of the model), then stop the iterative training. During the model iteration, manually labeled task allocation features can be used to correct the recognition accuracy and performance of the model.

[0057] In this embodiment, for example, the steps of using the random forest model to perform model training may be as follows: Collect and organize relevant data (training set) for task assignment; Preprocess the data, including missing value processing, outlier processing and data standardization; Determine the parameter settings of the random forest model, such as the number of trees, tree depth, etc. Divide the dataset into training set and test set (for example, 7:3); Train the random forest model using the training set data; Use the test set data to verify the model performance; Adjust model parameters to optimize model performance; Use the trained random forest model to predict task assignment features.

[0058] Model verification can be performed using indicators such as accuracy, recall, and F1 score, which are specifically performed by the administrator.

[0059] If the verification is successful, the task AI allocation model is deployed to the task manager and the model application parameters are configured. The specific model working parameters can be configured by the administrator.

[0060] The task manager can perform AI task allocation for each data processing task based on the deployed model and manage task allocation intelligently.

[0061] like Figure 3 As shown, preferably, when the allocation log data of several historical tasks are collected from the task log library, it also includes: constructing search terms and search conditions for collecting the distribution log data according to the task attributes of the distribution log data; Based on the search term and the search condition, a large language model prompt word is generated, and the large language model prompt word is input into a preset LLM large language model; Through the LLM large language model, the task log library is searched based on the large language model prompt words, and the allocation log data of the corresponding historical tasks are retrieved and collected from the task log library and output.

[0062] In order to improve data processing efficiency and speed up model training and construction, the present invention adopts the LLM large language model to assist in data collection. The LLM large language model can adopt an existing large language model such as Baidu's Wenxinyiyan. The administrator can log in and call the LLM large language model through the system API. Or directly deploy an LLM large language model on the system (device background). For the deployment and application of the LLM large language model, reference can be made to the prior art.

[0063] The LLM large language model retrieves data from the library based on the input prompt words. The steps can be referred to as follows: ‌Data Preparation‌: Collect and prepare large amounts of text data, including training data and retrieval data, which can be obtained from the Internet according to the needs of specific use cases.

[0064] ‌Build a retrieval system‌: Use existing search engine technology or more advanced deep learning techniques, such as Transformer-based retrieval models, to build a retrieval system for retrieving relevant information from large-scale text corpora.

[0065] ‌Data Retrieval‌: When a prompt word is input, the LLM large language model first retrieves information related to the prompt word from the text corpus through the retrieval system and generates a response: The retrieved information is used as context to enhance the generative capabilities of the LLM, enabling it to provide more precise and contextually appropriate responses.

[0066] This process ensures that the LLM large language model can efficiently process and respond to user input and provide accurate and useful information.

[0067] Therefore, in the early stage, you need to input the corresponding large language model prompt words into the LLM large language model. The large language model prompt words need to be constructed in combination with the search words and search conditions for the assigned log data. The search words and search conditions for the assigned log data can refer to the attribute words and keywords in each task attribute, and the search conditions can refer to the task attributes with dependency / parent-child relationships to construct the search conditions.

[0068] Specifically, the administrator can construct prompt words based on the attribute values ​​(type, authority, etc.) of the previously assigned log data and the task allocation relationship, and input them into the large language model.

[0069] like Figure 4 As shown, the application system composition of this system (device) is shown.

[0070] This device consists of a task receiver, a task manager, and a router, and the interactions are as follows: Preferably, S2, receiving and parsing a task data packet to obtain a number of data processing tasks, including: Receive the task data packet sent by the administrator through the task receiver; After parsing the task data packet to obtain a plurality of the data processing tasks, each of the data processing tasks is sent to the task manager in sequence.

[0071] Preferably, S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes, includes: The task manager sequentially reads and identifies the task allocation characteristics of the data processing tasks, identifies the task attributes from the task allocation characteristics, and sends the current data processing tasks and their task attributes to the router; The router receives the current data processing task and forwards the current data processing task to the corresponding task node according to the task type, parent-child task relationship, task authority, task size and / or task text keywords in the task attributes.

[0072] Preferably, the router is pre-configured with task attributes of the data processing tasks adapted by each task node.

[0073] The task manager can refer to the task manager in the existing system, and the task receiver can be a task receiving collection port or collection module, etc. It mainly receives the task data packet issued by the administrator through the management center, and parses the task data packet to obtain several data processing tasks.

[0074] Data processing tasks include tasks with different task attributes, so they need to be handed over to the task manager. The task manager calls the AI ​​model for intelligent allocation, and the assigned tasks will be routed to the corresponding task nodes by the router according to the task nodes that match the task attributes.

[0075] The router can pre-configure a mapping table of task attributes that are compatible with each task node. After receiving the task forwarded by the task manager, the router maps and matches the routing address according to the tasks distributed by the task manager and the task attributes it identifies. According to the task characteristics in the attributes, the current task route is sent to the task node that matches its attributes, thereby realizing intelligent and automatic task allocation.

[0076] The task manager can read and identify the task allocation characteristics of each data processing task in turn, and read the task attributes of the current task from the output task allocation characteristics. The AI ​​model will identify and output them, and then send the current task and its task attributes to the router, which will then map and allocate them according to the above-mentioned routing allocation method.

[0077] Routing configuration: Configure your router's routing rules to direct article requests to the script. This may involve network settings such as port forwarding and DNS configuration.

[0078] The components of the system router mainly include the following parts: ‌Hardware Components‌: ‌CPU‌: Responsible for performing the work required to process data packets, such as routing protocols, information exchange, etc.‌

[0079] ‌Memory‌: includes RAM, NVRAM, Flash memory and ROM, used to store system software, configuration files, etc.

[0080] ‌Interface‌: Includes console port, auxiliary port, LAN interface, and WAN interface for connecting to and managing the router.

[0081] ‌Others‌: such as power supply, casing, motherboard, wireless chip, etc.

[0082] ‌Software Components‌: ‌Bootloader‌: Used to start the router.

[0083] ‌Operating system‌: Such as IOS, responsible for the overall operation of the router.

[0084] ‌Configuration File‌: includes startup configuration file and running configuration file.

[0085] ‌Management Program‌: used for configuration and management of the router.

[0086] These components work together to ensure the proper functioning of the router and efficient data transmission.

[0087] The system router can also be changed into a message middleware, which can distribute tasks by using the message distribution mechanism based on KAFKA.

[0088] In addition, the present invention can also deploy a task AI allocation model in the router, so as to directly complete task management and intelligent distribution in the router, which can save the management resources of the task manager and reduce its operating pressure.

[0089] After the task manager sends the current task and its task attributes to the router, it can also synchronously forward the current data processing task to be assigned and its task allocation features to the corresponding task node through the router. Because the task manager has identified the task allocation features of the current data processing task through the AI ​​allocation model, and identified and output the task attributes of the current data processing task in advance, such as task type, permission replication, task relationship, etc., these task allocation features can be synchronously sent to the assigned task node through the router. After receiving the data processing task forwarded by the router, the task node does not need to perform task analysis, keyword mining, etc. It can directly use the task attributes in the task allocation features identified and output by the previous task manager, and directly enter the task analysis and processing process, which can provide a direct and fast task data processing mode for subsequent task nodes, reducing the data processing pressure of each task node below.

[0090] Figure 4 The present invention is a block diagram of a data processing task AI allocation and processing device based on a deep learning model according to an exemplary embodiment, and the device is used for a data processing task AI allocation and processing method based on a deep learning model. Figure 4 , a data processing task AI allocation and processing device based on a deep learning model, the data processing task AI allocation and processing device based on a deep learning model is used to implement the data processing task AI allocation and processing method based on a deep learning model, the device includes: A task receiver is used to receive and parse task data packets to obtain a number of data processing tasks; A task manager, used to traverse each of the data processing tasks through the preset task AI allocation model, identify and output the task allocation characteristics of the data processing tasks; A router is used to read the task attributes in the task allocation characteristics of the data processing task, and allocate the data processing task to the corresponding task node according to the task attributes; and synchronously send the task allocation characteristics of the data processing task to the allocated task node.

[0091] Please understand the composition and application of the above-mentioned device in conjunction with the above-mentioned method, and will not be elaborated here.

[0092] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include the above Figure 4 The data processing task AI allocation and processing device based on the deep learning model is shown. Optionally, the electronic device 410 may include a first processor 2001.

[0093] Optionally, the electronic device 410 may further include a memory 2002 and a transceiver 2003 .

[0094] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0095] Combine the following Figure 5 The components of the electronic device 410 are described in detail: The first processor 2001 is the control center of the electronic device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0096] Optionally, the first processor 2001 can execute various functions of the electronic device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0097] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0098] In a specific implementation, as an embodiment, the electronic device 410 may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0099] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0100] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0101] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0102] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0103] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0104] It should be noted that Figure 5 The structure of the electronic device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0105] In addition, the technical effects of the electronic device 410 can refer to the technical effects of the data processing task AI allocation and processing method based on the deep learning model described in the above method embodiment, which will not be repeated here.

[0106] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0107] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0108] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0109] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0110] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0111] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0112] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0117] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0118] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data processing task AI allocation and processing method based on a deep learning model, characterized in that: The method comprises: S1. Build and deploy a task AI allocation model based on historical task allocation big data; S2, receiving and parsing task data packets to obtain several data processing tasks; S3. Through the preset task AI allocation model, traverse each of the data processing tasks, identify and output the task allocation features of the data processing tasks; S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes; S5. Synchronously sending the task allocation characteristics of the data processing task to the allocated task nodes.

2. The method for AI allocation and processing of data processing tasks based on deep learning models according to claim 1, characterized in that: S1. Build and deploy a task AI allocation model based on historical task allocation big data, including: Collecting the allocation log data of several historical tasks from the task log library, and obtaining the historical task allocation big data after preprocessing, wherein the allocation log data includes at least one of the following task information: task attributes and the assigned task nodes, wherein the task attributes include task type, parent-child task relationship, task authority, task size and / or task text keywords; Performing feature extraction on the historical task allocation big data to obtain a task allocation feature set of the historical task allocation big data includes: Traversing the allocation log data of each historical task in the historical task allocation big data; By using a text mining algorithm, task allocation features in the allocation log data of each historical task are mined and extracted, and after feature integration processing, the task allocation feature set consisting of the task allocation features of each historical task is obtained; Dividing the task allocation feature set into a training set and a validation set according to a preset ratio; Input the training set into a preset deep learning model, use the task allocation features in the training set to train the model, and when a preset optimization iteration stop condition is reached, stop the training to generate the task AI allocation model; Use the validation set to perform performance validation on the task AI allocation model: If the verification is successful, the task AI allocation model is deployed to the task manager, and the model application parameters are configured; Otherwise, repeat the above model training steps.

3. The method for AI allocation and processing of data processing tasks based on deep learning models according to claim 2, characterized in that: When collecting the allocation log data of several historical tasks from the task log library, it also includes: constructing search terms and search conditions for collecting the distribution log data according to the task attributes of the distribution log data; Based on the search term and the search condition, a large language model prompt word is generated, and the large language model prompt word is input into a preset LLM large language model; Through the LLM large language model, the task log library is searched based on the large language model prompt words, and the allocation log data of the corresponding historical tasks are retrieved and collected from the task log library and output.

4. The method for AI allocation and processing of data processing tasks based on deep learning models according to claim 2, characterized in that: S2. Receive and parse the task data packet to obtain several data processing tasks, including: Receive the task data packet sent by the administrator through the task receiver; After parsing the task data packet to obtain a plurality of the data processing tasks, each of the data processing tasks is sent to the task manager in sequence.

5. The method for AI allocation and processing of data processing tasks based on deep learning models according to claim 2, characterized in that: S4, reading the task attributes in the task allocation feature of the data processing task, and allocating the data processing task to the corresponding task node according to the task attributes, including: The task manager sequentially reads and identifies the task allocation characteristics of the data processing tasks, identifies the task attributes from the task allocation characteristics, and sends the current data processing tasks and their task attributes to the router; The router receives the current data processing task and forwards the current data processing task to the corresponding task node according to the task type, parent-child task relationship, task authority, task size and / or task text keywords in the task attributes.

6. The method for AI allocation and processing of data processing tasks based on deep learning models according to claim 5, characterized in that: The router is pre-configured with task attributes of data processing tasks adapted by each task node.

7. A data processing task AI allocation and processing device based on a deep learning model, the data processing task AI allocation and processing device based on a deep learning model is used to implement the data processing task AI allocation and processing method based on a deep learning model as claimed in any one of claims 1 to 6, characterized in that: The device comprises: A task receiver is used to receive and parse task data packets to obtain a number of data processing tasks; A task manager, used to traverse each of the data processing tasks through the preset task AI allocation model, identify and output the task allocation characteristics of the data processing tasks; A router is used to read the task attributes in the task allocation characteristics of the data processing task, and allocate the data processing task to the corresponding task node according to the task attributes; and synchronously send the task allocation characteristics of the data processing task to the allocated task node.

8. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images