Task management method for data medium table

Through the task management method of the data middle platform, an IoT data resource pool was formed, which solved the problems of high computing costs and fixed result templates in the existing data middle platform, and realized efficient computing and personalized results push of the distributed computing platform, improving the Computational efficiency and data communication efficiency.

CN119094604AInactive Publication Date: 2024-12-06HUNAN CHAOHUA INTELLIGENT TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411152527.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing data middle platform has high cost, large layout area and fixed calculation result templates during the calculation process, which cannot meet the needs of operators. The calculation result needs to be sent by the client, which is inefficient.

Method used

Through the task management method of the data middle platform, an IoT data resource pool is formed, and the account permission module is used to restrict permissions and databases. Operators generate template call data, the task management module cuts data, distributes it to a distributed computing platform, performs encrypted calculations, and returns The result is pushed to the client.

Benefits of technology

It realizes the separation of the computing platform and the data middle platform, improves computing efficiency, reduces costs, meets the personalized needs of operators, and improves data communication efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094604A_ABST
    Figure CN119094604A_ABST
Patent Text Reader

Abstract

The invention discloses a task management method for a data medium table, and belongs to the field of data processing. The method specifically comprises the following steps: S1, a manager logs in at a back end and regulates and controls the operation of a data medium station; s2, generating a corresponding task, and adding the task into a task management module; s3, the data synchronization module synchronizes the corresponding data from the corresponding database to the data warehouse according to the task, and finally, packaged data is formed; after encryption, distributing to a distributed computing platform, and returning to obtain a computing result; s4, integrating the calculation results by using a task management module in the data middle table, and generating a result corresponding to a task result template; and S5, performing corresponding pushing on the result by using a pushing module. By the adoption of the method, rapid task generation is achieved, character segmentation is reasonably carried out, the segmented tasks can be reasonably and respectively calculated on a distributed calculation platform, the calculation efficiency can be effectively improved, and a task result can be rapidly obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data middle stations, and in particular to a task management method for a data middle station. Background Art

[0002] In the current big data processing in today's big data era, the use of data middle platforms can liberate the productivity of developers, greatly shorten the data value extraction process, and improve the ability of enterprises to extract data value. However, in the existing data middle platforms, data analysis and calculations are often only performed through local servers, which has high deployment costs and requires a larger area for deployment, which in turn increases costs. In addition, the templates of existing calculation results are fixed and cannot meet the needs of existing operators. At the same time, the calculation results are often sent only when the client makes a request. Summary of the invention

[0003] The purpose of the present invention is to provide a task management method for a data middle station, form a data resource pool of the Internet of Things, limit account permissions and callable databases through an account permission module, and operators use task generation templates to call data to form packaged data, and cut the packaged data through the task management module, respectively, into different calculation methods, so as to find a suitable architecture for calculation on a distributed computing platform, thereby having better computing efficiency, separating the computing platform and the data middle station, and using the data middle station to coordinate and classify data, thereby improving the computing efficiency of the computing platform.

[0004] To achieve the above object, the present invention provides a task management method for a data center, comprising the following steps:

[0005] S1: The administrator logs in to the backend to regulate the operation of the data center;

[0006] S2: The operator logs in according to the account authority module, enters the task data at the front desk, fills in the task generation template, generates the corresponding task, and adds the task to the task management module;

[0007] S3: The data synchronization module synchronizes the corresponding data from the corresponding database to the data warehouse according to the task. The task management module divides the task, generates computing tasks, and allocates the corresponding data in the data warehouse according to the computing tasks to form packaged data.

[0008] The task management module encrypts the packaged data, distributes it to the distributed computing platform, and returns the calculation results;

[0009] S4: Use the task management module in the data center to integrate the calculation results and generate results corresponding to the task result template;

[0010] S5: Use the push module to push the results accordingly.

[0011] Preferably, in step S1, the control process includes setting an account authority module, setting a task generation template, setting a task management module, setting a task result template, setting a push module and running maintenance;

[0012] Set up account permission modules, including departments, operators, available databases, and operation records;

[0013] The specific setting of available databases is to set the databases that operators can use according to the actual rights management of the department;

[0014] Set up task generation templates, including reference data content, data processing, task generation, target address, and task result templates;

[0015] The referenced data content specifically includes the database name, referenced content, start time and end time;

[0016] Data processing will record the requirements for data analysis or statistics on the referenced data content;

[0017] Task generation will package the corresponding reference data content, data processing, target address and task result template to obtain the task, and send the task to the task management module;

[0018] The task management module specifically includes task recording, task segmentation, calculation order, data encryption, task dispatching, and filling in task result templates;

[0019] Task record is responsible for recording all currently executed tasks; task segmentation is to segment tasks to obtain several computing tasks; the linear order of segmented computing tasks is set as the computing order; data encryption is to encrypt computing tasks and allocation data to obtain encrypted data; task dispatch is to package the encrypted data and send it to the distributed computing platform; fill in the task result template, the distributed computing platform decrypts and calculates the encrypted data, and returns the calculation result to the task management module, which fills the calculation result into the task result template according to the calculation order, and converts the target address to obtain the push address;

[0020] There are two ways to set up task result templates, namely fixed templates and custom templates. Fixed templates are templates provided by different departments when managers set up task generation templates. Custom templates are templates set up by operators themselves.

[0021] The push module is used to actively send content to the client.

[0022] Preferably, in step S3, the process of synchronizing the corresponding data to the data warehouse is as follows:

[0023] The data warehouse specifically includes the source library, data warehouse, index library and task library. The data synchronization module includes data access synchronization and data governance modules.

[0024] The working process of the data access synchronization module is as follows: obtain data from the data source, including two working modes, one is interface acquisition, the other is direct database synchronization; in interface acquisition, according to the referenced data content, obtain content from the database in a targeted and quantitative manner, and use regular expressions to verify the data; in direct database synchronization, use full or incremental synchronization to synchronize to the source library;

[0025] The data governance module standardizes the data in the source library, including generating data annotations and data indexes, including data extraction, data cleaning, data correction, data comparison and data association, and then generates data annotations and data indexes, which are stored in the data warehouse and index library respectively;

[0026] The task library is used to store the returned calculation results, relevant data of the task generation template, and relevant data of the task management module.

[0027] Preferably, in step S3, the process of segmenting and encrypting the packaged data is as follows:

[0028] The segmentation process is divided into two computing tasks: logical calculation and floating-point calculation according to the actual calculation content. The calculation order is set according to the linear order of the calculation tasks, the calculation order is recorded in the task management module, and the packaged data is encrypted.

[0029] Preferably, in step S6, the push module is specifically as follows:

[0030] The push module includes push code, content code, push type, push address, push process and push deduplication;

[0031] The push code is the identifier of the push process and is used to calibrate the push process;

[0032] The content code is the identifier of the content pushed this time, and is used to mark the pushed content;

[0033] Push types include HTTP, TCP and UDP;

[0034] The push address is the target address, and there are different push addresses for different push types;

[0035] The push process includes direct forwarding and processed forwarding. Direct forwarding is to push the content in the corresponding post source library, and processed forwarding is to push the content in the task library.

[0036] Push deduplication is used to remove duplicate pushed content, including the identification process and the identification cycle. The identification process will save the content code corresponding to the sent content. During the identification cycle, the sent content is deduplicated based on the content code.

[0037] Therefore, the present invention adopts the above-mentioned task management method of a data middle station, which has the following advantages:

[0038] In the present invention, the computing platform and the data middle station are separated. Instead of deploying a local computing server for computing, the computing tasks are classified by the data middle station, which are then sent to the distributed computing platform after encryption. The classified computing tasks can more easily find a better computing architecture in the distributed computing platform, thereby improving computing efficiency. A push module is also set up. Compared with the client looking for the data middle station to send a request, the corresponding data is actively sent to the client, which can effectively improve communication efficiency and avoid the client's constant contact during the period when the task is not completed.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flowchart of a task management method for a data center. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations. The specific model specifications need to be selected and determined according to the actual specifications of the device, and the specific selection calculation method adopts the existing technology in the field, so it will not be described in detail.

[0042] Example

[0043] like Figure 1 As shown, the present invention provides a task management method for a data center, comprising the following steps:

[0044] S1: The administrator logs in to the backend to regulate the operation of the data center. The regulation process includes setting the account permission module, setting the task generation template, setting the task management module, setting the task result template, setting the push module and operation and maintenance.

[0045] Set up an account permission module to record personnel permissions and operation processes, including departments, operators, available databases, and operation records;

[0046] The record department and operator are convenient for corresponding to the actual operator;

[0047] The specific setting of available databases is to set the databases that operators can use according to the actual rights management of the department, so as to avoid using the corresponding confidential databases;

[0048] Recording operation logs can clearly identify the operator's behavior at a later stage;

[0049] Set up a task generation template. The task generation template is mainly used to generate tasks and needs to be filled in by the operator. The task generation template includes reference data content, data processing, task generation, target address and task result template;

[0050] The referenced data content specifically includes the database name, referenced content, start time and end time;

[0051] Data processing will record the requirements for data analysis or statistics on the referenced data content;

[0052] Task generation will package the corresponding reference data content, data processing, target address and task result template to obtain the task, and send the task to the task management module;

[0053] The task management module specifically includes task recording, task segmentation, calculation order, data encryption, task dispatching, and filling in task result templates;

[0054] Task recording is responsible for recording all currently executed tasks; task segmentation is to segment tasks to obtain several computing tasks; the computing order is the linear order of the computing tasks after segmentation; data encryption is to encrypt computing tasks and allocation data to obtain encrypted data; task dispatch is to package the encrypted data and send it to the distributed computing platform; filling the task result template, the distributed computing platform decrypts and calculates the encrypted data, and returns the calculation result to the task management module, which fills the calculation result into the task result template according to the calculation order, and converts the target address to obtain the push address;

[0055] The task result template generates corresponding reports or data as required. There are two sources for the template: fixed template and custom template. Fixed templates are templates provided by different departments when managers set task generation templates. Custom templates are templates set by operators themselves, which are convenient for operators to customize and meet personalized needs.

[0056] The push module is used to actively send content to the client.

[0057] S2: The operator logs in according to the account authority module, enters the task data at the front desk, uses the task generation template to generate the corresponding task, and adds the task to the task management module;

[0058] S3: The data synchronization module synchronizes the corresponding data from the corresponding database to the data warehouse according to the task. The process of synchronizing the corresponding data to the data warehouse is as follows: The data warehouse specifically includes the source library, data warehouse, index library and task library. The data synchronization module includes data access synchronization and data governance modules.

[0059] The working process of the data access synchronization module is as follows: obtain data from various data sources, including two working modes, one is interface acquisition, and the other is direct database synchronization; in interface acquisition, according to the referenced data content, obtain content from the database in a targeted and quantitative manner, and use regular verification data; in direct database synchronization, use full or incremental synchronization to synchronize to the source library; the data governance module standardizes the data in the source library, including generating data annotations and data indexes, specifically including data extraction, data cleaning, data correction, data comparison and data association, and then generates data annotations and data indexes, which are stored in the data warehouse and index library respectively;

[0060] The task management module divides the tasks into two types of computing tasks: logical computing and floating-point computing according to the actual computing content, and sets the computing order according to the linear order of the computing tasks. The computing order is recorded in the task management module, and the computing tasks are generated. The corresponding data in the data warehouse is allocated according to the computing tasks to form packaged data. The task management module encrypts the packaged data, distributes it to the distributed computing platform, and returns the computing results.

[0061] The task library is used to store the returned calculation results, the relevant data of the task generation template, and the relevant data of the task management module;

[0062] S4: Use the task management module in the data center to integrate the calculation results and generate results corresponding to the task result template;

[0063] S5: Use the push module to push the results accordingly. The push module is as follows, including push code, content code, push type, push address, push process and push deduplication;

[0064] The push code is the identifier of the push process, which is used to mark the push process and facilitate tracking of the push process;

[0065] The content code is used to identify the content of this push and to indicate the specific content of this push.

[0066] Push types include HTTP, TCP and UDP. Different push types have different push addresses. For push type, enter the push address in the input box. For HTTP type, fill in the corresponding URL, such as http: / / 192.168.31.164:999 / dataAccess. For TCP and UDP types, fill in the IP and port, such as 192.168.31.164:999.

[0067] The push process includes direct forwarding and processed forwarding. Direct forwarding is to push the content in the corresponding post source library, and processed forwarding is to push the content in the task library; corresponding to whether to process the data or not

[0068] Push deduplication is used to remove duplicate pushed content, including the identification process and the identification cycle. The identification process will save the content code corresponding to the sent content. During the identification cycle, the sent content is deduplicated based on the content code.

[0069] Set the identification period to the time to save the content code. When it is set to 7 days, the content code will be saved for 7 days. During the 7 days of the identification period, when the same content code is identified in the content to be pushed, the content corresponding to the same content code in this content will be deleted. This can ensure that the same content is only sent once within the identification period, avoiding leakage caused by repeated information sending.

[0070] Therefore, the present invention adopts a task management method for a data middle platform to achieve the following functions: forming a data resource pool for the Internet of Things, restricting account permissions and callable databases through an account permission module, and operators using task generation templates to call data to form packaged data, which are cut through the task management module into different calculation methods, so as to find a suitable architecture for calculation on a distributed computing platform, thereby achieving better computing efficiency, separating the computing platform and the data middle platform, and using the data middle platform to coordinate and classify data, thereby improving the computing efficiency of the computing platform.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A task management method for a data center, characterized in that: The following steps are involved: S1: The administrator logs in to the backend to control the operation of the data center. S2: The operator logs in according to the account authority module, enters the task data in the front desk, fills in the task generation template, generates the corresponding task, and adds the task to the task management module; S3: The data synchronization module synchronizes the corresponding data from the corresponding database to the data warehouse according to the task. The task management module divides the task into tasks, generates computing tasks, and allocates the corresponding data in the data warehouse according to the computing tasks to form packaged data. The task management module encrypts the packaged data, distributes it to the distributed computing platform, and returns the calculation results; S4: Use the task management module in the data center to integrate the calculation results and generate results according to the task result template; S5: Use the push module to push the results accordingly.

2. A task management method for a data middle platform according to claim 1, characterized in that: In step S1, the control process includes setting the account authority module, setting the task generation template, setting the task management module, setting the task result template, setting the push module and running maintenance; Set up account permission modules, including departments, operators, available databases, and operation records; The specific setting of available databases is to set the databases that operators can use according to the actual rights management of the department; Set up the task generation template, including reference data content, data processing, task generation, target address, and task result template; The referenced data content specifically includes the database name, referenced content, start time and end time; Data processing will record the requirements for data analysis or statistics on the referenced data content; Task generation will package the corresponding reference data content, data processing, target address and task result template to obtain the task, and send the task to the task management module; The task management module specifically includes task recording, task segmentation, calculation order, data encryption, task dispatching, and filling in task result templates; Task recording is responsible for recording all currently executed tasks; task segmentation divides tasks into several computing tasks; sets the linear order of the split computing tasks as the computing order; data encryption encrypts computing tasks and allocation data to obtain encrypted data; The task dispatcher packages the encrypted data and sends it to the distributed computing platform. The task result template is filled in. The distributed computing platform decrypts and calculates the encrypted data, and returns the calculation results to the task management module. The task management module fills the calculation results into the task result template according to the calculation order and converts the target address to obtain the push address. There are two ways to set up task result templates: fixed templates and custom templates. Fixed templates are templates provided by different departments when managers set up task generation templates. Custom templates are templates set by operators themselves; The push module is used to actively send content to the client.

3. A task management method for a data middle platform according to claim 2, characterized in that: In step S3, the process of synchronizing the corresponding data to the data warehouse is as follows: The data warehouse specifically includes the source library, data warehouse, index library and task library. The data synchronization module includes data access synchronization and data governance modules. The data access and synchronization module works as follows: It obtains data from the data source in two ways: interface acquisition and direct database synchronization. In interface acquisition, data is obtained from the database in a targeted and quantitative manner based on the referenced data content, and regular expressions are used to verify the data. In direct database synchronization, full or incremental synchronization is used to synchronize data to the source repository. The data governance module standardizes the data in the source library, including generating data annotations and data indexes. Specifically, it includes data extraction, data cleaning, data correction, data comparison and data association, and then generates data annotations and data indexes, which are stored in the data warehouse and index library respectively. The task library is used to store the returned calculation results, relevant data of the task generation template, and relevant data of the task management module.

4. A task management method for a data middle platform according to claim 3, characterized in that: In step S3, the process of segmenting and encrypting the packaged data is as follows: The segmentation process is divided into two computing tasks: logical calculation and floating-point calculation according to the actual calculation content. The calculation order is set according to the linear order of the calculation tasks, the calculation order is recorded in the task management module, and the packaged data is encrypted.

5. A task management method for a data middle platform according to claim 4, characterized in that: In step S6, the push module is specifically as follows: The push module includes push code, content code, push type, push address, push process and push deduplication; The push code is the identifier of the push process and is used to calibrate the push process; The content code is the identifier of the content pushed this time, and is used to mark the pushed content; Push types include HTTP, TCP and UDP; The push address is the target address, and there are different push addresses for different push types; The push process includes direct forwarding and processed forwarding. Direct forwarding is to push the content in the corresponding post source library, and processed forwarding is to push the content in the task library. Push deduplication is used to remove duplicate pushed content, including the identification process and identification cycle. The identification process will save the content code corresponding to the sent content. During the identification cycle, the sent content is deduplicated based on the content code.

Citation Information

Patent Citations

  • Digital research and development method and platform based on middle platform

    CN115330226A

  • Data processing method and device supporting multi-source data center joint security computing

    CN116208666A

  • Data middle station for earth observation data

    CN118394750A